Uncertainty Quantification for Deep Regression using Contextualised Normalizing Flows
Authors: Adriel Sosa Marco
John D. Kirwan
Alexia Toumpa
Simos Gerasimou
Venue: NeurIPS 2025 poster ()
Keywords: uncertainty quantification
safe deep learning
Primary area: deep_learning

Abstract: Quantifying uncertainty in deep regression models is important both for understanding the confidence of the model and for safe decision-making in high-risk domains. Existing approaches that yield prediction intervals overlook distributional information, neglecting the effect of multimodal or asymmetric distributions on decision-making. Similarly, full or approximated Bayesian methods, while yielding the predictive posterior density, demand major modifications to the model architecture and retraining. We introduce MCNF, a novel post hoc uncertainty quantification method that produces both prediction intervals and the full conditioned predictive distribution. MCNF operates on top of the underlying trained predictive model; thus, no predictive model retraining is needed. We provide experimental evidence that the MCNF-based uncertainty estimate is well calibrated, is competitive with state-of-the-art uncertainty quantification methods, and provides richer information for downstream decision-making tasks

Uncertainty Quantification for Deep Regression using Contextualised Normalizing Flows
Adriel Sosa Marco John Daniel Kirwan', Alexia Toumpa², Simos Gerasimou
AArquimea Research Center, Spain Department of Computer Science, University of York, York, UK * DDpartmeett of Elect. Eng., and Computer Science and Eng., Cyprus University of Technology, Cyprus {asosa, (arwwann@rrqumma.... com
{alexia. toumpa, simos. ierks ac.
Abstract
Quantifying uncertainty in deep regression models is important both for under- standing the confidence of the model and for safe decision-making in high-risk domains. Existing approaches that yield prediction intervals overlook distribu- tional information, neglecting the effect of multimodal or asymmetric distributions on maaiin.. Similarly, full or approximated Bayesian methods, while yielding the predictive posterior density, demand major modifications to the model architecture and retraining. We introduce MCNF, a novel post hoc uncertainty quantification method that produces both prediction intervals and the full condi- tioned predictive distribution. MCNF operates on top of the underlying trained predictive model; thus, no predictive model retraining is needed. We provide exper- imental evidence that the MCNF-bbsed uncertainty estimate is well calibrated, is competitive with state-of-thh-aar uncertainty quantification methods, and provides richer information for downstream decision-making tasks.
Introduction
Deep regression models have been widely used in applications involving the prediction of continuous variables MI. including drug discovery [2], credit scoring [3] and energy forecasting 44. Despite their broad adoption, safety-critical applications, like medical diagnostics [5], entail high-stake decisions, mandating the development of robust deep regression techniques that equip decision-mate with complementary knowledge about the predictive uncertainty of a regression model.
Uncertainty quantification methods (UQ) are fundamental in establishing the predictive uncertainty of regression models [6]. Recent advances target primarily the investigation of epistemic uncer- tainty (caused by the lack of evidence or knowledge during training) and aleatoric uncertainty (the irreducible uncertainty due data stochasticity). Building on the foundational work quantile regression [7]. Monte Carlo Dropout (MCD) g and deep ensembles
[9] leverage dropout layers models, respectively, to approximate inference time deep Gaussian process and estimate the predictive distribution. Both methods, however, incur extra com- putational overheads and suffer from inefficient sampling, particularly at the predictive distribution tails [10 Bayesian-basse based approaches that perform full Bayesian estimation [I]] or its variational counterpart [12]. albeit rigorous in uncertainty incorporation in the posterior distribution, incur prohibitive computational costs for any modern deep learning model. Conformal prediction (CP) [3] yields statistically rigorous uncertainty intervals that contain the ground truth based on user-defined error rate [14] but its reliance on calibration dataset restricts its applicability.
and multiple
Motivated the need for rigorous UQ safety-critical applications, we introduce MCNE, post hoc distribution-free UQ method for deep regression models underpinned by contextulliedd normalizing
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

flows (NF), MCNF uses a trained deep regression model equipped with dropout layers and yields statistically rigorous uncertainty intervals arising from the full predictive density function. Under the hood, MCNF exploits MCD sampling more efficiently to produce a prediction set per input leveraging key statistical information (e.g., mean, variance) to condition (contextualize) the normalizing flow [15] Our experimental evaluation using a diverse set of datasets and state-of-the-art UQ methods
14 16 demonstrates that MCNF achieves competitive results in terms of marginal coverage while also having lower error values and narrower intervals. Its applicability to deep learning architectures other than feed-forward regression networks is also showcased. Similarly to CQR [14] and MCCP [6], MCNF operates at inference time while also being capable of representing arbitrarily complex uncertainty distributions, which neither CQR nor MCCP support. Our concrete contributions are:
The MCNF method for the uncertainty quantification whose estimates are in the form of distribution-aanosiic predictive distribution.
A comprehensive MCNF evaluation against state-of-the-art UQ methods (MCD, CQR, MCCP) on various standard benchmarks and a physicochemical dataset.
A prototype open-source MCNF tool and case study repository, available at
com/ alexiatoumpa/MCuF
2 Related Work
Uncertainty Quantification (UQ) deep regression models remains an open-ended question [T]_ Quantifying uncertainty in deep learning models enables reasoning about the model's confidence in its predictions. Quantile Regression (QR) 7 constructs prediction intervals by modeling the relationship between a set of independent variables and quantiles of target variables. A typical approach for UQ is training Bayesian Neural Networks (BNNs) [77,, comprising neural networks with probability distribution for the model parameters that learn the predictive posterior of the target variable. Although the output probability distribution of a BNN captures the model uncertainty, BNNs computationalllyinnenss
demanding significantly more training time than other methods [I]].
Monte Carlo Dropout (MCD) g samples weights in each layer using binomial distribution at the selected dropout rate. Each forward pass produces a new estimate coming from the predictive posterior, which approximates Bayesian inference Gaussian process [18]. This technique draws parallels to deep ensembles [9], but shares weights across model realizations.
Variational inference approximates full Bayesian approaches by introducing family of distributions that make the modeling problem tractable, commonly amortizing the parameters of the posterior [99 with an auxiliary function trained on the data.
Conformal Prediction (CP) 13 20 is a distribution-fies non-parametric forecasting method which exploits past experience to determine the level of confidence for new predictions. CP produces prediction intervals indicating the confidence level of the model, which is inversely-related to the interval size (21]. Conformal Quantile Regression (CQR) [14] combines quantile regression with conformal prediction techniques, aiming to construct prediction intervals without distributional assumptions. MCCP [16] combines Monte Carlo dropout and conformal prediction techniques by dynamically adapting the conventional MCD with a convergence condition and employing advanced conformal prediction techniques for the synthesis of robust prediction intervals.
3 Preliminaries
Normalizing Flows (NF) 22 23
is a modeling framework for the characterization of arbitrarily target distribution px (x), where a set of
complex probability distributions, often referred invertible and differentiable transformations is applied over simple probability density functions (e.g., uniform, standard normal) or a base distribution Pz (z). NF samples can be drawn from the target distribution by sampling from the base distribution and applying the set of transformations that convert the latent variable Z into the original random variable X, and estimating the density for given value of the random variable X.
Let X E d be a random variable with an intractable probability density function and Z E Rd another random variable with a known and tractable probability density function. Let also g = 91 o gr. be a set of L differentiable and invertible functions composition (i.c., flow) such that X g(Z).

---

The density function of the target random variable can be expressed in terms of the base density by:
px(x) pz(g pz(6='(x)) II det (J (xx))
(I)
Since NFs can estimate the density function, they can be trained by maximizing the likelihood with respect to the parameters of the transformation functions of the given data [15].
Neural Spline Flows (NSF) 24 are type of transformation flow that fulfills the NF requirements of invertibility and differentiability [15] Monotonic aational-quadrati splines endow the transforma- tions with high non-linearity and flexibility, where the support vector (or knot) widths and heights and the derivatives of the polynomial on the internal knots are estimated using neural network with learnable parameters. These transformations can easily integrate conditions to model conditioned density functions px (x(c), where is the condition (or context) vector.
4 MCNF
Monte Carlo Normalizing Flow (MCNF), whose high-level workflow shown in Fig. I enables quantifying the predictive uncertainty for regression tasks. Let D = {x, y) denote a dataset of size N for regression task, where x and y are the predictor vector and the predicted variable, respectively. Then, in MCNF, the predictive posterior distribution p(y|x x, DD resulting from the Monte Carlo Dropout (MCD) g is calibrated post hoc and is fully decoupled from the training of the predictive (base) model that describes the data. Thus, MCNF separates the modeling task from UQ. and employs MCD derive prior estimations, leveraging the commonly used dropout regularization technique in deep learning architectures.
MCNF, like CP [14,, estimates p(y] x, D) without making distributional assumptions about the uncertainty. Specifically, we achieve this distribution free concept by applying Normalizing Flow (NF) [15] downstream post-processing of the MCD (CD-ggnerated samples. Rather than modeling the predictive variable directly, MCNF uses the NF to describe the distribution of the prediction errors conditioned the prior prediction, propagating the epistemic uncertainty encoded by the MCD prior estimate and combining it with the aleatoric uncertainty of the underlying process.
4.1 Formal deseription
MCNF uses estimates from MCD (μMCD) as a latent variable acting as a prior to the actual predictive distribution. Thus, we express the predictive probability distribution p(y) x, D) by marginalizing the joint probability distribution p(y, JMCD |x, D) over the prior.
S
piu|x, D)
p(u. UMCD xx D) dyMCD = p(u lUMCD- D)P(UMCD |x, D) dyMcD
Eprpsen (x,D) jp(y (MCDD x, D)]
Therefore, the predictive distribution is expressed in the form of mathematical expectation of the conditional distribution p(v) BMCD, x, D). Since Equation (2) is intractable, we resort to Monte Carlo approximation to estimate the predictive distribution of y, resulting in:
TINCD
ply|x x, D) = Epply I) [p(y LYMCD, x, D)
Σ
ply LUMDD x, D)
(3)
7MMD
where WMDD is the number of prior samples to approximate the predictive distribution.
To account for the epistemic uncertainty propagation from the prior to the predictive distribution, we introduce the change of variable, 5 y YMCD. which reflects the prediction error. Hence, the conditioned probability distribution is now expressed in terms of å instead of y. given by p(o N/0lYMC,, x, D). This change does not affect the calculation of Equation (2) nor Equation (3). due to the linearity of the transformation, so p(y |MMCD. x, D) p(ō |JMCD, x, D). Therefore, Equation can be expressed in terms of as:
RMCD
Dulxx
Σ
p(o, I D)
TIMCD

---

(
Morte modely
Tobuu data
Transbmatoon
Dospoe
Groph data
Secuanca
Predition
Line
(200 y 10.0
e=(uli (3 ((0))
Samplingg trrd de
(-44
Cometted swetct cartsukon
Palerr Chocigh Spine
Poriane Gusdell Spen
Laterebase distitction
pile MM..
Normalichg Flow moduls
Insp (s)
Poulf +)
Lege Ja
Figure I: Overview of the proposed MCNF UQ method for regression tasks. First, a set of samples drawn using Monte Carlo Dropout (MCD). These samples are used build a context vector that encodes the MCD predictive posterior and, ultimately, the input observation using convenient set summary statistics. These summary statistics are provided the Normalizing Flow-based model context. Depending the task, either requested number of samples can be drawn from the Normalizing Flow by sampling the base distribution and submitting those samples through the forward pass, or the likelihood of the input observations can be assessed.
Accordingly, the prior distribution be approximated using the Monte Carlo Dropout sampling process, while the conditioned distribution of the prediction errors, 8, is modeled through the normalizing flow, FF (6, yMCD. x) ( (BMCD, x, D) providing efficient correction of the prior. In the latter expression, and ab are, respectively, the flow transformation parameters and the base distribution parameters.
4.2 Building the MCNF Context
In MCNF, the prediction errors are conditioned on the prior estimate YMCD and the observed input features (or descriptors) x. These two variables configure context vector e that should be fed to the normalizing flow head of MCNF as a context to locate and scale the predictive distribution. Through this contextualization, MCNF defines a convenient procedure so that the context definition is agnostic to the imput data structure, and in such a way that MCNF can still be applied post hoc. To accomplish this, define proxy, hxx), over the input descriptors based on an internal representation of the regression model, d, with its learnt parameters, E, Φ(x; E). This approach leverages the feature extraction made by the regression model and uses it to define proper context for the normalizing flow. Since one or more layers may be selected at different depths, this becomes a tunable hyperparameter of MCNF that directly impacts the dimensionality of the context vector. This procedure normalizes the shape of the proxy, which helps to generalize the method.
The context vector should be completed by virtue of the latent variable: the prior estimate /MCD- In practice, we observed that achieving numerical stability and computational efficiency entailed replacing each sample generated using MCD by the estimates of the two first moments of the prior

---

distribution. Thus, the context vector is expressed as follows:
UMCD, log s² (fMCD) h(x))
(5)
where UMCD and log S (ÜMCD) ) are the sample estimates of the expectation and variance of the prior distribution.
Finally, we adapt the definition of the density function of the predictive distribution described in Equation according the proposed context vector (Equation (5)) to obtain our working equation.
TRMCD
pe. (y/x, x. D)
(6; BMCD, log s² (jMcD) h(x),
TiMCD
Note that Equation (6) is not equivalent to Equation (4) However, considering the hypothesis of normality of the distribution estimated with MCD, assume that using the first moments of the distribution of the latent random variable allows us to obtain a reasonable approximation of the marginalized distribution.
4.3
MCNF Training
MCNF training is based on the forward Kullback- Leibler divergence DKL (Equation 7 equivalent minimizing the negative log- log-likelihood of the training data with respect to the model parameters (for the transformation flows) and ab (for the base distribution).
15 This
CN.. (0, 2/) DKL. [Po) x (3/x) Poove (v|x, D)
-Ep (vix) [log pe, (y/x, D) + const.
1
Σ log (9- (0nic, + log det J (ym c, 0) + const.
(7)
is well established that minimizing the log-likelihood may lead to model overfitting and deformed distributions. This leads to uncalibrated uncertainty estimates, which become more apparent for low-uncertainty effects corrupted by large outliers. To rule this out, we regularize Equation (7) by weighing observations proportionally the prior density. Since MCNF provides post hoc corrections of the MCD predictive posterior, we use this as the prior and reformulate LNL (0, ab) as:
(NI = Σ Wn (og pa (g- (Hni Cn, 0)) - log [det Jg- (yni Cn, 0)l
n=l
(8)
log PMCD (yn (ym|X (xx)) (3nlx(n);)) ;T)
where W,, =
E (0,
By introducing in Equation we scale the importance of each observation in mini-batch according to departure from the prior. We use a softmax function o()) such that weights fall within the (0,1) interval. To mitigate the effect of highly informative prior, for MCD, we temperature scale the softmax function using a hyperparameter т. Accordingly, the effect of outliers can be mitigated without affecting the fitting of the regular data. We note that Equation (8) reduces to Equation (7) when + oc (reflecting a design decision when there is knowledge that outliers are absent from the data).
bini-batches are constructed by resampling the MCD samples to account for the variability encoded by the prior. Further details are provided in Algorithm in the Appendix.
4.4 MCNF Inference
MCNF operates hierarchically, i.c., samples need to be drawn from the predictive model using MCD to generate the context prior to assessing the predictive distribution of the uncertainty using the Normalizing Flow head. This means that generate NF samples from the approximated predictive distribution Po. Py, D), we first need to generate uMCD samples from the base model ø(x, E). We will refer this set of samples the prior or warm-up samples.
The required input data to make estimates according to Algorithm connsist of the number of prior samples NMCD, as well as the architecture of the Normalizing Flow F-1 c) and the number of
5

---

Algorithm 1 Steps involved in full forward pass of the proposed MCNF method for inference
Input: D = [Xi, Un J Newt
In=1
Parameters: NMCDD 0(xx E). F(6, c). ANF
Output: Upred and/or po, (o)xx
1: Run RMCD forward passes of c(x,,, E) times using MCD for every every observation, X
2: Collect hidden states, h(x,), generated in each forward pass.
3: Aggregate hidden states from the forward passes.
4: Generate context vector, Cn. according to
5: Feed Cn F(8, Cn)
6: Draw ANF samples, on = (dk,n) PNNE from Zk,n Po. (7) - 6R.n = F(z,,n Ca)
7: Estimate Yn as y/ JA..(K). ōn,k; where j(k) ~ 21 (1,1MMCD)
8: Estimate each sample density feeding 1 k C.)) into 6
9: return (Yo,k k=
ANE and (po, (Vte Ix.. D) k= 1.P E R
P X
samples requested to characterize the approximate predictive distribution uNF. The inference process starts by building the context This entails drawing the requested RMCD samples from the regression model o(x, E) using MCD, which are then summarized to generate the sample mean and log-var to partially construct the context. Since every forward pass é(x, E) produces one proxy of the input, we average these estimates to complete the context vector c. Then, the context vector is given as an input to F c) (line 5). For sampling tasks, we first generate samples from the NF base distribution 2 that are then submitted through the forward pass to obtain ōn = F(Zn.k Ca) and then correct the prior Vn.j((k),MCD with the sampled prediction error to finally get Yr.k (line 8). During the same forward pass, the Jacobian of the transformation flows is also assessed, enabling likelihood estimation by solving Equation (T for pe, & n Xk.n D). For density estimation tasks, an observation yo. is passed along with the context vector C,,, Then, the likelihood is calculated by running the reverse pass of the NF (Equation (T).
5 Evaluation
Base Predictive Model. The base predictive model is Deep Quantile Regressor (DQR) comprising a batch normalization input layer, two fully connected layers (with ReLU nonlinearities and dropout layers with rate 0.1) and an output layer with three linear units for the quantiles The model is trained for 100 epochs using the Adam optimizer, a custom pinball loss function that aggregates the quantile errors, and a batch size of 32. We fixed the learning rate to 5e-4 and the weight decay regularization factor to 1e-6. For the training and testing sets, we use an 80:20 split.
{0.05, 0.5, 0.95}.
Probabilistic Model. The Normalizing Flow component of MCNF uses a sequence of two Neural Splines flows (24), with a 3-layered multilayer perception comprising 64 hidden units which produces the 16 support vectors of the spline transformation and their inner derivatives. As a base distribution, we use factorized Gaussian distribution with trainable parameters sp. The context size determined by the size of the input proxy plus the two statistics (sample mean, and log-var) used to summarize the MCD samples, i.e., (h(x) + 2). To keep the computational overhead related to the prior Monte Carlo Dropout sampling low, we set WMCD 50. The training includes the same partition as the base predictive model, using a batch size of 32 with the Adam optimiser and a 0.001 learning rate. We set 1e10 to instantiate Equation (8) giving the same weight to all observations in a mini-batch.
Comparative Methods. We assessed MCNF against five of-th-a UQ methods that also use the predictions from the base predictive DQR model for uncertainty estimations. Thus, MCQR involves resampling 1000 times DQR using MCD and averaging the q= (0.05, 0.95} quantile outcomes to yield the prediction intervals. MCD derives (McD(y|x) by resampling from the median (q = 0.5) in DQR 1000 times, resembling the conventional Monte Carlo Dropout uncertainty quantification method. MCD is the only other method that can also produce predictive distribution. We also used Conformalized Quantile Regression (CQR) [4), which applies a non-conformity score to DQR to conformalize the prediction intervals, and Monte Carlo Conformal Prediction (MCCP) [16], which conformalizes the prediction intervals obtained with MCQR. Both CP-based methods used 20% of the test set for calibration and prediction intervals aimed at achieving 90% marginal coverage. Note

---

that for all the considered UQ methods that employ MCD sampling, the aleatoric term of the original formulation 8 is left out as is used to propagate the epistemic uncertainty only,
Benchmarks. Adopting the evaluation procedure from CQR 14] and MCCP [16], we used the fol-
lowing datasets to evaluate MCNF: the Boston Housing dataset 25 (506 observations, 14 attributes);
the Concrete dataset [26] (1030 observations. 9 attributes); the Abalone dataset (27] (4177 observa-
tions, 11 attributes); the Tertiary Protein Structure dataset [28] (45730 observations, 10 attributes); the
wave energy dataset [29] (63600 observations, 149 attributes), and the superconductivity dataset [30] (21263 observations, 81 attributes). These datasets were obtained from [3]] Similar to [4] and to examine the MCNF's ability to capture the uncertainty of complex distributions, we also include two synthetic datasets: the dataset from [4] with univariate predictor samples and few large outliers (termed Romano-Origee Original) and an extension (termed Romano- Mood) with a multimodal distribution.
Performance Metrics. To compare the performance of all methods, we used the metrics reported below. We report results across 20 independent runs of the MCNF training procedure described to account for randomness. The training/test partitioning was generated per run and held constant to train both the base predictive model and the comparative methods (MCNF, CQR, MCCP, MCCQR).
Marginal coverage signifies the proportion of the test set over which the predicted quantile intervals include the ground truth, given by C(X, Y) N Σ
N and g1 91-a(1)) are the predicted quantiles given and
I (ys E (x,) 91 (2:)]), } where ga (3:4
0.05.
Interval size signifies how well the prediction intervals capture the aleatoric and epistemic un- certainties, with smaller intervals denoting easier inputs and larger intervals harder ones, given by
Y(X,Y)= = median, (q1-a (z;) - qa (x:)), where qa. Q1-a and a are as above.
Accuracy assesses if the prediction intervals under- or overestimate uncertainty, specified by the mean absolute error (MAB) for 0.5 and given by M MAB(X,Y) = # > lys 90.5(x;)).
5.1 Results Summary
Accuracy and coverage. Table I summarizes the performance results of our evaluation. MCNF overall yielded the smallest MAE, closely followed by MCD, indicating that training the NF MCNF- component helps improve the accuracy of the predicted interval. MAEs for the other methods are larger, usually by one order of magnitude, compared to MCNF and MCD. This is expected since the prediction adjustment carried out in the CP-based methods CQR and MCCP is homogeneous for both upper and lower intervals, which is only useful when the distribution is close to a Gaussian.
Considering marginal coverage C, we observe that all methods, except for MCD, provide values close to the theoretical 90% coverage, showing similar capabilities and some degree of conservativeness. This is especially true for the conformalized methods (i.e., CQR and MCCP) for the smallest datasets (Boston housing and Concrete). However, MCD does not account for alcatoric uncertainty and, thus, the intervals generated from the quantiles of its predictive distribution are highly non-conservative. For larger datasets, the miscalibration of the conformalized methods becomes smaller and more stable the number of observations calibrate the prediction intervals increases. MCNF outperforms its MCD counterpart, highlighting the benefits of providing post hoc corrections over the latter. In addition, MCNF is computationally more efficient than MCD, especially when the number of samples to approximate the predictive distribution increases.
Interval size. Although all methods show some sensitivity to the actual uncertainty associated with the given dataset (as shown by the variability of the interval sizes Á in Table 1b. MCNF yields the smallest interval sizes while maintaining the expected 90% marginal coverage. While MCD also has small interval sizes, its marginal coverage is much worse than the other methods, failing to meet the expected threshold. Likewise, CQR and MCCP, in order to achieve the 90% through conformalization, yield conservative intervals which are reflected to their corresponding interval sizes. Accordingly, MCNF provides the best tradeoff between coverage and interval size, showing smaller interval sizes for similar marginal coverages.
Complex distribution. Unlike the multivariate datasets examined so far (e.g., Boston housing, Concrete), the Romano- Original and Romano-Mod univariate synthetic datasets allow for clear visual comparison in reconstructing the data distribution. Figure the prediction intervals from MCNE, MCD, and MCCP, alongside the predictive distributions across the predictor variable x. The
7

---

Table Experimental results by dataset and metric over 20 runs. Methods are columns: each cell
shows mean + std of coverage (C), median MAE (MAE), and median prediction interval size (Å).
Data Metric CQR
DQR
MCCP
MCD
MCQR
MCNF
NF
Concrete Boston H.
0.957=0.044
0.950+0.031
0.000
0.726::0.059
0.949=0.032
0.904+00044
0.782=0.062
MAE
0.224
0.31510..40
0.19
0.078.10.009
0,318 0.040
0.078.10.009
0.073
0.820:10.481
0.543:10.034
0.777=0.404
0.254:1.0023
0.541
0. 0.409110.038
0.037
C
0.908
0.952=00.02
0,942=0.041
0.601
0,949=0.014
0.920=t0.021
0.814=0.035
MAE
0.004
0.372##00099
0.375=0.038
0.113::00012
0.378=0.038
0.085=00012
0.084=0.008
A
0.660.10.078
0.689100.03
0.
0.290.10.016
0.688 0.004
0.491110.031
0.366 0.026
Abalome
0.904:00.025
0.915:10.014
0.030
0.341::0.027
0.914=0.014
0. 0.866:000017
0.874
MAE
0.344
0.350
0.344=0.038
0. 0:0
0.354=0.031
0. 0.099-10.007
0.098
0.574=0.047
0.586=0.0066
0.569=0.048
0.132:00.007
0.587=0.025
0.514++00000
0.507=0.050
Protein
C
0.899=0.0009
0.922+00009
0,001
0,354::0.018
0.921
0,927=000006
0.887 0.887=0.018
MAE
0.004
0.952+00.038
0.937=0.042
0.245=:0.010
0,952=0.039
0.202-:00009
0,186=0.008
A
1.827=00.02
1.861
1.828=000019
0.406::0.001
1.857::0.019
+00033
1.073
Wave
C
0..099
0.962.10.008
0.000
0.828110.0117
0,964_10.006
19.01000000
0.807
MAE
0.005
0.008+0.001
0.005=0.001
0.001++0.0003
0.008=0.001
0. 0.002+000001
0,002=0.001
0.010==0.001
0.016::00002
0.010=0.001
0.009::000003
0.016=0.002
0.013+t0.001
0.006=0.001
Super
0.899110.012
0.934:1..006
0.902-100014
0.482.1.0001
0.934=00007
0.912.1.0009
0.866 0.011
MAE
0.283 0.002
0.209:10.02
0,289 0.224
0.110.1..006
0.003:000024
0.103:10.005
0.096
0.847
0.879#000028
0.857=0.028
0.270:::00014
0.888=0.026
0.791=000035
0.679=0.011
R-OG
0.905
0.900=00.019
0.030
0.239
0,901=0.020
0.926=1000014
0.912=0.019
MAB
2.208:10.110
2.176:10.134
0.150
0.511:10.103
0.182=0.134
0.406:10.206
0.154
A
3.651 0.2207
3.567.10.159
3.604_00.188
0.348.10.037
0.519
3.321110.232
3.371 :: 0.234
R-OD
0.912-10.028
0.948:10.027
0.918=00032
0.5401:0.0513
0.966=0.015
0.952:40.015
0.876=0.088
MAR
0,201 0.002
0.234:10009
0.223-0.032
0.074:100008
0.274=0.036
0.067-10.006
0.082
0.054
0.495:10.029
0.401 0.033
0.138:100001
0.505 0.009
0.438:100026
0.382 0.031
Solubilay
0.922
0.860:10.046
0.047
0.537:1:0003
0.910=0.025
0.8911:0.045
0.831=0.000
MAE
0.749=0.289
0.490++0.118 115
0,836=0, 185
0.221
0.637=0.108
0:207=40.014
0.215=0.024
1.700=:0.601
1.155=0.108
1.685::0.345
0,504::00038
1,289:00.108
1.14400.154
1.021#0.133
Romano-Mon dataset exhibits heteroskedasticity and varying distributions of the predicted variable for different J values. MCNF effectively captures the multimodality of the predicted variable y for small x, transitioning to unimodal distribution as increases. This result is further corroborated by the overall better results achieved by MCNF against CQR, MCCP and DQR across the marginal coverage C, interval size Á and MAE in both Romano-Original and Romano-Mod datasets.
Predictive Model Impact. Since MCNF is a post hoc UQ method, we evaluated the impact of the base predictive model on MCNF's performance. Thus, we assessed the performance using a well-trained predictive model and an underfitted model on the Concrete, Superconductivity, and Protein datasets. Selecting the well-trained predictive model entailed training 15 predictive models of the same architecture, for 100 epochs each, and selecting the model with the lowest RMSE. Similarly, selecting the underfitted model entailed training 15 predictive models for 6 epochs and selecting the model with the highest RMSE. Figure Figure3]shows the coverage and confidence interval for these two experiments, with results for the well-trained predictive model presented with a darker color and results for the underfitted model shown with a lighter shade color. Although the marginal coverage between the two predictive models and across most UQ methods (except from MCD) are similar and around the 90% threshold, the confidence interval plots (bottom) indicate that a well-trained predictive model provides, expectedly, narrower interval sizes. More importantly, though, even with an underfitted predictive model, MCNF yields narrower interval sizes than the -fate-of-the-art UQ methods and the coverage values are comparable to the well-traines predictive model. Accordingly, these results demonstrate that MCNF can adapt its UQ estimate based on the quality of the predictive model.
Epistemic uncertainty propagation. The performance metrics for MCNF without propagating the epistemic uncertainty through prediction errors via MCD sampling are shown in the last column of Table [ (reported as NF). To produce these results, the MCNF workflow remains the same but prediction errors are not resampled. It is noteworthy that with this setup, there is a general
8

---

3.0
Observation within coverage
MCNF 90% ai (smoothed)
Observation outside coverage
MCNF Predictive Density
MCNF Median Prediction
MCD Predictive Density
MCCP
2.0
1.5
0.5
0.0
Figure 2: MCNF predictions (y) against the synthetic Romano- dataset, generated as described in the Appendix. Blue circles represent data observations within the 90% marginal coverage of MCNF, whereas the pink circles fall outside this range. The orange interval delineates the MCNF smoothed marginal coverage (superimposed over the unsmoothed interval, in gray). The broken green line represents the median marginal coverage. The ridge lines represent kernel density estimates of the predictive distributions for the MCNF samples (gray) and the prior MCD samples (black).
concrete
supercenductiviy
100
protein
coverige (meai stt)
80-
80
90-
70-
70-
70-
F
80-
40-
1
40-
40-
3.4
2.0
1.8
1.4
1.2
12-
Green
12-
2.8
0.8
0.-
04-
090
maq
100
most
Method
Method
Method
Figure 3: (best viewed in color) Coverage and confidence interval results for well-trained predictive model (dark colored) and an underfitted predictive model (light colored).
performance decay for across all datasets included in the evaluation. When this source of uncertainty is not explicitly propagated, the prediction intervals deduced from the predictive distribution are consistently narrower than those obtained with the full MCNF.
5.2 MCNF application on Graph Neural Networks
We demonstrate the generalizability of MCNF other deep learning model architectures through its application Graph Neural Network (GNN) predictive model B22 to make structure- based predictions of the physicochemical properties of molecules. The GNN model features three graph convolution layers based on the message passing mechanism. followed by an average readout layer to combine node-level encodings into a graph-level encoding. The next layers are batch normalization layer to center and scale graph-level encoding, and sequence of two dense layers of 300 neurons each (first a linear layer, then a second applying ReLU). Dropout was applied globally rate of 0.15. We use the Solubility dataset (33 that includes solubility data of 829 drug-like like molecules.

---

We challenge MCNF by making it retrieve context for the NF component using an internal repre- sentation of a pre-trained GNN. Therefore, the parameters of the GNN remain constant while fitting the NF component of MCNF. The results for this experiment are shown at the bottom of Table ] The obtained results are in agreement with those observed for the other datasets (Boston housing, Concrete, etc.) using deep regression models. In particular, all methods exhibit good performance in terms of marginal coverage, except for MCD. Among the well-calibrated methods, MCNF stands out by providing the best trade-off between coverage C, prediction interval sizes Á and MAE.
Conclusion and Future Work
MCNF is a post hoc method that quantifies uncertainty in deep regression models by estimating the predictive distribution of the predicted variable. To achieve this, MCNF utilizes pre-trained deep regression models with dropout layers and models prediction errors using a shallow normalizing flow to correct prior MCD estimates. Through a comprehensive experimental evaluation comprising diverse datasets and state-of-the-art UQ methods, we demonstrate that prediction intervals from MCNF are well-calibrated, with smaller median sizes, providing richer information than baseline methods. The approach generalizes well to pre-trained GNNs, showing good calibration and adaptivity, Additionally, we mitigate the negative impact of outliers on the forward Dollback-Leiblee divergence loss funetion by using MCD before weighing observations in mini- mini-batch. Future work could involve extending the MCNF formalism to classification problems and further characterizing the knowledge arising from the predictive model and its underlying assumptions. Furthermore, we will investigate techniques to improve the computational efficiency of MCNF by reducing the number of required MCD samples and coupling it with more efficient alternatives for building the prior knowledge.
Acknowledgments and Disclosure of Funding
This work has been supported by the projects QCircle (grant agreement No 101059999) and SO- PRANO, GuardAl and A14Work (grant agreements No 101 120990, 101 168067 and 101135990, respectively), funded by the European Union. Views and opinions expressed are however those of the author(s) only and do not necessarily reflect those of the European Union. Neither the European Union nor the granting authority can be held responsible for them.
References
[1] Stéphane Lathuilière, Pablo Mesejo, Xavier Mlameda-Pinedae and Radu Horaud. A comprehen- sive analysis of deep regression. IEEE transactions on pattern analysis and machine intelligence, 42(9):2065-22) 20881 2019.
[2] Yuanqui Du, Arian R. Jamasb, Jeff Guo, Tianfan Fu, Charles Harris, Yingheng Wang, Chenru Duan, Pietro Liò, Phillippe Schwaller, and Tom Blundell. Machine learning-aide generative molecular design. Nature Machine Intelligence, 6:589-604, 2024.
[3] Ralf Kellner, Maximilian Nagl, and Daniel Rösch. Opening the black box- cox-tuantile neural networks for loss given default prediction. Journal of Banking & Finance, 134:106334, 2022.
[4] Yixiao Yu, Ming Yang. Xueshan Han, Yumin Zhang, and Pingfeng Ye. A regional wind power probabilistic forecast method based deep quantile regression. IEEE Transactions Industry Applications, 57(5):4420-444 4427, 2021.
[5] Raghavendra Selvan, Frederik Faye, Jon Middleton, and Akshay Pai. Uncertainty quantification in medical image segmentation with normalizing flows, 2020.
[6] Moloud Abdar, Farhad Pourpanah, Sadiq Hussain, Dana Rezazadegan, Li Liu, Mohammad Ghavanzadeh, Paul Fieguth, Xiaochun Cao, Abbas Khosravi, U Rajendra Acharya, al. A review of uncertainty quantification in deep learning: Techniques, applications and challenges. Information fusion, 76:243-297, 2021.
[7] Roger Koenker and Gilbert Bassett. Regression quantiles. Econometrica: journal of the Econometric Society, pages 33-50, 1978.
10

---

[8] Yarin Gal and Zoubin Ghahramani. Dropout as a bayesian approximation: Representing model uncertainty in deep learning. In international conference on machine learning, pages 1050-1059. PMLR, 2016.
[9] Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. Advances neural information processing systems, 30, 2017.
[10] Jakob Gawlikowski, Caio R. N. Tassi, Muhammad Ali, et al. A survey of uncertainty in deep neural networks. Artificial Intelligence Review, 56(Suppl 1):1513-1889, 2023.
[11] Radford M Neal. Bayesian learning for neural networks, volume 118. Springer Science & Business Media, 2012.
[12]
Diederik P Kingma and Max Welling. Auto-encoding variational bayes, 2014.
[13] Vladimir Vovk, Alexander Gammerman, and Glenn Shafer. Algorithmic learning in random world, volume 29. Springer, 2005.
[14] Yaniv Romano, Evan Patterson, and Emmanuel Candes. Conformalized quantile regression. Advances in neural information processing systems, 32, 2019.
[15] George Papamakarios, Eric Nalisnick, Danilo Jimenez Rezende, Shakir Mohamed, and Balaji Lakshminarayanan. Normalizing flows for probabilistic modeling and inference. Journal of Machine Learning Research, 22(57): 10:::-166 164, 2021.
[16] Daniel Bethell, Simos Gerasimou, and Radu Calineseu. Robust uncertainty quantification using conformalised monte carlo prediction. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 10939-20944) 2024.
[17] David JC MacKay. practical bayesian framework for backpropagation networks. Neural computation, 4(3):448 472, 1992.
[18) Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, llya Sutskever, and Ruslan Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting. The journal of machine learning research, 1929-1992,- 2014.
[19] Durk P Kingma, Tim Salimans, and Max Welling. Variational dropout and the local reparame- terization trick. Advances in neural information processing systems, 28, 2015.
[20] Matteo Fontana, Gianluca Zeni, and Simone Vantini. Conformal prediction: a unified review of theory and new challenges. Bernoulli, 29(1):1-23 2023.
[21] Anastasios Angelopoulos, Stephen Bates, Jitendra Malik, and Michael I Jordan. Uncertainty sets for image classifiers using conformal prediction. arXiv preprint arXiv:2009.14199 2020.
[22] Ivan Kobyzev, Simon JD Prince, and Marcus A Brubaker. Normalizing flows: An introduction and review of current methods. IEEE transactions on pattern analysis and machine intelligence. 43(11):3964-3979 2020.
[23] E. G. Tabak and Cristina V. Turner. A family of nonparametric density estimation algorithms. Communications Pure and Applied Mathematics, 66(2):145-164 2013.
[24] Conor Durkan, Artur Bekasov, Iain Murray, and George Papamakarios. Neural spline flows. 2019.
[25] David Harrison and Daniel L. Rubinfeld. Hedonic prices and the demand for clean air. Journal of Environmental Economics and Management, 1978. Environ. Econ. Manag.
[26] I. Yeh. Modeling of strength of high tigh-peeformance concrete using artificial neural networks. Cement and Concrete Research, 1998.
[27] W. Nash, T. L. Sellers, S. R. Talbot, A. Cawthorn, and W. B. Ford. The population biology of abalone (haliotis species) in tasmania. i. blacklip abalone (h. rubra) from the north coast and islands of bass strait. Marine and Freshwater Research, 1994.

---

[28] Prashant Rana. Physicochemical Properties of Protein Tertiary Structure. UCI Machine Learning Repository, 2013. DOI: htpps//dcoo.rr//0224442/////
[29] Mehdi Neshat, Bradley Alexander, Nataliia Y. Sergiienko, and Markus Wagner. Optimisation of large wave farms using multi-strategy evolutionary framework. Proceedings of the 2020 Genetic and Evolutionary Computation Conference, 2020.
[30] Kam Hamidieh. data-driven statistical model for predicting the critical temperature superconductor. Computational Materials Science, 2018.
[31] Markelle Kelly, Rachel Longjohn, and Kolby Nottingham. The uci machine learning repository. attps://archive.e ics uci edu Accessed: 2024-04-01.
[32] Zonghan Wu, Shirui Pan, Fengwen Chen, Guodong Long, Chengqi Zhang, and Philip S Yu. A comprehensive survey on graph neural networks. IEEE transactions on neural networks and learning systems, 32(1 3:42):4244 24, 2020.
[33] Mario Lovrié, Kristina Pavlović, Petar Zuvela, Adrian Spataru, Bono Lučić, Roman Kern, and Ming Wah Wong. Machine learning in prediction of intrinsic aqueous solubility of drug-like compounds: Generalization, complexity, or predictive ability? Journal of Chemometrics, 35(7-8), jul 2021.
12

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The claims of contributions made in the abstract are properly supported by the evaluation section of the manuscript.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
The conclusion section outlines the limitations and direction for future work of MCNE
Justification:
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-speciiiiatinn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
13

---

Answer: [NA]
Justification:
The paper does not present any theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof's can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification: The experimental results described in full detail: the datasets used, the model architecture, the hyperparameters for training and testing MCNF.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, descríbing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is new model (e-g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path reproducing verifying the results. 5. Open access to data and code
14

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification:
We provide prototype open-source MCNF tool and case study repository.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.com cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" an acceptable answer. Papers cannot be rejected simply for including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ praip..cc/publiccguu.eee CoodeSumeeieeeeee ssi iessonPolll molyicyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: All training/testing setup details, as well as model architecture, are described in the manuscript, support reproducibility.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: The experimental results have been run multiple times, thus the mean and standard deviation values of the reported metrics presented.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
15

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma enror bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: We provide details regarding the computational setup used for our evaluation in the Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https: ptttps://neuripptt http://wwwrrup.ccpupie/eeeeeeeet http:pp........ccccccccccc.e.eaa
Answer: [Yes]
Justification: anonymity.
The authors have reviewed the NeurIPS Code of Ethics, and preserve
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification:
The manuscript has no societal impact of the work conducted.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
16

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The work presented does not pose any risks of misuse.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: Owners of the assets and contributors properly credited, and the appropriate license is used.
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
17

---

If assets are released, the license, copyright information, and terms of use in the thcode. has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
package should be provided. For popular datasets, paperswi
latasets.paperssit
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: The code base for MCNF will be made public. This is made available with appropriate instructions and documentation.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: There was no crowdsourcing involved in this work, nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: There no crowdsourcing involved in this work, research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
18

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
19

---

A
MCNF Training
Mini-batch strategy
Algorithm 2 Mini-batch building scheme to train the Normalizing Flow head of MCNF
Inputs: Xn, Un Jn-n=1 (3),
PMCD
Parameters: b = batch size
Output: 0
1: Initialize 0 = {}. and J = (1,
2: while [J]
do
3: Initialize 0 = 0
4: for h: 1 to do
5:
Generate a random index, n ~ (1) and pick (Xn. Un)
6:
Generate random index ~ (1,1,MMC and draw Un,
7:
Calculate prediction error 5, = Un MnD
8:
Aggregate n-th observation current mini-batch, Update indices set, J = J {n1
((X,, 8n)}
9:
10:
Update mini-batches set, 0 0 U 0,
During MCNF training, in order to effectively propagate epistemic uncertainty pa (ylx, D) when the latter is approximated as Equation
(6. we bootstrap the prediction errors. To this end, each mini-batch used evaluate the gradients of the module based on the NFs is obtained by bootstrapping on a set of TCCDD samples, (Ht 1
PMCD previously generated from the distribution p(yMCD x).
Thus, the original training dataset = (x,, Vn Jwm further processed at each iteration of the training steps recalculate the prediction
dn indicated Algorithm Consequently, the implemented mini-batch construction strategy enables every mini-batch to have a different realization of the prediction error per observation each iteration.
B Synthetic dataset
Romano-Mod dataset. A univariate stochastic process is introduced, by adapting the equation provided in Appendix B in [14]. The distribution that characterizes the uncertainty incorporates heteroskedastictty, which is a particularly relevant validation case to test whether the method adapts correctly to the local distribution of the data. The updated stochastic process is given:
y Poisson (sin + 4) + (Be, + +) 5(u .7.2
where ō() represents the Delta Kronecker function and and follow a standard Gaussian dis- tribution. The parameters of the model uncertainty distribution, as well as y and ū. which control the magnitude and rate of outliers in the sample generated. We introduce b to induce linear correlation between the predictor x and the predicted variable y. Table summarize the values used for these parameters and Figure 4 iillsttrates the data generated.
and 4, which condition the heteroskedasticity of the
Table 2: Parameters for the creation of the synthetic dataset.
Data
Parameters
Romano- Romano-Original
0.1
0.05
0.0
0.0
Romano-Mod
0.05
2.0
20

---

1=100, b=0, y=25
-=100, b-2, y=150
Data Outside
ment lb
20
ment
Predictive-med an
10
-20
-40
-5
-60
-0.5
y=50000, b=0, 2=
T=50000, b=0, y=150
40
30
20
10
-10
-10
-20
0.5
Figure 4: Visualisation of the synthetic data produced based Equation (9).
C MCNF performance details
C.1 Adaptivity of prediction intervals
In terms of prediction interval adaptivity, all examined UQ methods, including MCNE, sensitive to the uncertainty inherent in the data, as evidenced by the variability of the quantiles of the interval sizes shown in Table 3 MCD has the narrowest intervals but covers much smaller proportion of the true values, achieving a very poor coverage overall. Considering the UQ methods that perform well and converge the expected marginal coverage (90%), MCNF provides the narrowest intervals and maintains similar marginal coverage to the other methods. Quantile MAE (MAF,)) where
Di y (x:)) + 134 90.95 (x:)). which provides additional evidence of the improved trade-off between coverage and interval sizes for MCNF, exhibits smaller values for similar marginal coverages.
MAE MAE,(X, YX Σ>
C.2 Ablation study
C.2.1 Hyperparameter study
We also examined the performance of MCNF for the following hyperparameters: number of epochs (epochs E {20,50, 50, 100, 150}). number of normalizing flow samples (Πır {100, 200, 500)), and number Monte-Carlo Dropout samples (IмCD E {50, 100, 150)). For this evaluation, we employ the same prediction model across all tests, which was selected based on the lowest RMSE value, as described in Section [.1
Initially, we evaluate the number of epochs specified for training MCNF, examining epochs (20, 50, 100, 150), as shown in Figure whilst keeping the number of normalizing flow samples and Monte-Carlos Dropout samples fixed to NNF 500 and DMCD 50, respectively. Figure @ iillustratss
21

---

Table 3: Prediction interval sizes A, () for v E {0.05, 0.25, 0.5, 0.75, 0.95], where
(X Y) = Qi= (go (2;) 40 (2;)) [p]] and Quantile MAE, where MAE
+DW yi - 90.05 (0,05(z,) + 1t:- - g0 (z,) for all UQ methods and datasets.
N
Data Metric| CQR
DQR
MCCP
MCD
MCQR
MCNF
NF
Åo.os
0.650 + 0.481
0.374
0.605
0.409
0
0.373 + 0.030
0.300 = 0.027
0.163=0.019
Boston H.
20.25
0.730 + 0,482
0.458 +: 0.030
0.690 1
0 0.018
0,456 + 0.029
0.352 + 0.033
0.218=0.030
Ao.5
0.820 + 0.481
0.543 + 0.034
0,777 + 0,404
0.254 + 0.023
0,541 + 0.032
0.409 = 0.038
0.277=0.034
Ao.rs
0,971 + 0.494
0.695 + 0.058
0,932 + 0,400
0,318 + 0.029
0.692 + 0.057
0.507 = 0.059
0.363=0.017
A0.05
1.315 +: 0.520
1.048 + 0.084
1,269
+ 0.401
0.497
+: 0.054
1.044 + 0.081
0.834 +: 0.092
0.61740.132 132
MAB,
0.824 +0.482
0.551 +0.038
0,785 +0.402
0.279
+0.023
0.549 #0.037
0.428 #0.037
0.296=0.029
Ao.os
0.430 + 0.081
0.461 + 0.019
0,458 + 0.105
0,183 + 0.015
0.463 # 0.019
0.298 + 0.027
0.193=0.021
Comcrete
A0.25
0.542 + 0.075
0.574 + 0.030
0.569 + 0.097
0.238 1 0.014
0.573 + 0.030
0.394 + 0.025
0.283
Ao.5
0.660 = 0.078
0.689 0.033
0.682 0.102
0.292 4 0.016
0.688 + 0.034
0.491 + 0.031
0.366=
A0.75
0.832 0.099
0.859
0,852 0.115
0.028
1 0.052
0.621 1: 0.053
0.469100.027
A0.95
1.045
+: 0.103
1,076 4 0.068
1.72 4: 0.113
0.528
0.049
1.022 0,066
0.818 +: 0.091
0.066
MAE,
0.636
+:00799
0.667 10,040
0.663
1:0.00
0.329
100026
0,666
1:0.039
0.472 10.032
0.352
Ao.os
0.357
0.367
0.349
0.007
0.370 1 0.020
0.294 1 0.017
0.277
Abalome
20.26
0,446 + 0.046
0.457
0.025
0.439
0.048
0.109 0.007
0,459 + 0.025
0,380 1 0.019
0.371 0.0039
An.s
0.574 + 0.047
0.586 + 0,026
0,569
0.048
0.132 + 0.007
0.587 + 0.025
0.514 + 0.020
0.000
A0.75
0.748 1 0.043
0.757 1 0.027
0,739 1 0.048
0,163 1 0.007
0,758 1 0.027
0.744 1 0.040
0.724.10.045
Ao.95
1.117 + 0.037
L,18 + 0.038
1,109 + 0.046
0,249 + 0.013
1.127 + 0.040
1.098 = 0.052
1.092
MAE.
0.551
10.043
+0.021
10.045
0.222 10.008
10.021
0.499 100017
0.494=0.030
Ao.os os
0.709 1 0.080
0.744 1 0.078
0,720 1 0.081
0,113 1 0.015
0.749 1 0.080
0.341 1 0.046
0.270110.00
Protein
A0.2s
1.549 + 0.035
1,585 = 0.026
1.557 + 0.032
0,227 + 0.019
1.584 + 0.026
1.244 = 0.111
1.155
An.s
1.827 + 0.022
1.861 + 0.019
1.828 + 0.019
0.406 + 0.011
1.577 + 0.019
1.781 = 0.033
1.673=0.062
A0.75
2.010 + 0.023
2,044 + 0.022
2.009 0.020
0.524 + 0.009
2.038 + 0.022
2.061 + 0.026
0.904
An.95
2.249
0.032
2.284
2,250
0.082
0.762
2.279 0.038
2.393 = 0.043
2.230=0.013
MAE.
1,349 10.088
1,383 10.01
1.354 10.04
0,576 100014
1,382 10.04
1.172 00059
1.048 + 0.088
Ao.o5
0,004
0.001
0.010 0.001
0.001
0.000
0,010 1 0.001
0.008 1 0.001
0.003
A0.25
0.007
0.001
0.002
0.001
0007
0,013 0.002
0.010 0.001
0.001=0.001
Wave
Ao.s
0.010 + 0.001
0.016 0.002
0.010 : 0.001
0.009
0.000
0,016 : 0.002
0.013 : 0.001
0.006
A0.76
0.020 = 0.001
0.026 + 0.002
0.020 + 0.001
0.012 0.000
0,026 = 0.002
0.022 = 0.002
0.016=0.002
A 05
0,040
1: 0.002
0.046 1 0.003
0,041 1 0.002
0.021 1: 0.001
0.047 1 0.003
0.043 1: 0.014
0.039
MAF
0.009
10.001
+0.002
10.000
10.0002
0,016 10.002
0.013 10.001
0.006=0.001
oo.os
0.163 + 0.015
0. + 0.012
0,165 + 0.014
0.058 + 0.007
0,196 + 0.012
0,141 = 0.014
0.096
Ao.25
0,286 1 0.015
0.318 1 0.012
0,289 1 0.015
0,111 1 0.006
0.319 1 0.011
0.270 1 0.024
0.220 0.002
Super
Ao.5
0.847 + 0.032
0.879 1 0.028
0,857
0.028
0.270 1: 0.014
0,888 1 0.026
0.791 1 0.035
0.679 0.044
Ao.76
1.769 + 0.058
1802 + 0.061
1.768 4 0.059
0.479 + 0.020
1.799 + 0.061
1,471 = 0.089
1.336
0.08
A0.
2,074
2.105
0.059
2.097 1 0.064
2.087 1 0.106
1.988 0.071
MAP
0.610
+0.025
0.641 +0.025
0.612 100029
0.292 +0.007
0.642 10.024
0.566 100023
0.462=0.022
Ao.os
1.764 0.181
1.698 + 0.145
1.751
0.207
0.110 : 0.040
1.708 0.146
1.832 - 0.345
2.009
A0.25
2.374 0.225
2.309
0.200
2,361
0.222
0.046
2.316 : 0.196
2.852 1: 0.193
2.785
R-OG
Ao.s
3.631 0.207
3.567 0.159
3,604
0.348 0.037
3,573 + 0.159
3.321 = 0.232
3.371=0.22 231
An.rs
4.262 = 0.245
4,198 + 0.225
0.226 4.249
0.458 + 0,035
4,209 : 0.223
4.211 + 0.091
0.093=0.124
ao
4.516 = 0.297
4,446 + 0,276
4.505
0.281
0,557 + 0.037
4,463 + 0.276
4,373 + 0.098
4. 4.296=0.129
MAE,
2.615 + 0.350
2.583 0.350
0.242 2.597
1.117 0.166
2.589 + 0.350
2.977 + 0.395
7.886=0.255
Ao.es
0.329 + 0.043
0.340 +: 0,025
0,319
0.041
0.079 + 0.007
0,339 + 0.025
0.230 + 0.016
0.188
RMOD
20.25
0,382 1 0.042
0.392 1 0.023
0,372 1 0.041
0,105 1 0.007
0.391 1 0.022
0,300 1 0.025
0.235.10.036
Ao.s
0.491 + 0.042
0.495 + 0.029
0,483 + 0.042
0.141 + 0.006
0.494 + 0.029
0.438 + 0.026
0.382 031
Ao.75
0.543 + 0.045
0.543 + 0.031
0.031
0.535 1 0.042
0. 1: 0.007
0.543 1 0.030
0.511 : 0.025
0.408
Ao.05
0.661
+ 0.044
0.660
0.036
0.656
0.044
0.201 + 0.009
0.660 + 0.038
0.599 + 0.036
0.528=0.048
MAE.
0.364 = 0.064
0.433 + 0.028
0,332 # 0.037
0.187 + 0.012
0,434 = 0.029
0.350 = 0.032
0.288
Ao.05
1.157 1 0.633
0.610
1,153 0.369
0.021
0.756 1 0.081
0.614 1 0.084
0.446
Solubiiit
A0.25
1.423
+ 0.611
0.867
0.083
1.416 0.365
0.380 0.033
1.030
0.086
0.933 - 0.130
0.765
An.s
1.700
+ 0.601
1.155 + 0.108
1.685
+ 0,345
0.504
+ 0.039
1,289 + 0.108
1.184 = 0.154
0..32
Ao.r5
2.055
+ 0.582
1,511 = 0.142
2.020 0.338
06188 0.044
1.616 + 0,139
1,500 0.164
0..93
Ao.95
2.693
+
2.156 0.214
2.603 0,373
0.749
: 0.043
2.220 + 0.206
1.972 = 0.152
1.723=0.255
MAE
1.665
+ 0.590
1.124 2 0.087
1.336
0.597 0.031
1.242
0.085
1.119 - 0.118
00.14
the results of evaluating the different normalizing flow sample values ONF E {100, 2100,200, 500). setting epochs- 100 and NMCD 50. Lastly, Figure visualizes the ablation results for the different Monte- Carlo Dropout sample values DNCD E (50, 100, 500), whilst training the MCNF for epochs = 100 and sampling anF = 500 normalizing flow samples. Our results demonstrate that all tested DMCD
22

---

52, 100,00000000001 =500 Merco
boston, housing
abatons
R
3
convegy (twan
20
00
40-
0.600
0.00
0.94
0.2-
0.00
042
0.99
0.400
0.078
0.100
0.00-
0.109
0.070
0.102
0.006-
0.903
0.000
0/29
0.89
0.008
0.34
110
100
opoohs
opocho
spo07s
Figure 5: Evaluation of MCNF on Boston Housing, Abalone, and Concrete datasets for different epoch values (20, 50, 100, and 150).
values perform similarly in terms of the coverage metrics, and the difference in the confidence interval and the median MAE is negligible. Similar observations have been derived for the various epoch and ONF values evaluated.
C.2.2 Outliers handling: testing impact of r hyperparameter
As outlined in recent UQ survey papers (6) UQ methods can be impacted by outliers. Thus, we investigated the performance and response of MCNF to outlier data. Table 4 shows experiments carried out for different T, b and ry value combinations on MCNF coverage (C'), median MAE (MAE), and Quantile MAE (MAE.,). We observe that MCNF was sensitive to outliers when the effect size was close to zero, which likely results from biasing of the maximum likelihood estimator. The effect size relates to the degree of correlation between the feature variables and the response variable. We sought to mitigate this impact, given that low effect sizes (as well as sporadic large outlier values) frequently characterize data sets in many disciplines.
To test the impact of different 7 values in Equation synthesize data (Section B) by modulating the slope (b) of the governing generative process to generate multiple sets of simulated data (Figure 8). where b E {0, 0.1, 0.2, 0.3, 1, 2, As the scale of the outliers relative the rest of the data also changed with the slope, we modulated the y parameter to keep the outliers in the same approximate range. The outliers proportion was 1%. For each level of slope, we predicted from the MCNF method using the following T values 7 E (102, 2- 102 3- 102, 103, 2- 10 5 103, 10 10 109). The data distributions are shown in Figure
with respect to changing effect size, we
Generally, the marginal coverage increases with the value of T as the effect of the temperature scaling assigns equal importance to outliers and non-outliers. However, this comes at the cost of overly conservative prediction intervals and less adaptivity to the uncertainty in the data. No single value for F can be considered universally optimal, as shown in Figure g This hyperparameter should be fine-tuned, potentially using a calibration set. When the ratio of the outlier magnitude over the effect size becomes smaller (larger values of the slope in this case) the importance of the former on the regular maximum likelihood procedure decreases, and values of can be smaller.
23

---

epochs=100, 0n==[100,200,500] 5occo=50
boxton, housing
abatons
concnete
E
82
80
80-
00
00-
43-
40-
20-
20-
0.55
0.50
0.64
0.54
(1 A411
0.40
0.40
0.90
0.52
0.58
0.52
0.42
0.50
0.42 -
0.00
0.075
0.955
0.102
1790 predee (trees
0.070
0.100
0.088
0.085
0.030
0.003
3.000
0.030
0.355
0.000
0.034
90
500
100
200
500
Figure 6: Evaluation of MCNF on Boston Housing, Abalone, and Concrete datasets for different ONF values (100, 200, and 500).
epachs=100, ny,=500, (soco=5501100,00000
boston_housing
abatone
concrete
100
102
103
1
mawwrggh (teen) 106)
80
60-
40-
0-
0-
0.55
0.64
0.58
0.54
0.48
0.2
0.53
0.52
0.51
0.60
0.50
0.50
0.42
0.50
0.0755
0.100
0.000
Srakke (nteen 1863
0.0700
0.0075
0.102
0.090 -
0.000
0.100
00005
20000
0.000
0.00
0.005
0.060
0.675
8
000
50
RACE
Figure 7: Evaluation of MCNF on Boston Housing, Abalone, and Concrete datasets for different DMCD values (50, 100, and 500).
24

---

Table 4: Evaluation of the impact of outliers for different 7, b, and value combinations MCNF
coverage (C), median MAE (MAE), and Quantile MAE (MAE.)
C
MAE
MAE,
100
0
25
0,8100
0.2061
1.3039
100
0.7883
0.1845
0,9799
100
25
0.7900
0.2307
0,9087
100
0.3
25
0.8200
0.1271
0.7115
100
2
150
0,9117
0.2258
0.0474
200
25
0.8067
0.2887
1.4055
200
0.1
25
0.8467
0.3229
1 1.2045
200
0.3
25
0.8800
0.1594
0.8330
200
150
0,9383
0.0625
0.2758
300
25
0.8467
0.2001
1.3449
300
00.1
25
0.8467
0.3229
1.2045
300
0.3
25
0,8800
0.1594
0.8330
300
0,9283
0.1070
0,4160
300
150
0,9283
0.0570
0,2793
0.9067
0.6610
1,9665
1000
0.1
25
0.8867
0.1307
1,8857
1000
0.9017
0.1413
0.9706
1000
25
0,9133
0.1875
0.8647
1000
100
0,9283
0.0978
0. 0,3980
1000
0,9367
0,9000
0.2752
2000
0.1
25
0,9400
0.1788
1,4421
2000
0.2
25
0,9233
0.30
1.1068
2000
25
0,9183
0.1163
0.8374
2000
100
0,9350
0.0807
0.4679
0.0576
5000
25
0.9083
0.8049
I 1.0066
5000
25
0,9467
0.1651
1,2981
5000
0.2
25
0,9300
0.2350
1.1888
5000
100
0,9200
0.0957
0.4036
5000
0,9500
0.0439
10000
0
25
0,9200
0.3249
3,4255
0,9467
10000
0.2
25
0,9133
0.1299
1.0917
0,9300
0.10
10000
100
0,9417
0.11113
0.5516
10000
150
0.9400
0.0632
0.2897
50000
0
25
0,9800
0.2216
6.4317
50000
0.2
25
0.9667
0.1667
1,8143
50000
0.3
25
0.9433
0.1299
1 1,2884
50000
0,9417
0.11113
0.5516
50000
150
0,9400
0.0599
0.3046
100000
0
25
0.9867
0.3689
7,2465
100000
0.1
25
0.9867
0.2644
6.5211
100000
25
0.9800
0.2321
2.9301
100000
0.3
25
0,9650
0.1731
1.8576
100000
150
0,9833
0.0654
0.6146
25

---

1,000
0.00
0.975
0.20
0.30
0.950
coverage
0.850
0.825
0.800
10'
10
10
(og
Figure 8: MCNF marginal coverage with respect to for different slope b values.
26


Reviews:
[{"review_id": "PFkfRduxxK", "reviewer": "Reviewer_8wsk", "summary": "This paper introduces Monte Carlo Normalizing Flow (MCNF), a novel post-hoc method for uncertainty quantification (UQ) in deep regression models. The core idea is to improve upon the uncertainty estimates generated by Monte Carlo Dropout (MCD) by using them to \"contextualize\" a normalizing flow (NF). This NF then models the distribution of the prediction errors. The method aims to provide not just prediction intervals but the full predictive posterior distribution, which can capture complex features like asymmetry and multimodality.", "questions": "The paper mentions that using the first two moments is a reasonable approximation under a normality assumption for the MCD distribution. How does MCNF's performance change if this assumption is strongly violated, for instance in a clearly bimodal case?\n\nThe training loss includes a temperature-scaled softmax to weigh observations, mitigating the effect of outliers. In the experiments, τ is set to a very large value (1e10), effectively giving equal weight to all observations. Why was this choice made?\n\nHow does MCNF compare to other methods that also use normalizing flows for uncertainty quantification, but perhaps in a different manner ?\n\nA strength of conformal prediction is that it enjoys coverage guarantees for the queried alpha. Does MCNF enjoy some theoretical properties (even under ideal assumptions) ?", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "pros:\n- The proposed MCNF method presents a clever and novel way to combine existing techniques (MCD and NFs)\n- The proposed approach can be deployed on already trained deep learning models with dropout layers without the high computational cost of retraining.\n\ncons:\n- The approach relies on MCD and the MCD context provided to the normalizing flow might be misleading\n- The approach introduces additional computational steps at inference time (operations of the normalizing flow).\n- The results are difficult to read as several (antagonistic) criteria are necessary to compare the methods. It is hard to know which method delivers the best trade-off", "was_revised": true, "final_justification": "I thank the authors for the detailed answers provided to my questions. Based on the additional empirical evidence and arguments they gave, I raised my score by one unit. \n\nAs the authors indicate that the paper is focused on post hoc uncertainty quantification, I recommend this position to appear in the very title of the paper.", "rebuttal": "Dear reviewer 8wsk,\n\nThank you very much for your valuable review and your questions! Please find answers to your questions below:\n\n> The paper mentions that using the first two moments is a reasonable approximation under a normality assumption for the MCD distribution. How does MCNF's performance change if this assumption is strongly violated, for instance in a clearly bimodal case?\n\nA: In MCNF, the sample mean and log variance of the samples drawn from the MCD distribution are used to build the Normalizing Flow’s context vector only (Equation 5), ultimately affecting the assessment of the likelihood as given by Equation. 6. Due to the introduction of the prediction error $\\delta_{n,k} = y_{n} - y_{,n,k\\text{MCD}}$ for every n-th observation (input) and k-th estimate drawn from the MCD distribution, the Normalizing Flow (NF) model is learning from the actual MCD distribution which reflects the epistemic uncertainty. Accordingly, even if the ground truth was a completely deterministic variable (i.e., no aleatoric uncertainty), and the distribution of the epistemic uncertainty was bimodal, the NF would be aware of that bimodality by virtue of the samples drawn for the prediction error. Summarizing the MCD distribution by using the sample mean and log variance is made to inform the NF head of the MCNF method about the approximate location and spread of the prior distribution to assist in the reconstruction of the actual distribution of the prediction error ($\\delta$), which is learnt from the data.\n\n\n> The training loss includes a temperature-scaled softmax to weigh observations, mitigating the effect of outliers. In the experiments, τ is set to a very large value (1e10), effectively giving equal weight to all observations. Why was this choice made?\n\nA: The proposed loss function was designed so that it falls back to the original Negative Log-Likelihood (NLL) for sufficiently large values of the $\\tau$ parameter. For datasets containing no outliers, there is no reason to penalize observations differently, and this principle guided our approach to deal with all of the datasets taken from the UCI Benchmark. Romano’s synthetic dataset (Appendix B) was used as a counterfactual example to demonstrate the benefits of this modeling framework in mitigating the negative impact of the presence of outliers on the maximum likelihood approach. In these cases, $\\tau$ also moderates the importance of highly informative priors, as is the case for the proposed MCNF. In Section C.2.2. of the Appendix (supplementary material), we argue that $\\tau$ should be set by using a hold-out calibration set and discuss the effect of this hyperparameter on the performance metrics (coverage, quantile MAE, and prediction interval sizes) by analyzing a set of different $\\tau$ values.\n\n\n> How does MCNF compare to other methods that also use normalizing flows for uncertainty quantification, but perhaps in a different manner ?\n\nA: The existing literature on the use of Normalizing Flows for uncertainty quantification can be grouped into two main categories:\n\n* full Bayesian and variational approaches [1-2] that extend their formalism using normalizing flows to model priors or, alternatively, as the variational family of distributions to enrich the modelling of the posterior distribution over the parameters, respectively. The full Bayesian approach often resorts to some modelling assumptions to preserve tractability, which may limit the expressiveness of the modelling framework. This is the case of [1] where authors hypothesize a Gaussian distribution of the prediction variable uncertainty. The variational approach deduced in [2] has the inherent limitations of Variational Autoencoders (VAEs) in approximating the true density.\n* the second category include approaches using Normalizing Flows as the probabilistic deep learning framework that enable describing the uncertainty in the form of a predictive distribution, $p(y|\\mathbf{x})$ [3-5], characterizing the distribution of uncertain parameters on physical simulations [4], or using normalizing flows to accurately model the In-Distribution data, enabling Out-of-Distribution detection based on the estimated likelihood.\n\nThese methods were left out of the related work as we mainly focused on methods that could be applied post-hoc, as this is one of the key requirements considered to propose MCNF. None of the reviewed methods enable this feature. Approach [5] shares with our method the use of Normalizing Flows to describe reconstruction (prediction) errors; however, the method is not post-hoc as its architecture is not independent from the data structure. It is also important to stress that propagating epistemic uncertainty through the prediction error is crucial to improve the calibration of the uncertainty quantification. To support this latter argument, we refer the reviewer to the table below, in which the results of using NF to model prediction errors without propagating the epistemic uncertainty are shown. It is noteworthy that when no MCD is used to resample prediction errors, the NF becomes significantly less calibrated in most of the datasets included in the UCI Benchmark. The Table below includes results from the MCNF as proposed in Equation 3 and the results for Normalizing Flows not using prediction error resampling, demonstrating that MCNF yields significantly higher coverage than Normalizing Flows without prediction error resampling.\n\n|Dataset|**method**| **mae_median_mu** | **mae_median_sd**| **mae_quant_mu**| **mae_quant_sd**| **coverage_mu**| **coverage_sd**|\n|---|---|---|---|---|---|---|---|\n|Abalone| mcnf| 0.099| 0.007| 0.499| 0.017| 0.886| 0.017|\n|Abalone| nf| 0.098| 0.005| 0.494| 0.039| 0.874| 0.023|\n|Boston| mcnf| 0.078| 0.009| 0.428| 0.037| 0.904| 0.043|\n|Boston| nf| 0.075| 0.008| 0.300| 0.046| **0.784**| 0.081|\n|Concrete| mcnf| 0.085| 0.012| 0.472| 0.032| 0.920| 0.021|\n|Concrete| nf| 0.084| 0.008| 0.352| 0.020| **0.814**| 0.038|\n|Protein| mcnf| 0.202| 0.009| 1.172| 0.059| 0.927| 0.006|\n|Protein| nf| 0.186| 0.008| 1.048| 0.088| **0.887**| 0.018|\n|Romano-OG| mcnf| 0.406| 0.206| 2.977| 0.395| 0.926| 0.014|\n|Romano-OG| nf| 0.496| 0.154| 2.886| 0.254| 0.912| 0.019|\n|Wave| mcnf| 0.002| 0.001| 0.013| 0.001| 0.938| 0.030|\n|Wave| nf| 0.002| 0.001| 0.006| 0.001| **0.807**| 0.149|\n|Super| mcnf| 0.103| 0.005| 0.566| 0.023| 0.912| 0.009|\n|Super| nf| 0.096| 0.004| 0.462| 0.022| **0.866**| 0.011|\n\n\n[1] B. Zhang, W. Sui, Z. Huang, M. Li, and M. Qi, \"Normalizing flow based uncertainty estimation for deep regression analysis,\" Neurocomputing, vol. 585, p. 127645, 2024.\n\n[2] R. Selvan, F. Faye, J. Middleton, and A. Pai, \"Uncertainty quantification in medical image segmentation with normalizing flows,\" in Proc. Int. Workshop on Machine Learning in Medical Imaging (MLMI), MICCAI, 2020.\n\n[3] R. Orozco, M. Louboutin, A. Siahkoohi, G. Rizzuti, T. van Leeuwen, and F. J. Herrmann, \"Amortized normalizing flows for transcranial ultrasound with uncertainty quantification,\" in Proc. Med. Imaging with Deep Learning (MIDL), vol. 227, Jul. 10–12, 2024, pp. 332–349.\n\n[4] A. Dasgupta and Z. W. Di, “Uncertainty quantification for ptychography using normalizing flows,” arXiv preprint arXiv:2111.00745, 2021.\n\n[5] S. Benaïchouche, G. Morel, F. Rousseau, and R. Fablet, “Divergence-free continuous normalizing flows for uncertainty quantification,” preprint, Oct. 2022.\n\n\n> A strength of conformal prediction is that it enjoys coverage guarantees for the queried alpha. Does MCNF enjoy some theoretical properties (even under ideal assumptions)?\n\nA: Currently, we cannot make any claims about MCNF inheriting properties from the conformal prediction theoretical framework. However, the nature of the maximum likelihood principle supporting our MCNF method favours (although does not guarantee) the achievement of calibrated coverages, as maximizing the likelihood of the ground truth observations also maximizes the probability of building prediction intervals containing the latter. The experimental work carried out (Table 1, Figure 3) provides strong evidence that MCNF provides comparable miscalibration rates to those provided by the baseline methods based on conformal prediction (e.g. CQR, MCCP)."}, {"review_id": "M3cS01m3JW", "reviewer": "Reviewer_fJpF", "summary": "The paper proposes a normalizing-flow-based post-hoc uncertainty quantification method that works on top of an already trained monte-carlo dropout model. The prediction mean and variance (or rather log-variance) of the output of the MCD model along with certain hidden layers of the MCD model are used to to form the condition (or context) of the NF model. The NF model itself represents the density of the residual \\delta around the mean of the base model prediction. The method is explained in detail, and then empirical evaluation is provided.", "questions": "- The original monte-carlo dropout method outputs an aleatoric mean and variance, and the dropout part is only used to quantify epistemic uncertainty (ie, uncertainty of the mean and variance parameters themselves). Is this considered in the design of the NF method? Is it considered during MCD evaluation as a baseline?\n- The mcd samples are reduced to mean and variance before feeding them to NF model (mind you, if I understand correctly, this is the mean and variance of the \\mu parameter of the gaussian aleatoric distribution predicted by MCD). This appears to lose any shape information that the base model might predict for the mean (and any information predicted by the model for the aleatoric variance). Can you comment on this?\n- Is there a reason that no NLL evaluation is done on the datasets, as is standard? Similarly, is there a reason that the quasi-standard UCI regression uncertainty benchmark is not presented along with the evaluations currently in the paper? (As is done e.g. in the original monte-carlo dropout paper and a large number of papers since).\n- In the beginning, e.g. eq 3, we are made to believe that the normalizing flow transforms the output distribution of the base model into the aleatoric posterior of y through monte-carlo sampling. Yet after a while it turns out that in fact the output distribution of the base model is summarized into mean/var as context for the NF, and the actual base-distribution being transformed is in fact a certain z_{n,k} (e.g. line 180), I assume a gaussian, as is usual in normalizing flows. Does this not mean that the initial description is quite misleading? Did I miss something here?", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Overall, this is a quite strong paper. Strengths:\n- Innovative idea, providing a method for distribution-free uncertainty quantification\n- Well structured paper, (mostly) clear descriptions\n- Sufficient evaluation\n- Proper source-code provided\n\nWeaknesses:\n- Standard evaluation not included (UCI Benchmark)\n- Possible confusion around around MCD predictive distributions (see questions)\n- Unclear theoretical predictive power w.r.t. being distribution free (see questions)\n- Confusing framing of the base-distribution of the normalizing flow (see questions)\n\nWeak reject for now, but I expect to increase the score if the questions are answered in a convincing way.", "was_revised": true, "final_justification": "The authors managed to address most of my concerns in their rebuttal. However, I still believe that one of the baselines (MCD) is presented unfairly in the experiments section: if my understanding is correct, the constant term for aleatoric variance is not included in the calculation, severely handicapping the method. This is not a critical issue, as it is only one baseline of 3, but it does weaken the paper.", "rebuttal": "Dear reviewer fJpF,\n\nThank you very much for your valuable review and your questions! Please find answers to your questions below:\n\n> The original monte-carlo dropout method outputs an aleatoric mean and variance, and the dropout part is only used to quantify epistemic uncertainty (ie, uncertainty of the mean and variance parameters themselves). Is this considered in the design of the NF method? Is it considered during MCD evaluation as a baseline?\n\nA: Monte Carlo Dropout (MCD) approximates the epistemic uncertainty of a model's estimates of the predicted variable **y**. On top of this, the method optionally accepts augmenting the number of outputs of the predictive model to include estimates of the aleatoric mean and variance, in which case a Gaussian assumption of the aleatoric uncertainty is made. Therefore, MCD allows obtaining a distributional uncertainty over the Gaussian assumption for this specific implementation. However, MCD can also be applied to approximate the distribution of the epistemic uncertainty of the predicted variable **y** itself. This is the framework that we have considered in this work. For MCFN, no distributional assumption is made for the total uncertainty (epistemic + aleatoric). Epistemic uncertainty is introduced via MCD sampling, while aleatoric uncertainty is inherent to the data used for training. Both of them are propagated together through the prediction error, $\\delta_{n,k} = y_{n} - y_{n,k,\\text{MCD}}$, which is learnt by the Normalizing Flow (NF) head underpinning MCNF.\n\n\n> The mcd samples are reduced to mean and variance before feeding them to NF model (mind you, if I understand correctly, this is the mean and variance of the \\mu parameter of the gaussian aleatoric distribution predicted by MCD). This appears to lose any shape information that the base model might predict for the mean (and any information predicted by the model for the aleatoric variance). Can you comment on this?\n\nA: Following the discussion on the previous question, the prediction error, $\\delta_{n,k}$, propagates the actual distributional information of MCD applied to the base model (epistemic uncertainty) as well as that of the aleatoric uncertainty (intrinsic to the training data).\n\nMCD prior estimates are summarised as the reviewer states, but with the sole purpose of building the context vector for the NF head of the MCNF method. This contextualization aims at providing location and spread information of the prior distribution. During training, different realizations of the prediction error are passed to the model by drawing samples from the MCD distribution as described in Algorithm 2 in Appendix A, describing the MCNF training.\n\n\n> Is there a reason that no NLL evaluation is done on the datasets, as is standard? Similarly, is there a reason that the quasi-standard UCI regression uncertainty benchmark is not presented along with the evaluations currently in the paper? (As is done e.g. in the original monte-carlo dropout paper and a large number of papers since).\n\nA: Concerning showing the NLL values, we did not consider incorporating these as, in our opinion, NLL is not an easily interpretable quantity, as its value depends on the data underlying distribution. Consequently, its magnitude cannot be used to make comparisons across datasets. This becomes even more apparent for continuous distributions as the NLL has no lower bound since densities may be larger than one, leading to negative NLLs. Besides, we cannot report NLL for all of the baselines included in our comparison. Should the reviewer believe that adding this information would add further clarity to our paper, we can easily incorporate these values for all of the datasets.\n\nConcerning the experimental set used, we used the same datasets employed in the state-of-the-art methods evaluated in our papers, i.e., CQR [1], MCCP [2] and MCD [3]. More specifically, \n\n* the Boston Housing dataset is used in [2] and [3]\n* the Concrete dataset is used in [1-3]\n* the Concrete dataset is used in [1-3]\n* the Abalone dataset  is used in [3] \n* the Protein Structure dataset is used in [2] and [3] \n* the Wave and Superconductivity datasets are used in [2]\n* the Romano-Original is a synthetic dataset produced in [1]\n\nAs reported in the Evaluation section - Benchmarks, the datasets mentioned above have been retrieved from the UCI machine learning repository (https://archive.ics.uci.edu)\n\nFurthermore, we extended the Romano-Original dataset that has univariate predictor samples and few large outliers with support for multimodal distributions. The distribution that characterizes the uncertainty incorporates heteroskedasticity, which is a particularly relevant validation case to test whether MCNF adapts correctly to the local distribution of the data.\n\nAlso, we demonstrated the applicability and benefits of MCNF by comparing it against the other methods using a GNN as the predictive model and the solubility dataset [4] that includes solubility data of 829 drug-like molecules.\n\nTaken together, our empirical evaluation adopts the established practice in the area, using the benchmark datasets from the UCI repository and conforming to the approach adopted in [1-3]. \n\n[1] Romano, Y., Patterson, E., & Candes, E. (2019). Conformalized quantile regression. Advances in neural information processing systems, 32.\n\n[2] Bethell, D., Gerasimou, S., & Calinescu, R. (2024, March). Robust uncertainty quantification using conformalised Monte Carlo prediction. In Proceedings of the AAAI Conference on Artificial Intelligence (Vol. 38, No. 19, pp. 20939-20948).\n\n[3] Gal, Y., & Ghahramani, Z. (2016, June). Dropout as a bayesian approximation: Representing model uncertainty in deep learning. In international conference on machine learning (pp. 1050-1059). PMLR.\n\n[4] Mario Lovri´c, Kristina Pavlovi´c, Petar Žuvela, Adrian Spataru, Bono Luˇci´c, Roman Kern, and Ming Wah Wong. Machine learning in prediction of intrinsic aqueous solubility of drug-like compounds: Generalization, complexity, or predictive ability? Journal of Chemometrics, 35(7-8), jul 2021\n\n\n> In the beginning, e.g. eq 3, we are made to believe that the normalizing flow transforms the output distribution of the base model into the aleatoric posterior of y through monte-carlo sampling. Yet after a while it turns out that in fact the output distribution of the base model is summarized into mean/var as context for the NF, and the actual base-distribution being transformed is in fact a certain z_{n,k} (e.g. line 180), I assume a gaussian, as is usual in normalizing flows. Does this not mean that the initial description is quite misleading? Did I miss something here?\n\nA: In MCNF, the MCD prior (which is a source of the epistemic uncertainty) is propagated through the prediction error $\\delta$ and used to inform the NF head of the method about the location and spread of this epistemic uncertainty through the context, c, as argued in previous answers; therefore Equation 3 and, particularly, Equation 6 hold. On the other hand, z_{n,k} represents the latent representation of the NF, for which we chose a Gaussian distribution with trainable parameters. The key point is that $z_{n,k} \\neq y_{n,k,\\text{MCD}}$, entailing that the MCD prior should not be assumed to be the NF base distribution, which can be selected arbitrarily. We are sorry for the confusion that the writing may have caused and will clarify this aspect in the camera-ready version of our paper.\n\n\n> line 138 typo: sampled vs sample\n\nA: Thank you for carefully reviewing the manuscript and reporting this typo. We will correct it in the final version of our manuscript."}, {"review_id": "2oTgSVELNI", "reviewer": "Reviewer_eM1j", "summary": "The paper presents a posthoc UQ method leveraging upon Monte Carlo Dropout (MCD) and normalizing flows.  Specifically, the normalizing flow model is trained to learn a distribution for the output variable $y$ given a feature vector derived somewhere in the base prediction model and summary statistics of the mean and log-variance of the output of the base model under MCD. The experiments somewhat demonstrate that the error quantiles are better calibrated of the new method against other state of the art UQ methods for regression problems.", "questions": "Can the authors explain why the interval length $\\Delta$ matters and why it is not intrinsic with respect to the CDF of possible values of $y$?\n\nWhat are the values for $n_{MCD}$ and $n_{NF}$ used in the paper?  Have the authors investigated the robustness of the MCNF with respect to these parameters?\n\nIn Figure 3, I get why the interval spread should be larger for the underfitted (less trained) model, and this is prominent in the strong baselines. However, the increase in the spread for MCNF is less prominent.  Why would this be desirable?  I would think that epistemic uncertainty increases in the underfitted models, and thus $\\Delta$ should jump much higher.", "limitations": "The paper mentions the computational complexity as a possible limitation, especially for larger prediction models, in the concluding section. Also, the concluding section mentions adapting the method for classification as future work. The paper could discuss the computational complexity for inference in the experimental section with respect to the other UQ baseline methods.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 4, "strengths_and_weaknesses": "The proposed UQ methods makes sense and seems effective for capturing arbitrary probability distributions that can be multimodal with complex structure. The essence of the idea is that one can learn the distribution of possible output from a context variable consisting of features from a base forward pass model along with statistics of the output of the base model under MCD. This idea is intriguing and appears novel.\n\nThe experiments seem to compare with a sufficient number of baseline state-of-the-art methods. I appreciate that the paper considers both DQR and GNNs as base prediction models. \n\nI do have some concerns with the experiments. First, I am not clear why the interval length should matter.  The interval length is dictated by the unknown ground truth distribution of the output $y$ given knowledge of the input raw features $x$, and thus $\\Delta$ may need to large or small in light of the spread of this ground truth distribution.\n\nWhile it is great to measure the coverage for a desired coverage of 90%, it think it would be more useful to compute an expected absolute calibration error by considering the absolute difference of the coverage and the desired coverage of x% for x ranging from 0 to 100.\n\nI like Figure 2 as a means to show the need to accommodate  complex multimodal distributions in UQ methods. \n\nThe paper does not discuss the computational cost for inference via sampling for MCNF relative to the computational cost of the baseline methods.\n\nAs a reader, it would be easier for me to understand $\\log \\sigma^2$ as the log-variance rather than $\\log s^2$ as given in (5).\n\nIn (6), it is not clear what depends on the Monte Carlo index \"$i$\" for the summation. Somehow, it is hidden in $\\delta$, which is only revealed in Algorithm 1.", "was_revised": true, "final_justification": "After considering the author rebuttal and the other reviews, I happy with my recommendation of borderline accept.\n\nThe paper is an interesting post-training uncertainty quantification method based on MCD and NF. It is not of low computational complexity, but it is effective in capturing calibrated confidence intervals and exhibiting high accuracy. The concern of one reviewer about the use of the MCD baseline is reasonable. Overall, the paper has some novelty and mostly sufficient experiments with one flaw. Borderline accept seem appropriate. Personally, I am happy to see this paper included in the conference program if there is room.", "rebuttal": "Dear reviewer eM1j,\n\nThank you very much for your valuable review and your questions! Please find answers to your questions below:\n\n> As a reader, it would be easier for me to understand as the log-variance rather than as given in (5). In (6), it is not clear what depends on the Monte Carlo index \" \" for the summation. Somehow, it is hidden in, which is only revealed in Algorithm 1.\n\nA: Thank you for raising this notation-related point. With respect to the log-variance, we preferred $\\log s^2$ over $\\log \\sigma^2$ to denote that the statistic used to build the context is the sample variance and not the true (population) variance. Regarding Equation 6, we agree with the reviewer’s observation and will fix it accordingly to make the index explicit inside the summation operator.\n\n\n> Can the authors explain why the interval length Δ matters and why it is not intrinsic with respect to the CDF of possible values of y ?\n\nA: As the reviewer suggests, $\\Delta$ is intrinsic with respect to the underlying distribution of the predicted variable y. However, different distribution spreads may account for the same (or very) close marginal coverage. Therefore, for the same level of coverage level (or coverage miscalibration), narrower predictive distributions (MCD, MCNF) and intervals (for interval-based approaches, e.g. DQR, CQR or MCCP) are preferred. Additionally, the spread of predictive distributions and the width of the prediction intervals should adapt conveniently to the actual uncertainty (as discussed elsewhere, [2107.07511] A Gentle Introduction to Conformal Prediction and Distribution-Free Uncertainty Quantification), avoiding overly conservative assessments. Since the experiments carried out showed that MCNF coverage miscalibration is comparable to that of the selected baselines, $\\Delta$ becomes a meaningful metric for comparing the performance of all the considered methods. We also discuss the adaptivity, showing the prediction interval width variability (using sample quantiles) in Table 3 in Appendix C.2.1.\n\n\n> What are the values for n_MCF and n_NF used in the paper? Have the authors investigated the robustness of the MCNF with respect to these parameters?\n\nA: We have indeed investigated the robustness of MCNF using n_NF ∈ {100, 200, 500} and n_MCD ∈ {50, 100, 150} for the Boston Housing, Abalone, and Concrete datasets in Appendix C.2.1. Our results demonstrate that MCNF perform similarly in terms of the coverage metrics, and the difference in the confidence interval and the median MAE is negligible for all n_MCD values. Similar observations have been derived for the various n_NF values evaluated, with small differences observed in terms of slightly higher coverage as n_NF increases at the cost of slightly higher confidence intervals. These results demonstrate the overall robustness of MCNF with respect to its n_MCD and n_NF hyperparameters, indicating that MCNF adopters can comfortably find appropriate values for their domains and datasets.\n\n\n> In Figure 3, I get why the interval spread should be larger for the underfitted (less trained) model, and this is prominent in the strong baselines. However, the increase in the spread for MCNF is less prominent. Why would this be desirable? I would think that epistemic uncertainty increases in the underfitted models, and thus should jump much higher.\n\nA: Epistemic uncertainty indeed increases in underfitted models and that reflects on the larger widths of the prediction intervals for all of the UQ methods, as evidenced by Figure 3. The key factor that explains differences in how much the interval width increases for MCNF as compared to the remaining baselines lies in the fact that the baselines account not only for the data-intrinsic aleatoric uncertainty and the increase in the epistemic uncertainty, but also for the predictive model (DNN / GNN) bias. Conformal prediction methods (CQR and MCCP) need to significantly correct the uncalibrated intervals to guarantee the design marginal coverage while accounting for this bias. DQR and MCQR also require much larger intervals for the underfitted model. In contrast, MCNF trains the NF head of the method over the predictions of the underfitted predictive model, which recenters the uncertainty distribution, thus decoupling and removing the prediction bias from the aleatoric and epistemic sources of uncertainty. The first doesn’t change while the second is still propagated as shown in Figure 3.\n\n\n> The paper mentions the computational complexity as a possible limitation, especially for larger prediction models, in the concluding section. Also, the concluding section mentions adapting the method for classification as future work. The paper could discuss the computational complexity for inference in the experimental section with respect to the other UQ baseline methods.\n\nA: The computational complexity at inference time incurred by MCNF is linear with respect to n_MCD +n_NF. More specifically, this involves running n_MCD forward passes to build the context vector as per Equation 5 and then drawing n_NF samples to produce the prediction output y and its density. As discussed above, MCNF is robust to the n_MCD and n_NF values. We will add these clarifications in the final version of our paper."}, {"review_id": "8I7VbYLVtC", "reviewer": "Reviewer_onw7", "summary": "The Authors propose an uncertainty quantification technique for regression models based on neural networks. This technique makes use of Montecarlo dropout to get a prior estimate of the epistemic uncertainty of the model, and combines it post-hoc with normalizing flows to get a distribution of the prediction error (includung aleatoric uncertainty).\n\nAs reflected in my confidence score, my evaluation capabilities for this paper are limited as UQ is not my main area of research.", "questions": "- In the experimental evaluation, you report averaged results over 20 independent runs. Initialization and the stochastic gradient dynamics are a source of epistemic uncertainty. Do you have estimates on the magnitude of this uncertainty compared to the one introduced by MCD? Does this question make sense at all?", "limitations": "Not discussed", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "### Strengths\n- Thorough experimental evaluation addressing multiple metrics and datasets. Among the experimental evaluations, I also appreciated a lot the study on the impact of the pre-trained predictive model. \n- The paper is generally well written, although many variables are introduced with proper explanation (see weaknesses).\n\n#### Weaknesses\n- Many variables are introduced without proper explanation. Among these there are: $\\mathcal{D}$ on line 97, $\\Xi, \\phi$ on line 132, $\\theta, \\psi$ on line 124\n- Lacking details on the computational complexity of the method. The inference task, in particular, requires first performing Monte Carlo dropout sampling, and then a pass on the normalizing flow model.", "was_revised": true, "final_justification": "I am satisfied with the rebuttal, and I will keep my positive score.", "rebuttal": "Dear reviewer onw7,\n\nThank you very much for your valuable review and your questions! Please find answers to your questions below:\n\n> Many variables are introduced without proper explanation. Among these there are: $\\mathcal{D}$ on line 97, $\\Xi, \\phi$ on line 132, $\\theta, \\psi$ on line 124\n\nA: We appreciate the careful reading of the manuscript by the reviewer and the highlighting of some examples of undefined symbols. Below, symbols missing an appropriate description are defined:\n\n* $\\mathcal{D}$: is the set of observations $\\{\\mathbf{x}_n, y_n\\}_{n=1}^{N_{|\\mathcal{D|}} available for training.\n\n* $\\theta$, $\\psi$: these denote de parameters for the series of transformations making up the Normalizing Flow and the parameters for the base Gaussian distribution characterizing the NF latent space, respectively\n\n* $\\Xi$: this symbol denotes the parameters of the predictive model, $\\phi(\\mathbf{x};\\Xi)$.\n\nWe have gone through the paper and found an additional erratum in line 230 when defining the median prediction interval width:\n\nCurrent form:\n\n* $\\tilde{\\Delta}(X,Y) = median_{i=1}^n (q_{1-\\alpha}(x_i),q_{\\alpha}(x_i))$\n\nFixed form:\n\n* $\\tilde{\\Delta}(X,Y) = median_{i=1}^n (q_{1-\\alpha}(x_i) - q_{\\alpha}(x_i))$\n\nAll these changes will be added to the camera ready version of the manuscript, if accepted.\n\n\n> Lacking details on the computational complexity of the method. The inference task, in particular, requires first performing Monte Carlo dropout sampling, and then a pass on the normalizing flow model.\n\nA: The computational complexity at inference time incurred by MCNF is linear with respect to n_MCD +n_NF. More specifically, this involves running n_MCD forward passes to build the context vector as per Equation 5 and then drawing n_NF samples to produce the prediction output y and its density. \n\nWe have also investigated the robustness of MCNF using n_NF ∈ {100, 200, 500} and n_MCD ∈ {50, 100, 150} for the Boston Housing, Abalone, and Concrete datasets in Appendix C.2.1. Our results demonstrate that MCNF performs similarly in terms of the coverage metrics, and the difference in the confidence interval and the median MAE is negligible for all n_MCD values. Similar observations have been derived for the various n_NF values evaluated, with small differences observed in terms of slightly higher coverage as n_NF increases at the cost of slightly higher confidence intervals. These results demonstrate the overall robustness of MCNF with respect to its n_MCD and n_NF hyperparameters, indicating that MCNF adopters can comfortably find appropriate values for their domains and datasets.\n\nWe analysed the computational complexity of MCNF and the other UQ methods to assess the memory and computational time  consumed for the evaluated datasets.  Our findings indicate that MCNF runs significantly faster than MCCP, and in some datasets (abalone, concrete) similar to baselines CQR and DQR. \n\n\n\n> In the experimental evaluation, you report averaged results over 20 independent runs. Initialization and the stochastic gradient dynamics are a source of epistemic uncertainty. Do you have estimates on the magnitude of this uncertainty compared to the one introduced by MCD? Does this question make sense at all?\n\nA: Thank you for this interesting question, which definitely makes sense. As reported in the Evaluation-Performance Metrics section, each independent run includes sources of epistemic uncertainty unrelated to the number of parameter (i.e. models) sets of the base predictive model that could explain the data, which is captured by MCD. This is primarily due to the resulting data partition model and the effect of stochastic gradient dynamics.  Then, the 20 independent runs average out the impact of the sources of epistemic uncertainty.\n\nIn Figure 3, we compare the performance of MCNF using a weak and a strong base predictive model, thus focusing primarily on the effect of stochastic gradient dynamics due to MCNF training. These results show the impact that the quality of the base predictive model has on MCNF and the corresponding epistemic uncertainty arising from MCD, which is smaller than the corresponding impact in the other methods we evaluated (e.g., MCD, MCCP, CQR). \n\nPerforming a fine-grained analysis to further disentangle the uncertainty is definitely an interesting direction for future work in order to establish further the MCNF capabilities in working with base predictive models and datasets with diverse characteristics."}]

Author rebuttal:

Meta-review: This paper introduces a novel post-hoc method for uncertainty quantification (UQ) in deep regression models. To improve uncertainty estimation, the authors propose leveraging both the features and outputs of the base model (MCD) as contexts to help refine and scale the predictive distribution modeled by a Normalizing Flow. Experiments on several benchmark datasets across multiple metrics demonstrate that the proposed method outperforms baseline approaches.

The paper received four borderline accepts. Reviewers generally find the idea of contextualized Normalizing Flows innovative and the experimental results convincing. However, concerns were raised about the high inference cost, the effectiveness of the contextual information, lack of clarity in some technical details, and potentially unfair comparison with one of the baselines. The authors’ rebuttal is extensive and detailed, providing additional empirical evidence that alleviates most major concerns.
Decision: Accept (poster)