Title: Uncertainty Quantification for Deep Regression using Contextualised Normalizing Flows
Venue: NeurIPS 2025
Accepted: yes
Keywords: uncertainty quantification

safe deep learning
Abstract: Quantifying uncertainty in deep regression models is important both for understanding the confidence of the model and for safe decision-making in high-risk domains. Existing approaches that yield prediction intervals overlook distributional information, neglecting the effect of multimodal or asymmetric distributions on decision-making. Similarly, full or approximated Bayesian methods, while yielding the predictive posterior density, demand major modifications to the model architecture and retraining. We introduce MCNF, a novel post hoc uncertainty quantification method that produces both prediction intervals and the full conditioned predictive distribution. MCNF operates on top of the underlying trained predictive model; thus, no predictive model retraining is needed. We provide experimental evidence that the MCNF-based uncertainty estimate is well calibrated, is competitive with state-of-the-art uncertainty quantification methods, and provides richer information for downstream decision-making tasks

Uncertainty Quantification for Deep Regression using Contextualised Normalizing Flows

Abstract

Quantifying uncertainty in deep regression models is important both for understanding the confidence of the model and for safe decision-making in high-risk domains. Existing approaches that yield prediction intervals overlook distributional information, neglecting the effect of multimodal or asymmetric distributions on decision-making. Similarly, full or approximated Bayesian methods, while yielding the predictive posterior density, demand major modifications to the model architecture and retraining. We introduce MCNF, a novel post hoc uncertainty quantification method that produces both prediction intervals and the full conditioned predictive distribution. MCNF operates on top of the underlying trained predictive model; thus, no predictive model retraining is needed. We provide experimental evidence that the MCNF-based uncertainty estimate is well calibrated, is competitive with state-of-the-art uncertainty quantification methods, and provides richer information for downstream decision-making tasks.

1 Introduction

Deep regression models have been widely used in applications involving the prediction of continuous variables [1], including drug discovery [2], credit scoring [3] and energy forecasting [4]. Despite their broad adoption, safety-critical applications, like medical diagnostics [5], entail high-stake decisions, mandating the development of robust deep regression techniques that equip decision-makers with complementary knowledge about the predictive uncertainty of a regression model.

Uncertainty quantification methods (UQ) are fundamental in establishing the predictive uncertainty of regression models [6]. Recent advances target primarily the investigation of epistemic uncertainty (caused by the lack of evidence or knowledge during training) and aleatoric uncertainty (the irreducible uncertainty due to data stochasticity). Building on the foundational work in quantile regression [7], Monte Carlo Dropout (MCD) [8] and deep ensembles [9] leverage dropout layers and multiple functionally-equivalent models, respectively, to approximate at inference time a deep Gaussian process and estimate the predictive distribution. Both methods, however, incur extra computational overheads and suffer from inefficient sampling, particularly at the predictive distribution tails [10]. Bayesian-based approaches that perform full Bayesian estimation [11] or its variational counterpart [12], albeit rigorous in uncertainty incorporation in the posterior distribution, incur prohibitive computational costs for any modern deep learning model. Conformal prediction (CP) [13] yields statistically rigorous uncertainty intervals that contain the ground truth based on a user-defined error rate [14], but its reliance on a calibration dataset restricts its applicability.

Motivated by the need for rigorous UQ in safety-critical applications, we introduce MCNF, a post hoc distribution-free UQ method for deep regression models underpinned by contextualized normalizing

flows (NF). MCNF uses a trained deep regression model equipped with dropout layers and yields statistically rigorous uncertainty intervals arising from the full predictive density function. Under the hood, MCNF exploits MCD sampling more efficiently to produce a prediction set per input leveraging key statistical information (e.g., mean, variance) to condition (contextualize) the normalizing flow [15]. Our experimental evaluation using a diverse set of datasets and state-of-the-art UQ methods [14, 16] demonstrates that MCNF achieves competitive results in terms of marginal coverage while also having lower error values and narrower intervals. Its applicability to deep learning architectures other than feed-forward regression networks is also showcased. Similarly to CQR [14] and MCCP [16], MCNF operates at inference time while also being capable of representing arbitrarily complex uncertainty distributions, which neither CQR nor MCCP support. Our concrete contributions are:

The MCNF method for the uncertainty quantification whose estimates are in the form of a distribution-agnostic predictive distribution.

A comprehensive MCNF evaluation against state-of-the-art UQ methods (MCD, CQR, MCCP) on various standard benchmarks and a physicochemical dataset.

A prototype open-source MCNF tool and case study repository, available at https://github. com/alexiatoumpa/MCNF .

2 Related Work

Uncertainty Quantification (UQ) in deep regression models remains an open-ended question [10]. Quantifying uncertainty in deep learning models enables reasoning about the model's confidence in its predictions. Quantile Regression (QR) [7] constructs prediction intervals by modeling the relationship between a set of independent variables and quantiles of target variables. A typical approach for UQ is training Bayesian Neural Networks (BNNs) [17], comprising neural networks with a probability distribution for the model parameters that learn the predictive posterior of the target variable. Although the output probability distribution of a BNN captures the model uncertainty, BNNs are computationally-intensive, demanding significantly more training time than other methods [11].

Monte Carlo Dropout (MCD) [8] samples weights in each layer using a binomial distribution at the selected dropout rate. Each forward pass produces a new estimate coming from the predictive posterior, which approximates Bayesian inference of a Gaussian process [18]. This technique draws parallels to deep ensembles [9], but shares weights across model realizations.

Variational inference approximates full Bayesian approaches by introducing a family of distributions that make the modeling problem tractable, commonly amortizing the parameters of the posterior [19] with an auxiliary function trained on the data.

Conformal Prediction (CP) [13, 20] is a distribution-free, non-parametric forecasting method which exploits past experience to determine the level of confidence for new predictions. CP produces prediction intervals indicating the confidence level of the model, which is inversely-related to the interval size [21]. Conformal Quantile Regression (CQR) [14] combines quantile regression with conformal prediction techniques, aiming to construct prediction intervals without distributional assumptions. MCCP [16] combines Monte Carlo dropout and conformal prediction techniques by dynamically adapting the conventional MCD with a convergence condition and employing advanced conformal prediction techniques for the synthesis of robust prediction intervals.

3 Preliminaries

Normalizing Flows (NF) [22, 23] is a modeling framework for the characterization of arbitrarily complex probability distributions, often referred to as target distribution p X ( x ) , where a set of invertible and differentiable transformations is applied over simple probability density functions (e.g., uniform, standard normal) or a base distribution p Z ( z ) . NF samples can be drawn from the target distribution by sampling from the base distribution and applying the set of transformations that convert the latent variable Z into the original random variable X , and estimating the density for a given value of the random variable X .

Let X ∈ R d be a random variable with an intractable probability density function and Z ∈ R d another random variable with a known and tractable probability density function. Let also g = g 1 ◦ · · · ◦ g L be a set of L differentiable and invertible functions composition (i.e., a flow) such that X = g ( Z ) .

The density function of the target random variable can be expressed in terms of the base density by:

Since NFs can estimate the density function, they can be trained by maximizing the likelihood with respect to the parameters of the transformation functions of the given data [15].

Neural Spline Flows (NSF) [24] are a type of transformation flow that fulfills the NF requirements of invertibility and differentiability [15]. Monotonic rational-quadratic splines endow the transformations with high non-linearity and flexibility, where the support vector (or knot) widths and heights and the derivatives of the polynomial on the internal knots are estimated using a neural network with learnable parameters. These transformations can easily integrate conditions to model conditioned density functions p X ( x | c ) , where c is the condition (or context) vector.

4 MCNF

M onte C arlo N ormalizing F low (MCNF), whose high-level workflow is shown in Fig. 1, enables quantifying the predictive uncertainty for regression tasks. Let D = { x , y } N denote a dataset of size N for a regression task, where x and y are the predictor vector and the predicted variable, respectively. Then, in MCNF, the predictive posterior distribution p ( y | x , D ) resulting from the Monte Carlo Dropout (MCD) [8] is calibrated post hoc and is fully decoupled from the training of the predictive (base) model that describes the data. Thus, MCNF separates the modeling task from UQ, and employs MCD to derive prior estimations, leveraging the commonly used dropout as a regularization technique in deep learning architectures.

MCNF, like CP [14], estimates p ( y | x , D ) without making distributional assumptions about the uncertainty. Specifically, we achieve this distribution-free concept by applying Normalizing Flow (NF) [15] as a downstream post-processing of the MCD-generated samples. Rather than modeling the predictive variable directly, MCNF uses the NF to describe the distribution of the prediction errors conditioned on the prior prediction, propagating the epistemic uncertainty encoded by the MCD prior estimate and combining it with the aleatoric uncertainty of the underlying process.

4.1 Formal description

MCNF uses estimates from MCD ( y MCD) as a latent variable acting as a prior to the actual predictive distribution. Thus, we express the predictive probability distribution p ( y | x , D ) by marginalizing the joint probability distribution p ( y, y MCD | x , D ) over the prior.

Therefore, the predictive distribution is expressed in the form of a mathematical expectation of the conditional distribution p ( y | y MCD , x , D ) . Since Equation (2) is intractable, we resort to Monte Carlo approximation to estimate the predictive distribution of y , resulting in:

where n MCD is the number of prior samples to approximate the predictive distribution.

To account for the epistemic uncertainty propagation from the prior to the predictive distribution, we introduce the change of variable, δ = y -y MCD, which reflects the prediction error. Hence, the conditioned probability distribution is now expressed in terms of δ instead of y , given by p ( δ | y MCD , x , D ) . This change does not affect the calculation of Equation (2) nor Equation (3), due to the linearity of the transformation, so p ( y | y MCD , x , D ) = p ( δ | y MCD , x , D ) . Therefore, Equation (3) can be expressed in terms of δ as:

Input data,

Multi-dimensional

data

Graph data

Tabular data

Sequence data

Regression model

Transformation 1

Dropout 1

Transformation

L

Dropout

L

Output

s

nMCD

Prior

Context

processing

Latent/base

distribution

s

nNF

Normalizing Flow module

Rational Quadratic Spline 1

Rational Quadratic Spline 2

Rational Quadratic Spline n

context

vector,

Corrected predictive

distribution

DATA

Prediction Error Loss

Prediction error

Training track

Sampling track

General track

Accordingly, the prior distribution can be approximated using the Monte Carlo Dropout sampling process, while the conditioned distribution of the prediction errors, δ , is modeled through the normalizing flow, F -1 ( δ, y MCD , x ) ∼ p θ , ψ ( δ | y MCD , x , D ) providing an efficient correction of the prior. In the latter expression, θ and ψ are, respectively, the flow transformation parameters and the base distribution parameters.

4.2 Building the MCNF Context

In MCNF, the prediction errors are conditioned on the prior estimate y MCD and the observed input features (or descriptors) x . These two variables configure a context vector c that should be fed to the normalizing flow head of MCNF as a context to locate and scale the predictive distribution. Through this contextualization, MCNF defines a convenient procedure so that the context definition is agnostic to the input data structure, and in such a way that MCNF can still be applied post hoc. To accomplish this, we define a proxy, h ( x ) , over the input descriptors based on an internal representation of the regression model, ϕ , with its learnt parameters, Ξ , ϕ ( x ; Ξ) . This approach leverages the feature extraction made by the regression model and uses it to define a proper context for the normalizing flow. Since one or more layers may be selected at different depths, this becomes a tunable hyperparameter of MCNF that directly impacts the dimensionality of the context vector. This procedure normalizes the shape of the proxy, which helps to generalize the method.

The context vector should be completed by virtue of the latent variable: the prior estimate y MCD. In practice, we observed that achieving numerical stability and computational efficiency entailed replacing each sample generated using MCD by the estimates of the two first moments of the prior

distribution. Thus, the context vector is expressed as follows:

where ¯ y MCD and log s 2 (¯ y MCD ) are the sample estimates of the expectation and variance of the prior distribution.

Finally, we adapt the definition of the density function of the predictive distribution described in Equation (4) according to the proposed context vector (Equation (5)) to obtain our working equation.

Note that Equation (6) is not equivalent to Equation (4). However, considering the hypothesis of normality of the distribution estimated with MCD, we assume that using the first two moments of the distribution of the latent random variable allows us to obtain a reasonable approximation of the marginalized distribution.

4.3 MCNF Training

MCNF training is based on the forward Kullback-Leibler divergence D KL (Equation (7)) [15]. This is equivalent to minimizing the negative log-likelihood of the training data with respect to the model parameters θ (for the transformation flows) and ψ (for the base distribution).

It is well established that minimizing the log-likelihood may lead to model overfitting and deformed distributions. This leads to uncalibrated uncertainty estimates, which become more apparent for low-uncertainty effects corrupted by large outliers. To rule this out, we regularize Equation (7) by weighing observations proportionally to the prior density. Since MCNF provides post hoc corrections of the MCD predictive posterior, we use this as the prior and reformulate L NL ( θ , ψ ) as:

By introducing w n in Equation (8), we scale the importance of each observation in a mini-batch according to its departure from the prior. We use a softmax function σ ( · ) such that weights fall within the (0,1) interval. To mitigate the effect of a highly informative prior, as for MCD, we temperature scale the softmax function using a hyperparameter τ . Accordingly, the effect of outliers can be mitigated without affecting the fitting of the regular data. We note that Equation (8) reduces to Equation (7) when τ →∞ (reflecting a design decision when there is knowledge that outliers are absent from the data).

Mini-batches are constructed by resampling the MCD samples to account for the variability encoded by the prior. Further details are provided in Algorithm 2 in the Appendix.

4.4 MCNF Inference

MCNF operates hierarchically, i.e., samples need to be drawn from the predictive model using MCD to generate the context prior to assessing the predictive distribution of the uncertainty using the Normalizing Flow head. This means that to generate n NF samples from the approximated predictive distribution p θ , ψ ( y | x , D ) , we first need to generate n MCD samples from the base model ϕ ( x , Ξ) . We will refer to this set of samples as the prior or warm-up samples.

The required input data to make estimates according to Algorithm 1 consist of the number of prior samples n MCD , as well as the architecture of the Normalizing Flow F -1 ( δ, c ) and the number of

Algorithm 1 Steps involved in a full forward pass of the proposed MCNF method for inference

Input :

D = { x n , y n } N Test n =1

Parameters : n MCD , ϕ ( x , Ξ) , F ( δ, c ) , n NF

Output : y pred and/or p θ , ψ ( y | x , D )

1: Run n MCD forward passes of ϕ ( x n , Ξ) times using MCD for every every observation, x n .

2: Collect hidden states, h ( x n ) , generated in each forward pass.

3: Aggregate hidden states from the forward passes.

4: Generate context vector, c n according to (5).

5: Feed c n to F ( δ, c n )

7: Estimate y n as y n = y n,j ( k ) , MCD -δ n,k , where j ( k ) ∼ U (1 , n MCD )

6: Draw n NF samples, δ n = { δ k,n } n NF k =1 , from z k,n ∼ p 0 , ψ ( Z ) → δ k,n = F ( z k,n , c n )

8: Estimate each sample density feeding F -1 ( δ n,k , c n ) into (6)

9: return { y n,k } n NF k =1 and { p θ , ψ ( y k,n | x k,n , D ) } n NF k =1 } |D| n =1 ∈ R |D|× n NF

samples requested to characterize the approximate predictive distribution n NF . The inference process starts by building the context c . This entails drawing the requested n MCD samples from the regression model ϕ ( x , Ξ) using MCD, which are then summarized to generate the sample mean and log-var to partially construct the context. Since every forward pass on ϕ ( x , Ξ) produces one proxy of the input, we average these estimates to complete the context vector c . Then, the context vector is given as an input to F -1 ( y, c ) (line 5). For sampling tasks, we first generate samples from the NF base distribution z n,k that are then submitted through the forward pass to obtain δ n,k = F ( z n,k , c n ) and then correct the prior y n,j ( k ) , MCD with the sampled prediction error to finally get y n,k (line 8). During the same forward pass, the Jacobian of the transformation flows is also assessed, enabling likelihood estimation by solving Equation (1) for p θ , ψ ( y k,n | x k,n , D ) . For density estimation tasks, an observation y n is passed along with the context vector c n . Then, the likelihood is calculated by running the reverse pass of the NF (Equation (1)).

5 Evaluation

Base Predictive Model. The base predictive model is a Deep Quantile Regressor (DQR) comprising a batch normalization input layer, two fully connected layers (with ReLU nonlinearities and dropout layers with rate 0.1) and an output layer with three linear units for the quantiles q = { 0 . 05 , 0 . 5 , 0 . 95 } . The model is trained for 100 epochs using the Adam optimizer, a custom pinball loss function that aggregates the quantile errors, and a batch size of 32. We fixed the learning rate to 5e-4 and the weight decay regularization factor to 1e-6. For the training and testing sets, we use an 80:20 split.

Probabilistic Model. The Normalizing Flow component of MCNF uses a sequence of two Neural Splines flows [24], with a 3-layered multilayer perception comprising 64 hidden units which produces the 16 support vectors of the spline transformation and their inner derivatives. As a base distribution, we use a factorized Gaussian distribution with trainable parameters ψ . The context size is determined by the size of the input proxy plus the two statistics (sample mean, and log-var) used to summarize the MCD samples, i.e., ( | h ( x ) | +2 ). To keep the computational overhead related to the prior Monte Carlo Dropout sampling low, we set n MCD = 50 . The training includes the same partition as the base predictive model, using a batch size of 32 with the Adam optimiser and a 0.001 learning rate. We set τ = 1 e 10 to instantiate Equation (8), giving the same weight to all observations in a mini-batch.

Comparative Methods. Weassessed MCNF against five state-of-the-art UQ methods that also use the predictions from the base predictive DQR model for uncertainty estimations. Thus, MCQR involves resampling 1000 times DQR using MCD and averaging the q = { 0 . 05 , 0 . 95 } quantile outcomes to yield the prediction intervals. MCD derives p MCD ( y | x ) by resampling from the median ( q = 0 . 5 ) in DQR 1000 times, resembling the conventional Monte Carlo Dropout uncertainty quantification method. MCD is the only other method that can also produce a predictive distribution. We also used Conformalized Quantile Regression (CQR) [14], which applies a non-conformity score to DQR to conformalize the prediction intervals, and Monte Carlo Conformal Prediction (MCCP) [16], which conformalizes the prediction intervals obtained with MCQR. Both CP-based methods used 20% of the test set for calibration and prediction intervals aimed at achieving a 90% marginal coverage. Note

that for all the considered UQ methods that employ MCD sampling, the aleatoric term of the original formulation [8] is left out as it is used to propagate the epistemic uncertainty only.

Benchmarks. Adopting the evaluation procedure from CQR [14] and MCCP [16], we used the following datasets to evaluate MCNF: the Boston Housing dataset [25] (506 observations, 14 attributes); the Concrete dataset [26] (1030 observations, 9 attributes); the Abalone dataset [27] (4177 observations, 11 attributes); the Tertiary Protein Structure dataset [28] (45730 observations, 10 attributes); the wave energy dataset [29] (63600 observations, 149 attributes), and the superconductivity dataset [30] (21263 observations, 81 attributes). These datasets were obtained from [31]. Similar to [14], and to examine the MCNF's ability to capture the uncertainty of complex distributions, we also include two synthetic datasets: the dataset from [14] with univariate predictor samples and few large outliers (termed Romano-Original) and an extension (termed Romano-Mod) with a multimodal distribution.

Performance Metrics. To compare the performance of all methods, we used the metrics reported below. We report results across 20 independent runs of the MCNF training procedure described to account for randomness. The training/test partitioning was generated per run and held constant to train both the base predictive model and the comparative methods (MCNF, CQR, MCCP, MCCQR).

Marginal coverage signifies the proportion of the test set over which the predicted quantile intervals include the ground truth, given by C ( X,Y ) = 1 N ∑ N i =1 1 { y i ∈ [ q α ( x i ) , q 1 -α ( x i )] } , where q α ( x i ) and q 1 -α ( x i ) are the predicted quantiles given x i and α = 0 . 05 .

Interval size signifies how well the prediction intervals capture the aleatoric and epistemic uncertainties, with smaller intervals denoting easier inputs and larger intervals harder ones, given by ˜ ∆( X,Y ) = median N i =1 ( q 1 -α ( x i ) -q α ( x i )) , where q α , q 1 -α and α are as above.

Accuracy assesses if the prediction intervals under- or overestimate uncertainty, specified by the mean absolute error (MAE) for α = 0 . 5 and given by MAE ( X,Y ) = 1 N ∑ N i =1 | y i -q 0 . 5 ( x i ) | .

5.1 Results Summary

Accuracy and coverage. Table 1 summarizes the performance results of our evaluation. MCNF overall yielded the smallest MAE, closely followed by MCD, indicating that training the NF MCNFcomponent helps improve the accuracy of the predicted interval. MAEs for the other methods are larger, usually by one order of magnitude, compared to MCNF and MCD. This is expected since the prediction adjustment carried out in the CP-based methods CQR and MCCP is homogeneous for both upper and lower intervals, which is only useful when the distribution is close to a Gaussian.

Considering marginal coverage C , we observe that all methods, except for MCD, provide values close to the theoretical 90% coverage, showing similar capabilities and some degree of conservativeness. This is especially true for the conformalized methods (i.e., CQR and MCCP) for the smallest datasets (Boston housing and Concrete). However, MCD does not account for aleatoric uncertainty and, thus, the intervals generated from the quantiles of its predictive distribution are highly non-conservative. For larger datasets, the miscalibration of the conformalized methods becomes smaller and more stable as the number of observations to calibrate the prediction intervals increases. MCNF outperforms its MCD counterpart, highlighting the benefits of providing post hoc corrections over the latter. In addition, MCNF is computationally more efficient than MCD, especially when the number of samples to approximate the predictive distribution increases.

Interval size. Although all methods show some sensitivity to the actual uncertainty associated with the given dataset (as shown by the variability of the interval sizes ˜ ∆ in Table 1), MCNF yields the smallest interval sizes while maintaining the expected 90% marginal coverage. While MCD also has small interval sizes, its marginal coverage is much worse than the other methods, failing to meet the expected threshold. Likewise, CQR and MCCP, in order to achieve the 90% through conformalization, yield conservative intervals which are reflected to their corresponding interval sizes. Accordingly, MCNF provides the best tradeoff between coverage and interval size, showing smaller interval sizes for similar marginal coverages.

Complex distribution. Unlike the multivariate datasets examined so far (e.g., Boston housing, Concrete), the Romano-Original and Romano-Mod univariate synthetic datasets allow for a clear visual comparison in reconstructing the data distribution. Figure 2 shows the prediction intervals from MCNF, MCD, and MCCP, alongside the predictive distributions across the predictor variable x . The

Romano-Mod dataset exhibits heteroskedasticity and varying distributions of the predicted variable for different x values. MCNF effectively captures the multimodality of the predicted variable y for small x , transitioning to a unimodal distribution as x increases. This result is further corroborated by the overall better results achieved by MCNF against CQR, MCCP and DQR across the marginal coverage C , interval size ˜ ∆ and MAE in both Romano-Original and Romano-Mod datasets.

Predictive Model Impact. Since MCNF is a post hoc UQ method, we evaluated the impact of the base predictive model on MCNF's performance. Thus, we assessed the performance using a well-trained predictive model and an underfitted model on the Concrete, Superconductivity, and Protein datasets. Selecting the well-trained predictive model entailed training 15 predictive models of the same architecture, for 100 epochs each, and selecting the model with the lowest RMSE. Similarly, selecting the underfitted model entailed training 15 predictive models for 6 epochs and selecting the model with the highest RMSE. Figure 3 shows the coverage and confidence interval for these two experiments, with results for the well-trained predictive model presented with a darker color and results for the underfitted model shown with a lighter shade color. Although the marginal coverage between the two predictive models and across most UQ methods (except from MCD) are similar and around the 90% threshold, the confidence interval plots (bottom) indicate that a well-trained predictive model provides, expectedly, narrower interval sizes. More importantly, though, even with an underfitted predictive model, MCNF yields narrower interval sizes than the state-of-the-art UQ methods and the coverage values are comparable to the well-trained predictive model. Accordingly, these results demonstrate that MCNF can adapt its UQ estimate based on the quality of the predictive model.

Epistemic uncertainty propagation. The performance metrics for MCNF without propagating the epistemic uncertainty through prediction errors via MCD sampling are shown in the last column of Table 1 (reported as NF). To produce these results, the MCNF workflow remains the same but prediction errors are not resampled. It is noteworthy that with this setup, there is a general

3.0

Observationwithincoverage

MCNF 90% CI(smoothed)

Observationoutsidecoverage

MCNF PredictiveDensity

2.5

MCNF MedianPrediction

MCDPredictiveDensity

MCCP90%CI

2.0

> 1.5

1.0

0.5

0.0

1.5

1.0

0.5

0.0

0.5

1.0

1.5

100

06

concrete

十

100

06

superconductivity

100

06

protein

std)

80

80

08

coverage (mean ±

70

FOL

70

TI

60

60

60

50

50

工

50

40

40

40

享

30

30

20

20

20

1.4

2.0

1.6

工⊥

+

1.8

std)

1.2

1.4

1.2

1.0

王

1.6

1.4

(mean ±

王

1.0

1.2 -

0.8

80

1.0

90

TI

T

王亚

0.6

0.8

0.6

0.4

0.4

0.4

L

mcd

mccp

mcqr

mcnf

mcd

ubp

cqr

mccp

mcnf

mcd

mccp

mcqr

mcnf

Method

Method

Method

performance decay for across all datasets included in the evaluation. When this source of uncertainty is not explicitly propagated, the prediction intervals deduced from the predictive distribution are consistently narrower than those obtained with the full MCNF.

5.2 MCNF application on Graph Neural Networks

We demonstrate the generalizability of MCNF to other deep learning model architectures through its application on a Graph Neural Network (GNN) predictive model [32] to make structure-based predictions of the physicochemical properties of molecules. The GNN model features three graph convolution layers based on the message passing mechanism, followed by an average readout layer to combine node-level encodings into a graph-level encoding. The next layers are a batch normalization layer to center and scale graph-level encoding, and a sequence of two dense layers of 300 neurons each (first a linear layer, then a second applying ReLU). Dropout was applied globally at a rate of 0.15. We use the Solubility dataset [33] that includes solubility data of 829 drug-like molecules.

We challenge MCNF by making it retrieve context for the NF component using an internal representation of a pre-trained GNN. Therefore, the parameters of the GNN remain constant while fitting the NF component of MCNF. The results for this experiment are shown at the bottom of Table 1. The obtained results are in agreement with those observed for the other datasets (Boston housing, Concrete, etc.) using deep regression models. In particular, all methods exhibit good performance in terms of marginal coverage, except for MCD. Among the well-calibrated methods, MCNF stands out by providing the best trade-off between coverage C , prediction interval sizes ˜ ∆ and MAE.

6 Conclusion and Future Work

MCNF is a post hoc method that quantifies uncertainty in deep regression models by estimating the predictive distribution of the predicted variable. To achieve this, MCNF utilizes pre-trained deep regression models with dropout layers and models prediction errors using a shallow normalizing flow to correct prior MCD estimates. Through a comprehensive experimental evaluation comprising diverse datasets and state-of-the-art UQ methods, we demonstrate that prediction intervals from MCNF are well-calibrated, with smaller median sizes, providing richer information than baseline methods. The approach generalizes well to pre-trained GNNs, showing good calibration and adaptivity. Additionally, we mitigate the negative impact of outliers on the forward Kullback-Leibler divergence loss function by using MCD before weighing observations in a mini-batch. Future work could involve extending the MCNF formalism to classification problems and further characterizing the knowledge arising from the predictive model and its underlying assumptions. Furthermore, we will investigate techniques to improve the computational efficiency of MCNF by reducing the number of required MCDsamples and coupling it with more efficient alternatives for building the prior knowledge.

Acknowledgments and Disclosure of Funding

This work has been supported by the projects QCircle (grant agreement No 101059999) and SOPRANO, GuardAI and AI4Work (grant agreements No 101120990, 101168067 and 101135990, respectively), funded by the European Union. Views and opinions expressed are however those of the author(s) only and do not necessarily reflect those of the European Union. Neither the European Union nor the granting authority can be held responsible for them.

References

Stéphane Lathuilière, Pablo Mesejo, Xavier Alameda-Pineda, and Radu Horaud. A comprehensive analysis of deep regression. IEEE transactions on pattern analysis and machine intelligence , 42(9):2065-2081, 2019.

Yuanqui Du, Arian R. Jamasb, Jeff Guo, Tianfan Fu, Charles Harris, Yingheng Wang, Chenru Duan, Pietro Liò, Phillippe Schwaller, and Tom Blundell. Machine learning-aided generative molecular design. Nature Machine Intelligence , 6:589-604, 2024.

Ralf Kellner, Maximilian Nagl, and Daniel Rösch. Opening the black box-quantile neural networks for loss given default prediction. Journal of Banking & Finance , 134:106334, 2022.

Yixiao Yu, Ming Yang, Xueshan Han, Yumin Zhang, and Pingfeng Ye. A regional wind power probabilistic forecast method based on deep quantile regression. IEEE Transactions on Industry Applications , 57(5):4420-4427, 2021.

Raghavendra Selvan, Frederik Faye, Jon Middleton, and Akshay Pai. Uncertainty quantification in medical image segmentation with normalizing flows, 2020.

Moloud Abdar, Farhad Pourpanah, Sadiq Hussain, Dana Rezazadegan, Li Liu, Mohammad Ghavamzadeh, Paul Fieguth, Xiaochun Cao, Abbas Khosravi, U Rajendra Acharya, et al. A review of uncertainty quantification in deep learning: Techniques, applications and challenges. Information fusion , 76:243-297, 2021.

Roger Koenker and Gilbert Bassett. Regression quantiles. Econometrica: journal of the Econometric Society , pages 33-50, 1978.

Yarin Gal and Zoubin Ghahramani. Dropout as a bayesian approximation: Representing model uncertainty in deep learning. In international conference on machine learning , pages 1050-1059. PMLR, 2016.

Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. Advances in neural information processing systems , 30, 2017.

Jakob Gawlikowski, Caio R. N. Tassi, Muhammad Ali, et al. A survey of uncertainty in deep neural networks. Artificial Intelligence Review , 56(Suppl 1):1513-1589, 2023.

Radford M Neal. Bayesian learning for neural networks , volume 118. Springer Science & Business Media, 2012.

Diederik P Kingma and Max Welling. Auto-encoding variational bayes, 2014.

Vladimir Vovk, Alexander Gammerman, and Glenn Shafer. Algorithmic learning in a random world , volume 29. Springer, 2005.

Yaniv Romano, Evan Patterson, and Emmanuel Candes. Conformalized quantile regression. Advances in neural information processing systems , 32, 2019.

George Papamakarios, Eric Nalisnick, Danilo Jimenez Rezende, Shakir Mohamed, and Balaji Lakshminarayanan. Normalizing flows for probabilistic modeling and inference. Journal of Machine Learning Research , 22(57):1-64, 2021.

Daniel Bethell, Simos Gerasimou, and Radu Calinescu. Robust uncertainty quantification using conformalised monte carlo prediction. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 38, pages 20939-20948, 2024.

David JC MacKay. A practical bayesian framework for backpropagation networks. Neural computation , 4(3):448-472, 1992.

Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting. The journal of machine learning research , 15(1):1929-1958, 2014.

Durk P Kingma, Tim Salimans, and Max Welling. Variational dropout and the local reparameterization trick. Advances in neural information processing systems , 28, 2015.

Matteo Fontana, Gianluca Zeni, and Simone Vantini. Conformal prediction: a unified review of theory and new challenges. Bernoulli , 29(1):1-23, 2023.

Anastasios Angelopoulos, Stephen Bates, Jitendra Malik, and Michael I Jordan. Uncertainty sets for image classifiers using conformal prediction. arXiv preprint arXiv:2009.14193 , 2020.

Ivan Kobyzev, Simon JD Prince, and Marcus A Brubaker. Normalizing flows: An introduction and review of current methods. IEEE transactions on pattern analysis and machine intelligence , 43(11):3964-3979, 2020.

E. G. Tabak and Cristina V. Turner. A family of nonparametric density estimation algorithms. Communications on Pure and Applied Mathematics , 66(2):145-164, 2013.

Conor Durkan, Artur Bekasov, Iain Murray, and George Papamakarios. Neural spline flows, 2019.

David Harrison and Daniel L. Rubinfeld. Hedonic prices and the demand for clean air. Journal of Environmental Economics and Management , 1978. J. Environ. Econ. Manag.

I. C. Yeh. Modeling of strength of high-performance concrete using artificial neural networks. Cement and Concrete Research , 1998.

W. J. Nash, T. L. Sellers, S. R. Talbot, A. Cawthorn, and W. B. Ford. The population biology of abalone (haliotis species) in tasmania. i. blacklip abalone (h. rubra) from the north coast and islands of bass strait. Marine and Freshwater Research , 1994.

Prashant Rana. Physicochemical Properties of Protein Tertiary Structure. UCI Machine Learning Repository, 2013. DOI: https://doi.org/10.24432/C5QW3H.

Mehdi Neshat, Bradley Alexander, Nataliia Y. Sergiienko, and Markus Wagner. Optimisation of large wave farms using a multi-strategy evolutionary framework. Proceedings of the 2020 Genetic and Evolutionary Computation Conference , 2020.

Kam Hamidieh. A data-driven statistical model for predicting the critical temperature of a superconductor. Computational Materials Science , 2018.

Markelle Kelly, Rachel Longjohn, and Kolby Nottingham. The uci machine learning repository. https://archive.ics.uci.edu . Accessed: 2024-04-01.

Zonghan Wu, Shirui Pan, Fengwen Chen, Guodong Long, Chengqi Zhang, and Philip S Yu. A comprehensive survey on graph neural networks. IEEE transactions on neural networks and learning systems , 32(1):4-24, 2020.

Mario Lovri´ c, Kristina Pavlovi´ c, Petar Žuvela, Adrian Spataru, Bono Luˇ ci´ c, Roman Kern, and Ming Wah Wong. Machine learning in prediction of intrinsic aqueous solubility of drug-like compounds: Generalization, complexity, or predictive ability? Journal of Chemometrics , 35(7-8), jul 2021.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The claims of contributions made in the abstract are properly supported by the evaluation section of the manuscript.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: The conclusion section outlines the limitations and direction for future work of MCNF.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Justification: The paper does not present any theoretical results.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: The experimental results are described in full detail: the datasets used, the model architecture, the hyperparameters for training and testing MCNF.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: We provide prototype open-source MCNF tool and case study repository.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: All training/testing setup details, as well as model architecture, are described in the manuscript, to support reproducibility.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: The experimental results have been run multiple times, thus the mean and standard deviation values of the reported metrics are presented.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: We provide details regarding the computational setup used for our evaluation in the Appendix.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: The authors have reviewed the NeurIPS Code of Ethics, and preserve anonymity.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: The manuscript has no societal impact of the work conducted.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: The work presented does not pose any risks of misuse.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: Owners of the assets and contributors are properly credited, and the appropriate license is used.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: The code base for MCNF will be made public. This is made available with appropriate instructions and documentation.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: There was no crowdsourcing involved in this work, nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: There was no crowdsourcing involved in this work, nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

A MCNF Training

Mini-batch strategy

During MCNF training, in order to effectively propagate epistemic uncertainty to p θ , ψ ( y | x , D ) when the latter is approximated as in Equation (6), we bootstrap the prediction errors. To this end, each mini-batch used to evaluate the gradients of the module based on the NFs is obtained by bootstrapping on a set of n MCD samples, { y i, MCD } n MCD i =1 , previously generated from the distribution p ( y MCD | x ) . Thus, the original training dataset D = { x n , y n } N n =1 is further processed at each iteration of the training steps to recalculate the prediction error δ n , as indicated in Algorithm 2. Consequently, the implemented mini-batch construction strategy enables every mini-batch to have a different realization of the prediction error per observation at each iteration.

B Synthetic dataset

Romano-Mod dataset. A univariate stochastic process is introduced, by adapting the equation provided in Appendix B in [14]. The distribution that characterizes the uncertainty incorporates heteroskedasticity, which is a particularly relevant validation case to test whether the method adapts correctly to the local distribution of the data. The updated stochastic process is given:

where δ () represents the Delta Kronecker function and ϵ 1 and ϵ 2 follow a standard Gaussian distribution. The parameters of the model are β and ∆ , which condition the heteroskedasticity of the uncertainty distribution, as well as γ and ¯ u , which control the magnitude and rate of outliers in the sample generated. We introduce b to induce a linear correlation between the predictor x and the predicted variable y . Table 2 summarizes the values used for these parameters and Figure 4 illustrates the data generated.

τ

=100, b=0,

γ

=25

τ

=100, b=2,

γ

=150

τ

=50000, b=0,

γ

=25

τ

=50000, b=0,

γ

=150

C MCNF performance details

C.1 Adaptivity of prediction intervals

In terms of prediction interval adaptivity, all examined UQ methods, including MCNF, are sensitive to the uncertainty inherent in the data, as evidenced by the variability of the quantiles of the interval sizes shown in Table 3. MCD has the narrowest intervals but covers a much smaller proportion of the true values, achieving a very poor coverage overall. Considering the UQ methods that perform well and converge to the expected marginal coverage (90%), MCNF provides the narrowest intervals and maintains similar marginal coverage to the other methods. Quantile MAE ( MAE q ), where MAE q ( X,Y ) = 1 N ∑ N i =1 | y i -q 0 . 05 ( x i ) | + | y i -q 0 . 95 ( x i ) | , which provides additional evidence of the improved trade-off between coverage and interval sizes for MCNF, exhibits smaller values for similar marginal coverages.

C.2 Ablation study

C.2.1 Hyperparameter study

We also examined the performance of MCNF for the following hyperparameters: number of epochs (epochs ∈ { 20 , 50 , 100 , 150 } ) , number of normalizing flow samples (nNF ∈ { 100 , 200 , 500 } ), and number of Monte-Carlo Dropout samples (nMCD ∈ { 50 , 100 , 150 } ). For this evaluation, we employ the same prediction model across all tests, which was selected based on the lowest RMSE value, as described in Section 5.1.

Initially, we evaluate the number of epochs specified for training MCNF, examining epochs ∈ { 20 , 50 , 100 , 150 } , as shown in Figure 5, whilst keeping the number of normalizing flow samples and Monte-Carlos Dropout samples fixed to nNF = 500 and nMCD = 50 , respectively. Figure 6 illustrates

the results of evaluating the different normalizing flow sample values nNF ∈ { 100 , 200 , 500 } , setting epochs = 100 and nMCD = 50 . Lastly, Figure 7 visualizes the ablation results for the different MonteCarlo Dropout sample values nMCD ∈ { 50 , 100 , 500 } , whilst training the MCNF for epochs = 100 and sampling nNF = 500 normalizing flow samples. Our results demonstrate that all tested nMCD

epochs=[20,50,100,150],nNF=500,nmcD=50

boston_housing

abalone

concrete

1001

100

100

王

王

王

王

亚

(mean ±std)

80

80

80

60

09

60

coverage(

40

40

40

20

20

20

-0

0

-0

0.52

0.650-

0.68

0.625

0.50

0.66

009'0

0.64

0.575

0.46 -

0.62

0.550

Ω.

0.44

0.60

0.525

0.500

0.42

0.58

0.475 -

0.40

0.56

0.450

0.108

(mean ±std)

0.075

0.106

0.105

0.104

0.100

0.070-

0.102-

0.095

mae_median

0.065-

0.100

0.090

0.098

0.085

0.060

960°0

0800

9500

0.094

0.075

5

1

2

5

0

1

epochs

epochs

epochs

values perform similarly in terms of the coverage metrics, and the difference in the confidence interval and the median MAE is negligible. Similar observations have been derived for the various epoch and nNF values evaluated.

C.2.2 Outliers handling: testing impact of τ hyperparameter

As outlined in recent UQ survey papers [6], UQ methods can be impacted by outliers. Thus, we investigated the performance and response of MCNF to outlier data. Table 4 shows experiments carried out for different τ , b and γ value combinations on MCNF coverage ( C ), median MAE ( ˜ MAE ), and Quantile MAE ( MAE q ). We observe that MCNF was sensitive to outliers when the effect size was close to zero, which likely results from biasing of the maximum likelihood estimator. The effect size relates to the degree of correlation between the feature variables and the response variable. We sought to mitigate this impact, given that low effect sizes (as well as sporadic large outlier values) frequently characterize data sets in many disciplines.

To test the impact of different τ values in Equation (8), with respect to changing effect size, we synthesize data (Section B) by modulating the slope ( b ) of the governing generative process to generate multiple sets of simulated data (Figure 8), where b ∈ { 0 , 0 . 1 , 0 . 2 , 0 . 3 , 1 , 2 } . As the scale of the outliers relative to the rest of the data also changed with the slope, we modulated the γ parameter to keep the outliers in the same approximate range. The outliers proportion was 1%. For each level of slope, we predicted from the MCNF method using the following τ values τ ∈ { 10 2 , 2 · 10 2 , 3 · 10 2 , 10 3 , 2 · 10 3 , 5 · 10 3 , 10 4 , 10 5 , 10 6 }. The data distributions are shown in Figure 8.

Generally, the marginal coverage increases with the value of τ as the effect of the temperature scaling assigns equal importance to outliers and non-outliers. However, this comes at the cost of overly conservative prediction intervals and less adaptivity to the uncertainty in the data. No single value for τ can be considered universally optimal, as shown in Figure 8. This hyperparameter should be fine-tuned, potentially using a calibration set. When the ratio of the outlier magnitude over the effect size becomes smaller (larger values of the slope in this case) the importance of the former on the regular maximum likelihood procedure decreases, and values of τ can be smaller.

epochs=100,nnF=[100,200,500], nmcp=50

boston_housing

abalone

concrete

100

100

100

王

亚

王

(mean±std)

80

80

80

60

60

60

coverage(

40

40

40

20

20

20

0-

0

0.56

0.50

0.64

S0

0.48

0.62

0.46

09°0

0.52

0.44

0.58

0S'0

0.42

0.56

0.48

0.40

0.075

0.104

std)

0.102

0.070

060°0

0.100

0.085-

0.065

0.098

0.080

0.060

960°0

0.075

S500

0.094

100

100

F007

nNF

nNF

nNF

epochs=100, nnF=500, nmcd=[50,100,500]

boston_housing

abalone

concrete

100

100

100

80

80

80

60

60

60

40

40

40

20

20

20

-0

0

r0

0.56

0.50

0.64

SS'0

50

0.62 -

0.53

9t0

0.60

0.52

0.51

0.58

0S'0

0.42

0.49

0.56

一

0.0725

0.104

0.095

(p1S

0.0700

(mean

0.0675

0.102

0.090

0.0650

0.100

_median

0.0625

0.085

0.0600

8600

0.080

0.0575

0.0550

0.096

0.075

厂

厂

100

厂

10

nMCD

nMCD

nMCD

Coverage vs.

10 2

10 3

10 4

10 5

tau (log scale)

0.800

0.825

0.850

0.875

0.900

0.925

0.950

0.975

1.000

coverage

b

0.00

0.10

0.20

0.30

1.00

2.00

for differing b

Peer reviews:
comments: SUMMARY:
The Authors propose an uncertainty quantification technique for regression models based on neural networks. This technique makes use of Montecarlo dropout to get a prior estimate of the epistemic uncertainty of the model, and combines it post-hoc with normalizing flows to get a distribution of the prediction error (includung aleatoric uncertainty).

As reflected in my confidence score, my evaluation capabilities for this paper are limited as UQ is not my main area of research.

STRENGTHS AND WEAKNESSES:
### Strengths
- Thorough experimental evaluation addressing multiple metrics and datasets. Among the experimental evaluations, I also appreciated a lot the study on the impact of the pre-trained predictive model. 
- The paper is generally well written, although many variables are introduced with proper explanation (see weaknesses).

#### Weaknesses
- Many variables are introduced without proper explanation. Among these there are: $\mathcal{D}$ on line 97, $\Xi, \phi$ on line 132, $\theta, \psi$ on line 124
- Lacking details on the computational complexity of the method. The inference task, in particular, requires first performing Monte Carlo dropout sampling, and then a pass on the normalizing flow model.

QUESTIONS:
- In the experimental evaluation, you report averaged results over 20 independent runs. Initialization and the stochastic gradient dynamics are a source of epistemic uncertainty. Do you have estimates on the magnitude of this uncertainty compared to the one introduced by MCD? Does this question make sense at all?

LIMITATIONS:
Not discussed
TITLE: Official Review
CLARITY: 3
IMPACT: 2
ORIGINALITY: 2
DATE: 2025-06-24

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents a posthoc UQ method leveraging upon Monte Carlo Dropout (MCD) and normalizing flows.  Specifically, the normalizing flow model is trained to learn a distribution for the output variable $y$ given a feature vector derived somewhere in the base prediction model and summary statistics of the mean and log-variance of the output of the base model under MCD. The experiments somewhat demonstrate that the error quantiles are better calibrated of the new method against other state of the art UQ methods for regression problems.\n\nSTRENGTHS AND WEAKNESSES:\nThe proposed UQ methods makes sense and seems effective for capturing arbitrary probability distributions that can be multimodal with complex structure. The essence of the idea is that one can learn the distribution of possible output from a context variable consisting of features from a base forward pass model along with statistics of the output of the base model under MCD. This idea is intriguing and appears novel.\n\nThe experiments seem to compare with a sufficient number of baseline state-of-the-art methods. I appreciate that the paper considers both DQR and GNNs as base prediction models. \n\nI do have some concerns with the experiments. First, I am not clear why the interval length should matter.  The interval length is dictated by the unknown ground truth distribution of the output $y$ given knowledge of the input raw features $x$, and thus $\\Delta$ may need to large or small in light of the spread of this ground truth distribution.\n\nWhile it is great to measure the coverage for a desired coverage of 90%, it think it would be more useful to compute an expected absolute calibration error by considering the absolute difference of the coverage and the desired coverage of x% for x ranging from 0 to 100.\n\nI like Figure 2 as a means to show the need to accommodate  complex multimodal distributions in UQ methods. \n\nThe paper does not discuss the computational cost for inference via sampling for MCNF relative to the computational cost of the baseline methods.\n\nAs a reader, it would be easier for me to understand $\\log \\sigma^2$ as the log-variance rather than $\\log s^2$ as given in (5).\n\nIn (6), it is not clear what depends on the Monte Carlo index \"$i$\" for the summation. Somehow, it is hidden in $\\delta$, which is only revealed in Algorithm 1.\n\nQUESTIONS:\nCan the authors explain why the interval length $\\Delta$ matters and why it is not intrinsic with respect to the CDF of possible values of $y$?\n\nWhat are the values for $n_{MCD}$ and $n_{NF}$ used in the paper?  Have the authors investigated the robustness of the MCNF with respect to these parameters?\n\nIn Figure 3, I get why the interval spread should be larger for the underfitted (less trained) model, and this is prominent in the strong baselines. However, the increase in the spread for MCNF is less prominent.  Why would this be desirable?  I would think that epistemic uncertainty increases in the underfitted models, and thus $\\Delta$ should jump much higher.\n\nLIMITATIONS:\nThe paper mentions the computational complexity as a possible limitation, especially for larger prediction models, in the concluding section. Also, the concluding section mentions adapting the method for classification as future work. The paper could discuss the computational complexity for inference in the experimental section with respect to the other UQ baseline methods.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-06-27"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a normalizing-flow-based post-hoc uncertainty quantification method that works on top of an already trained monte-carlo dropout model. The prediction mean and variance (or rather log-variance) of the output of the MCD model along with certain hidden layers of the MCD model are used to to form the condition (or context) of the NF model. The NF model itself represents the density of the residual \\delta around the mean of the base model prediction. The method is explained in detail, and then empirical evaluation is provided.\n\nSTRENGTHS AND WEAKNESSES:\nOverall, this is a quite strong paper. Strengths:\n- Innovative idea, providing a method for distribution-free uncertainty quantification\n- Well structured paper, (mostly) clear descriptions\n- Sufficient evaluation\n- Proper source-code provided\n\nWeaknesses:\n- Standard evaluation not included (UCI Benchmark)\n- Possible confusion around around MCD predictive distributions (see questions)\n- Unclear theoretical predictive power w.r.t. being distribution free (see questions)\n- Confusing framing of the base-distribution of the normalizing flow (see questions)\n\nWeak reject for now, but I expect to increase the score if the questions are answered in a convincing way.\n\nQUESTIONS:\n- The original monte-carlo dropout method outputs an aleatoric mean and variance, and the dropout part is only used to quantify epistemic uncertainty (ie, uncertainty of the mean and variance parameters themselves). Is this considered in the design of the NF method? Is it considered during MCD evaluation as a baseline?\n- The mcd samples are reduced to mean and variance before feeding them to NF model (mind you, if I understand correctly, this is the mean and variance of the \\mu parameter of the gaussian aleatoric distribution predicted by MCD). This appears to lose any shape information that the base model might predict for the mean (and any information predicted by the model for the aleatoric variance). Can you comment on this?\n- Is there a reason that no NLL evaluation is done on the datasets, as is standard? Similarly, is there a reason that the quasi-standard UCI regression uncertainty benchmark is not presented along with the evaluations currently in the paper? (As is done e.g. in the original monte-carlo dropout paper and a large number of papers since).\n- In the beginning, e.g. eq 3, we are made to believe that the normalizing flow transforms the output distribution of the base model into the aleatoric posterior of y through monte-carlo sampling. Yet after a while it turns out that in fact the output distribution of the base model is summarized into mean/var as context for the NF, and the actual base-distribution being transformed is in fact a certain z_{n,k} (e.g. line 180), I assume a gaussian, as is usual in normalizing flows. Does this not mean that the initial description is quite misleading? Did I miss something here?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces Monte Carlo Normalizing Flow (MCNF), a novel post-hoc method for uncertainty quantification (UQ) in deep regression models. The core idea is to improve upon the uncertainty estimates generated by Monte Carlo Dropout (MCD) by using them to \"contextualize\" a normalizing flow (NF). This NF then models the distribution of the prediction errors. The method aims to provide not just prediction intervals but the full predictive posterior distribution, which can capture complex features like asymmetry and multimodality.\n\nSTRENGTHS AND WEAKNESSES:\npros:\n- The proposed MCNF method presents a clever and novel way to combine existing techniques (MCD and NFs)\n- The proposed approach can be deployed on already trained deep learning models with dropout layers without the high computational cost of retraining.\n\ncons:\n- The approach relies on MCD and the MCD context provided to the normalizing flow might be misleading\n- The approach introduces additional computational steps at inference time (operations of the normalizing flow).\n- The results are difficult to read as several (antagonistic) criteria are necessary to compare the methods. It is hard to know which method delivers the best trade-off\n\nQUESTIONS:\nThe paper mentions that using the first two moments is a reasonable approximation under a normality assumption for the MCD distribution. How does MCNF's performance change if this assumption is strongly violated, for instance in a clearly bimodal case?\n\nThe training loss includes a temperature-scaled softmax to weigh observations, mitigating the effect of outliers. In the experiments, τ is set to a very large value (1e10), effectively giving equal weight to all observations. Why was this choice made?\n\nHow does MCNF compare to other methods that also use normalizing flows for uncertainty quantification, but perhaps in a different manner ?\n\nA strength of conformal prediction is that it enjoys coverage guarantees for the queried alpha. Does MCNF enjoy some theoretical properties (even under ideal assumptions) ?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-03"}