Estimating Epistemic and Aleatoric Uncertainty with a Single Model
Authors: Matthew Albert Chan
Maria J. Molina
Christopher Metzler
Venue: NeurIPS 2024 poster (2024)
Keywords: uncertainty estimation
diffusion models
hypernetworks
Primary area: diffusion_based_models
TL;DR: Efficient approach to generate aleatoric and epistemic uncertainty estimates for inverse problems.

Abstract: Estimating and disentangling epistemic uncertainty, uncertainty that is reducible with more training data, and aleatoric uncertainty, uncertainty that is inherent to the task at hand, is critically important when applying machine learning to high-stakes applications such as medical imaging and weather forecasting. Conditional diffusion models' breakthrough ability to accurately and efficiently sample from the posterior distribution of a dataset now makes uncertainty estimation conceptually straightforward: One need only train and sample from a large ensemble of diffusion models. Unfortunately, training such an ensemble becomes computationally intractable as the complexity of the model architecture grows. In this work we introduce a new approach to ensembling, hyper-diffusion models (HyperDM), which allows one to accurately estimate both epistemic and aleatoric uncertainty with a single model. Unlike existing single-model uncertainty methods like Monte-Carlo dropout and Bayesian neural networks, HyperDM offers prediction accuracy on par with, and in some cases superior to, multi-model ensembles. Furthermore, our proposed approach scales to modern network architectures such as Attention U-Net and yields more accurate uncertainty estimates compared to existing methods. We validate our method on two distinct real-world tasks: x-ray computed tomography reconstruction and weather temperature forecasting.

Estimating Epistemic and Aleatoric Uncertainty with a Single Model
Matthew A. Chan
Maria Molina
Department of Computer Science University of Maryland College Park, MD 20742 mattchan@umd. edu
Department of Atmospheric and Oceanic Science
University of Maryland College Park, MD 20742
mjmol wollllnn@umm...mm edu
Christopher A. Metzler
Department of Computer Science
University of Maryland College Park, MD 20742 metzler@ume edu
Abstract
Estimating and disentangling epistemic uncertainty, uncertainty that is reducible with more training data, and aleatoric uncertainty, uncertainty that is inherent the task at hand, is critically important when applying machine learning to high- stakes applications such as medical imaging and weather forecasting. Conditional diffusion models' breakthrough ability to accurately and efficiently sample from the posterior distribution of dataset now makes uncertainty estimation conceptually straightforward: One need only train and sample from large ensemble of diffu- sion models. Unfortunately, training such an ensemble becomes computationally intractable as the complexity of the model architecture grows. In this work we in- troduce a new approach to ensembling, hyper-diffusion models (HyperDM), which allows one to accurately estimate both epistemic and aleatoric uncertainty with single model. Unlike existing single-model uncertainty methods like Monte-Carlo dropout and Bayesian neural networks, HyperDM offers prediction accuracy on par with, and in some cases superior to, multi-model ensembles. Furthermore, our proposed approach scales to modern network architectures such Attention U-Net and yields more accurate uncertainty estimates compared to existing methods. We validate method on two distinet real-world tasks: x-ray computed tomogra- phy reconstruction and weather temperature forecasting. Source code is publicly available ftthh:t::/// :/gg thub. com wwt.///maathhaaccaaa/yrree.ee
Introduction
Machine learning (ML) based inference and prediction algorithms are being actively adopted in range of high-stakes scientific and medical applications: ML already deployed within modern computed tomography (CT) scanners (10). ML is actively used to search for new medicines [29, and over the last year ML has begun to compete with tate-of-theearr weather and climate forecasting systems 44 36. 77. In mission-critical tasks like weather forecasting and medical imaging/diagnosis, the importance of reliable predictions cannot be overstated. The consequences of erroncous decisions range from massive financial costs more critically, the loss of human lives.
in these domains In this context, understanding and quantifying uncertainty a pivotal step towards improving the robustness and reliability of ML models.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

(a) Repeat M times to get ensemble weights.
C
0, E [0, M
Noise Vector
U.Net Weights
Hyper-Network
(b) Repeat N times per weight vector.
C
Condittin (t-0)
Stj.J E [0
Noise
Attention U-Net Diffusion Model (Frozen)
Stochastic Prediction
U-Net Weights
(c) Compute aggregate prediction and uncertainty.
Aggregate
Epistemic Uncertainty
Uncertainty
Figure I: General framework of HyperDM. (a) A Bayesian hyper-network is optimized to generate diffusion model weights from randomly sampled noise. This process is repeated M times to obtain an ensemble of M weights. (b) A diffusion model accepts fixed weights from the hyper-network to stochastically generate prediction. This process is repeated N times for each set of weights, yielding a total M x predictions. (c) The ensemble predictions are aggregated to produce final prediction and epistemic/ / aleatoric uncertainty map.
For an uncertainty estimate to be most useful, it must differentiate between aleatoric and epistemic uncertainty. Aleatoric uncertainty describes the fundamental variability and ill-posedness of the inference task. By contrast, epistemic uncertainty describes the inference model's lack of knowledge or understanding- whiih can be reduced with more diverse training data. Distinguishing between these two types of uncertainty provides valuable insights into the strengths and weaknesses of predictive model, offering pathways towards improving its performance.
applications like weather forecasting, epistemic uncertainty can be used to inform the optimal placement of new weather stations. Additionally, in medical imaging, decomposition of uncertainty into its alcatoric and epistemic components important for identifying disttribbititiooo measurements where model predictions should verified by trained experts.
This work presents new approach for estimating aleatoric and epistemic uncertainty using single model. Specifically, our approach uses novel pipeline integrating conditional diffusion model (33 and Bayesian hyper-network [4]] to generate ensemble of predictions. Conditional diffusion models allow one to sample from an implicit representation of the posterior distribution an inverse problem. Meanwhile, hyper-networks allow one to sample over collection of networks that are consistent with the training data. Together, these components can efficiently estimate both sources of uncertainty, without sacrificing inference accuracy. Our specific contributions summarized below:

---

Table I: Comparison of training and inference times. The time required to train M 10 member ensemble on the LUNAI6 dataset is shown in the second column. The third column shows the time required generate a predictive distribution of size M N 1000 for a single input.
METHOD
TRAINInG TIME (MINUTES)
EVALUATION TIME (MINUTES)
MC-DROPOUT [8
47.03
3.70
DPS-UQ [4]
441.09
3.31
HYPERDM
48.53
3.18
We apply Bayesian hyper-networks in a novel setting (i.c., diffusion models) to estimate both epistemic and aleatoric uncertainties from single model.
We conduct a toy experiment with ground truth uncertainties and show that the proposed method accurately predicts both sources of uncertainty.
We apply the proposed method on two mission-critical real-world tasks, CT reconstruction and weather forecasting, and demonstrate that our method achieves a significantly lower training overhead and better reconstruction quality compared to existing methods.
We conduct ablation studies investigating the effects of ensemble size and the number of ensemble predictions on uncertainty quality, which show (i) that larger ensembles improve out-o--diitribution detection and (ii) that additional predictions smooth out irregularities in aleatoric uncertainty estimates.
2 Related Work
2.1 Uncertainty Quantification
Probabilistic methods are commonly used to estimate uncertainty by first generating an ensemble of models and subsequently quantifying uncertainty as the variance or entropy over the ensemble's predictions [II] Deep ensembles (35] explicitly train such ensemble to predict epistemic un- certainty. However, with modern neural network architectures exceeding billion parameters, the computational cost required to train deep ensembles is prohibitively expensive.
Other methods attempt approximate deep ensembles while circumventing its training overhead. Bayesian neural networks (BNNs) [40 [4] use variational inference [19] 65) to model the posterior weight distribution. Monte-Carlo (MC) dropout [[8] leverages dropout [57 stochastically induce variability in the network's 's predictions. Recent works in weather forecasting 7 46] perturb network inputs with random noise similarly generate stochastic predictions. Still, each of these methods has notable trade-offs preventing their widespread adoption. BNNs incur runtime cost that scales proportionally with the number of model parameters, leading to slow inference and training times. MC dropout and input perturbation introduce noise into the inference process, which adversely affects model prediction quality. Moreover, perturbing inputs with noise not equivalent to deep ensembles (in the Bayesian sense) as these methods optimize the weights of single, deterministic model.
separate branch of research 48 explores distribution-Fi uncertainty estimation, which uses conformal prediction 4 and quantile regression (32) to estimate bounds on aleatoric uncertainty. Subsequent works (58] have extended this additionally estimate epistemic uncertainty; however, these methods use deep ensembles to do so so- leading to similar issues with computational complexity.
2.2 Hyper-Networks
Hyper-networks 20 employ a unique paradigm where one network-the "hyper- thtper-network" generates weights for another "primary" network. This framework cireumvents the need to train multiple task-specific or dataset-specific models. Instead, one need only train a single hyper-network to cover a range of tasks or datasets. Given an input token representing a specific task, hyper-network learns to generate reasonable weights with which the primary network can accomplish that task [6]] Note that, during training, losses are back pack-propagated such that only the hyper-network's parameters updated while the primary network's 's weights are purely generated by the heper.network.
3

---

Estimating Aleatoric Uncertainty
(b) Estimating Epistemic Uncertainty
Tegend
175-
Probudeliyy Density
150
Probakilty Deusiry
125-
20
25
[
Figure 2: Accurate uncertainty estimation using HyperDM. (a) HyperDM trained on four 1D datasets with aleatoric uncertainty determined by noise variance og Variances across diffusion model predictions are visualized as one distribution per training dataset. Aleatoric estimates (i.e., the mean of each distribution) accurately predict o (b) HyperDM is trained on four datasets with epistemic uncertainty determined by dataset size [D]. Prediction means are visualized as one distribution per training dataset. Epistemic estimates (i.e., the variance each distribution) grow inversely with [D].
33 extend hyper-networks quantify uncertainty. Rather than accepting task-specifer tokens as inputs, BHNs accept random noise and stochastically generate weights for primary network. BHNs thus serve as an implicit representation for the true posterior weight distribution 47 28). Epistemic uncertainty is measured as the variance across predictions yielded by the primary network for different weights sampled from the BHN.
Bayesian hyper-networks (BHNs)
2.3 Diffusion Models
represent a class of generative machine learning models that learns sample from target distribution. These models fit to the Stein score function 39 of the target distribution by iteratively transitioning between an easy-to-sample (typically Gaussian) distribution and the target distribution. During training, samples from the target distribution corrupted by running the forward "noising" diffusion process, and the network learns to estimate the added noise. To generate samples, the network iteratively denoises images of pure noise until they looks like they were sampled from the target distribution (23. DMs have shown success in generating high-quality, realistic images and capturing diverse data distributions [12. To date, however, there has been limited research 6 investigating the use of DMs for uncertainty estimation.
Diffusion models (DMs) 52 55 56
3 Problem Definition
Given measurements y y corresponding to signals of interest æ X, our objective is to train model which can simultaneously recover x and quantify the aleatoric and epistemic uncertainty of its predictions. The predictive distribution of such model is given by
p(rly, D) p(xly, ())do
(I)
where p(r Φ) is the likelihood function, and p(d) is the posterior over model parameters á for training dataset D [59]. Uncertainty on this distribution stems from distinct sources: alcatoric uncertainty and epistemic uncertainty 13
3.1 Aleatoric Uncertainty
Aleatoric uncertainty (AU) arises from inherent randomness in the underlying measurement process and is represented by the likelihood function Equation TT). Most notably, this source of uncertainty is irreducible for given measurement process [1
30 In the context of predictive modeling, AU represents how ill- posee the task is and often associated with noise, measurement errors, or inherent unpredictability in the observed phenomena.

---

Consider an inverse problem where the goal is to recover from measurements
(2) defined by forward operator F and non-zero measurement noise 1) ~ (N0 by learning the inverse mapping : - X. Even with perfect model capable of sampling from the true likelihood p(rly), irreducible errors are still present due to the ambiguity around which æ ~ p(x) explains the observed measurement y. This ambiguity captures the inherent randomness (i.e., the aleatoric uncertainty) of the inverse problem and is measured by the variance o² of n (27].
= F()) + 77,
3.2 Epistemic Uncertainty
Epistemic uncertainty (EU) relates to a lack of knowledge or incomplete understanding of a problem and is reducible with additional training data [27]. This type of uncertainty reflects limitations in a model's knowledge and its ability to accurately capture underlying patterns in the data. Assume we initialize M models with random weights { and sufficient capacity to perfectly capture the inverse model described in Section 3.T After training, discrepancies (i.e., epistemic uncertainty) inevitably arise in the final weights learned by each model, due to the random weight initialization pro- cess, As additional training data is provided, model weights converge more strongly crongly--oorespooding
to reduction in EU
4 Method
We measure uncertainty using variance and apply the law of total variance ||| total uncertainty (TU) across model predictions X p(zly, D) into its AU and EU components
60
50 to decompose
Var(X) = Var
Bawp(x v.o)
+ E Eapp(o/D)
Vara=p(=lvvd)
[x]
(3)
AU
The first term captures the explainable uncertainty, given by the variance of sampled weights á ~ (() over the expected values of samples è p(xlyy, Φ) from the likelihood function. This term ignores variance caused by the posepdsessess of the likelihood function and therefore represents EU. The second term captures the unexplainable uncertainty and is given by the expectation sampled weights Ø P() over the variance of samples & p(zly, 6) from the likelihood function. This term ignores variance caused by the sampling of weights from the posterior and therefore represents AU.
Both the likelihood function D() and the posterior p(z yy, σ) do not have an explicit closed-fore making computation of (3) intractable. To circumvent this, we instead learn their respective implicit distributions 228
(( (() and (zly)).
4.1 Implicit Likelihood Function
As demonstrated in [55], DMs enable sampling from an implicit conditional distribution ((>ly) by learning to invert diffusion process that gradually transforms a target data distribution into simple (typically Gaussian) data distribution [56]. The forward diffusion process can be described by T length Markov chain
:= : (01); (2000:;/1100 0f of
4
that transforms samples (0) from the data distribution into samples x (T) from a Gaussian distribu- tion 23). Conversely, the reverse diffusion process
(() (01)) y) := N (t (1)) of (0) log ()) of)
transforms pure noise into samples from p(xly)
63
Since explicit computation of the score function V., p(r (y) is intractable [16] neural network
s(a, fly, Φ) is typically trained to approximate via an 1.2 minimization objective
E(z,g)> [| log p(x (0) 0,
fyy,
(6)
5

---

(b) DPS-UQ
MC-Dropout
HyperDM
Anomalous Measurement
030000
50000
033115
EU
0.2900
03997
59X00
100033
G00E3S
AU
22202
0O0C99
6.000
Figure 3: Weather forecasting out-of-distrbuutin data. (a) An out-of-distribttion measurement is formed by synthetically inserting hot spot in the northeastern part of Canada. (b) Epistemic and aleatoric uncertainty maps are produced by each method on the provided measurement. Compared to other methods, HyperDM is best able to isolate the abnormal feature in its epistemic estimate.
where D { (oFo.y))... (2 yN)] represents the training dataset. Once the DM has finished training, we can sample from the implicit likelihood function g(xly) by first sampling random noise ((T) ~ [O, σ²) and iteratively denoising the image times following Equation (5) to obtain (()) ~ q(<ly)
4.2 Implicit Posterior Distribution
Similar to the likelihood function, the posterior weight distribution has no explicit closed-form representation,
instead make use of an implicit distribution (()) to approximate p(olD). As mentioned Section 2.2 BHNs [34 enable sampling from (()) by transforming samples AN00, o²) into weights
(()) for the primary network. In the of the inverse imaging
problem from Section 3.1 the primary network would be a network f-(()) with parameters ø that learns the inverse mapping from measurements to signals y + X.
Training a BHN differs from conventional deep learning methods in that the weights of the primary network & are generated by a hyper-network and are thus not learnable parameters. Instead, the weights 0 of BHN hg are optimized via the minimization objective
E (( [lf(y ho(2)) 2: =1
(7)
where he maps random input vectors 2 ~ N(0, o2) to weights Φ (see Figure I.). weights produced by the BHN do not collapse which yield plausible predictions with respect to 1.2 distance. As less data available during training, broader range of network weights reasonably explain that data.
Importantly, mode because there many network weights
4.3 Estimation of Aleatoric and Epistemic Uncertainty with Single Model
We leverage DMs and BHNs to implicitly model p(rly, d) and p()(D), respectively, thus enabling sampling from both distributions. Specifically, our framework consists of a BHN ha that generates weights 6; q()) for DM s(-l)), which we collectively refer to hyper-diffusion model (HyperDM).. At inference time, we sample E M weights from ha and for each weight Φ; generate j N samples from
distribution of M N predictions f..; (see Figure I]. This framework is transformation of Equation (I). where both posterior and likelihood have been replaced with implicit distributions yield tractable approximation of the predictive distribution
p(rly, D) ~ q(xlv)a(@)do.
(8)

---

Table 2:
Ensemble prediction quality real-world data. Baseline image quality assessment scores are calculated on test data from a CT dataset (i.e., LUNA16) and weather forecasting dataset
(i.e., ERA5). Best scores are highlighted in red and second best scores highlighted in blue.
LUNAI6
ERA5
METHOD
SSIM f
PSNR (DB) f CRPS 4
SSIM f
PSNR (DB) f CRPS
MC- DROPOUT [8]
0.77
30,25
0.023
0.93
31.34
0.034
DPS-UQ [4]
0.89
34.95
0.01
0.94
32.83
0.013
HYPERDM
0.87
35.16
0.01
0.95
33.15
0.012
Applying Equation (3)
uncertainty over the predictive distribution X {di.j.... (f.,)....,, ,.,,, decom- posed into its respective aleatoric and epistemic components,
AU
ExEM
VarjeN
x
(9)
EU
Var.eM
EjEN
[
(10)
such that TU AÚ + EU. Following existing ensemble methods 35
46
7
we compute the
aggregate ensemble prediction as the expectation over X, formally expressed as
BigM,iEN
X
(11)
Compared to other aggregation methods (e.g.. median, mode), we observe the best performance when taking the ensemble mean. Please refer to Figure and Table in the supplement for more details.
Unlike deep ensembles which require training M distinct models to compute EU and AU, HyperDM only requires training single model (i.e., BHN) theoretically consuming up to M-fold fewer computational resources. Furthermore, unlike many pseudo-ensembling methods [8 46), Hy- perDM doesn't 't need to exploit randomness caused by perturbations model p D avoiding adverse effects on model performance. Moreover, unlike BNNs, HyperDM is relatively cheap to sample from in terms of computational runtime and resources making it significantly faster and more scalable compared to BNN- based uncertainty estimation methods.
Differing from prior work (60). we make Gaussian assumptions the predictive distribution p(rly, D) nor on the likelihood function p( (xlyy. 6). This is because our method approximates p(zly. Φ) by repeatedly sampling & (<x|y)) from DM, rather than explicitly modeling the distribution Gaussian with mean ji and variance o². Therefore, our aggregate predictive distribution p(zly, D) not restricted Gaussian mixture model (x), over the collective mean μa, and variance of of all ensemble members.
Experiments
Please refer to Appendix A for training details (e.g., network architectures and loss functions).
Baselines. We focus on comparing HyperDM against methods which similarly capable of estimating both EU and AU. Our benchmark consists of a state-of-the-art method, deep-posterior sampling for uncertainty quantification [14] (henceforth referred as DPS-UQ), and dropout-based method (referred to as MC-Dropout). DPS-UQ is implemented as an A --memeer ensemble of deep-posterior sampling (DPS) DMs. MC-Dropout is implemented as a single DM with weights sampled from g()) using dropout instead of BHN. Despite its inability to jointly predict EU and AU, we also include a BNN baseline in our initial experiments illustrate the advantages of our method in terms of prediction speed and accuracy.
Metrics. We evaluate the quality of baseline predictions using both full-reference image quality met- rics and distribution-based metrics. Specifically, we compute peak signal-to-noise ratio (PSNR) [24] and struetural similarity index (SSIM) (62) between mean predictions (see Equation (1h)) and their corresponding ground truth references. We also compute the continuous ranked probability score (CRPS) 4 as holistic indicator of the quality of X, given by
CRPS(F, =
(F(a) 11 1laz] da,
(12)
7

---

where F is the cumulative distribution function of X and 1. is the Heaviside step function.
In initial experiments, we compare baseline estimates of AU and EU against ground truth uncertainty. However, extending such validation to more complex tasks and datasets is difficult because uncertainty is affected by wide variety of environmental factors (e-g., measurement noise, sampling rates) which are often unreported. As a result, in subsequent experiments, we follow [14] and evaluate uncertainty by generating out-of-distribution (OOD) measurements and verifying whether baseline estimates of EU correctly predict OOD pixels.
5.1 Toy Problem
We first evaluate our method toy inverse problem to establish the correctness of our uncertainty estimates under a simple forward model where ground truth uncertainty is explicitly quantifiable. Training datasets are generated using the function
sin(y) +
(13)
where 17 ~ NN o and measurements y u(-5,5), We conduct two separate experiments to validate our method's ability to estimate uncertainty against ground-truth EU and AU.
Estimating AU. To test our method's ability to estimate AU, we generate four training datasets using
Equation (3) with ground truth AU characterized by noise variances o E {0.01, 0.04, 0.16, 0.64]. Each training dataset has [D] 500 examples, and a HyperDM is trained each dataset for 500 epochs. After training, we sample M = 10 weights from la and N = 10000 realizations from s( s(-lo) to obtain distribution of M x N predictions. We compute AU for each ensemble member using Equation g and visualize ear [X] across all M weights in Figure 2 The AU estimates across the four datasets are AU = f0.02, 0.05, 0.21, 0.64), which closely match the ground-truth.
Estimating EU. We test our method's s ability to estimate EU by generating four training datasets of varying sizes [D] E {100, 200, 400, 800) and fixed noise variance of 0.01. Unlike AU, ground truth EU cannot be explicitly quantified because it is independent from the training data (5) As
result, we follow prior works [35] 13 38 and validate EU qualitatively. We train a HyperDM on each dataset for 500 epochs and draw M x N samples from it where M = 10, N 10000. We then
calculate EÚ using Equation (10) and plot E for all M weights in Figure 2 The EU estimates
across the four datasets are EU 1.92 10-1, 2.20 x 10-5, 1.17 x 10-5 1.83 x 10-0), ordered
by increasing [D]. As expected. EU decreases [D] grows increasingly larger.
To highlight the key advantages of HyperDM over traditional uncertainty estimation techniques, we also train BNN on the same datasets and sample M = 10000 predictions. The resulting estimates EU- = {0.091, 0.071, 0.050, 0.012] indicate similar inversely proportional relationship with (D). However, despite having the same backbone architecture and training observe aggregate predictions of lower individual and mean quality from the BNN when compared to HyperDM (see Table the supplement). Moreover, we observe that BNNs take over 2x longer to train compared to HyperDM (i.e., 70 seconds vs. 30 seconds), and inference is an order of magnitude slower (i.e., 8.7 seconds vs. 0.7 seconds to generate 10,000 predictions), due to BNN's 's need to sample from the weight distribution runtime.
we
5.2 Computed Tomography
In this experiment, we demonstrate our method's applicability for medical imaging tasks, specifically
CT reconstruction. Using the Lung Nodule Analysis 2016 (LUNA16) (5I) dataset, we form target image distribution X' by extracting 1,200 CT images, applying 4x pixel binning to produce 128 x 128 resolution images, and normalizing each image by mapping pixel values between [-1000, 3000] Hounsfield units to the interval 1-1, 1). We subsequently compute the sparse Radon transform with 45 projected views and add Gaussian noise with variance o² = 0.16 to the resulting sinograms. Using filtered back-proeection (FBP) [25]. we obtain low-quality reconstructions y' of original images X. The dataset is finally split into a training dataset comprised of 1,000 image-measurement pairs and a validation dataset of 200 data pairs. Following the training procedure described in Appendix A we train MC-Dropout, DPS-UQ, and HyperDM on LUNA16. For fair comparison, all baselines are sampled from using M = 10 and N 100 for total of M x N 1000 predictions. We refrain
8

---

(b)
Anomalous Measurement
MC-Dropout
DPS-UQ
HyperDM
E00019
EU
6.0013
0.6
11002)
cree00
00000
AU
00000
6000
000000
0.001
0000)
Figure 4: CT reconstruction on out-of-distribution data. (a) An out-of-distribution CT measure- ment formed by synthetically inserting metal implants along the spine. (b) Epistemic and aleatoric uncertainty maps are produced by each method on the out-of-distribttion measurement. Both DPS- UQ and HyperDM are able distinguish the abnormal feature in their epistemic prediction.
from training BNN baseline on this dataset due to the high computational resources and runtime required scale to the image domain.
In T we show average CRPS, PSNR, and SSIM scores computed over the test dataset. The rel- atively low image quality scores obtained by MC-Dropout indicative of the adverse effects caused by randomly dropping network weights at inference time. Meanwhile, DPS-UQ reconstructions achieve 15.5% higher average PSNR than MC-Dropout, but the cost of eight-fold increase in training time (see Table [ On the other hand, HyperDM yields predictions of similar and sometimes better-quality than DPS-UQ while only adding a 3% overhead in training time compared to MC-Dropout, Note that the discrepancy in training times between DPS-UQ and HyperDM will continually widen tional costs required to train M > 10 member deep ensembles, we limited baselines to ten-member ensembles for this experiment.
scale the ensemble size beyond M 10. However, due to the high computa-
To evaluate the quality of baseline uncertainty predictions, we first select a random in-distribution image X' and generate its corresponding OOD measurement first artificially inserting abnormal feature (i.e., metal implants along the spinal column) and subsequently computing the corresponding FBP measurement y. Results in Figure show that DPS-UQ and HyperDM yield comparable results in that their EU predictions successfully highlight the OOD implant. contrast, MC-Dropout fails to highlight OOD pixels in its BÚ prediction. While prior work [43] suggests that AU estimates are unreliable-aa and should be subsequently
seeggardedwwwheeeee EU is high, we nonetheless include AU results in Figure 4 demonstrate that HyperDM produces AU predictions similar to that of a deep ensemble.
5.3 Weather Forecasting
In this experiment, we demonstrate the applicability of HyperDM for climate seennce-seecffically two-meter surface temperature forecasting. Using the European Centre for Medium- -ange Weather Forecasts Reanalysis v5 (ERA5) dataset (22). generate dataset comprised of 1,240 surface air temperature maps sampled at six-hour time intervals (i.e., 00, 06, 12, 18 UTC) in January between 2009-2018. Images are binned down to 128 x 128 resolution and normalized such that pixel values between [210, 313] Kelvin map to the interval [-1, 1]. Following experiments done in [46), we form data pairs (z, y) using historical temperature data at time t as the initial measurement image y and data at time t + hours as the target image æ. total of 200 images are held-out and used for validation and testing purposes.
Using the same training procedure as Section 5.2 ERA5 and generate predictions with sampling rates M 10 and N 100. Baseline PSNR, SSIM,
we train MC-Dropout, DPS-UQ, and HyperDM on

---

and CRPS scores are reported in Table where we observe trends similar to the prior experiment: DPS-UQ achieves 5% higher average PSNR score compared MC-Dropout, and HyperDM achieves 1% higher average PSNR score compared to DPS-UQ. Training overhead for DPS-UQ remains around 8x that of MC-Dropout and HyperDM due to the need to repeat training M times.
To generate OOD measurements, we first obtain an in-distribution measurement and subsequently insert an anomalous hot spot over northeastern Canada. Inspecting results in Figure 3 we observe that HyperDM's EU prediction more accurately identifies OOD pixels than DPS-UQ. contrast, MC-Dropout fails to identify the hot spot in its EU prediction and instead incorrectly identifies regions in the central United States as OOD, Interestingly, all methods predict lower AU over the ocean versus the North American continent, which aligns with our expectations, as water has less temperature variability compared to land due to its higher specific heat. Additional qualitative results showing the decomposition of TÜ into its EU and AÚ components are provided in Figure 8 of the supplement.
Limitations and Future Work
We acknowledge two main limitations of our approach and identify potential avenues for improvement. Firstly, as a consequence of their iterative denoising process, inference on DMs is slow compared to inference classical neural network architectures. However, recent advances in accelerated sampling strategies have largely mitigated this issue and allow for few [53) (and in some cases single [54]) step sampling from DMs. Secondly, hyper-networks suffer from a scalability problem that their number of parameters scales with the number of primary network parameters. This stems from the fact that the dimensionality of the hyper-network' output layer is most cases) proportional the number of parameters in the primary network (9). Several works address this issue by proposing more efficient weight generation strategies 61 2 Nonetheless, these problems remain promising avenue for future research.
7 Conclusion
The growing application of ML to impactful scientific and medical problems has made accurate estimation of uncertainty more important than ever. Unfortunately, the gold standard for uncertainty estimation- deep ensembles- is prohibitively expensive to train, especially on modern network architectures containing billions of parameters. In this work, we propose HyperDMM, framework capable of approximating deep ensembles at fraction of the computational training cost. Specifically, combine Bayesian heper-networks and diffusion models generate distribution of predictions with which we can estimate total uncertainty and its epistemic and aleatoric sub-components. Our experiments weather forecasting and CT reconstruction demonstrate that HyperDM significantly outperforms pseudo-ensembling techniques like Bayesian neural networks and Monte Carlo dropout in terms of prediction quality. Moreover, when compared against deep ensembles, HyperDM achieves up to an M x reduction in training time while yielding predictions of similar (if not superior) quality, where M is the ensemble size. This work thus makes major stride towards developing accurate and scalable estimates of uncertainty.
Acknowledgments and Disclosure of Funding
This work was supported in part by a University of Maryland Grand Challenges Seed Grant and NSF Award No. 2425735. M.A.C. and C.A.M. were supported in part by AFOSR Young Investigator Program Award No. FA9550-22-1-0208 and ONR Award No. N00014-23-1-2252.
10

---

References
[1] A. F. Agarap. Deep learning using rectified linear units (relu). arXiv preprint arXiv:1803.08375. 2018.
[2] Y. Alaluf, O. Tov, R. Mokady, R. Gal, and A. Bermano. Hyperstyle: Stylegan inversion with hypernetworks for real image editing. In Proceedings of the IEEE/CVF conference on computer Vision and pattern recognition, pages 18511-1852) 2022.
[3] A. N. Angelopoulos, A. P. Kohli, Bates, M. Jordan, J. Malik, T. Alshaabi, S. Upadhyayula, and Y. Romano. Image-to-image regression with distribution- free uncertainty quantification and applications in imaging. International Conference on Machine Learning, pages 717-730. PMLR, 2022.
[4] V. Balasubramanian, S.-S. Ho, and V. Vovk. Conformal prediction for reliable machine learning: theory, adaptations and applications. Newnes, 2014.
[5] V. Bengs, E. Hüllermeier, and W. Waegeman. Pitfalls of epistemic uncertainty quantification through loss minimisation. Advances in Neural Information Processing Systems, 35:29205- 29216. 2022.
[6] L. Berry, A. Brando, and D. Meger. Shedding light on large generative networks: Estimating epistemic uncertainty in diffusion models. In The 40th Conference on Uncertainty in Artificial Intelligence.
[7] K. Bi, L. Xie, H. Zhang, X. Chen, X. Gu, and Q Tian. Accurate medium-range global weather forecasting with 3d neural networks. Nature, 61 619(7970):533-536 2023.
[8] C. Blundell, Cornebise, K. Kavukcuoglu, and D. Wierstra. Weight uncertainty in neural network. In International conference on machine learning, pages 1613-1622. PMLR, 2015.
[9] V. K. Chauhan, J. Zhou, P. Lu, S. Molaei, and D. A. Clifton. A brief review of hypernetworks deep learning. arXiv preprint arXiv:2306.0693 2023.
[10] J. Chen, Y. Li, L. Guo, X. Zhou, Y. Zhu, Q. He, H. Han, and Q. Feng. Machine learning techniques for et imaging diagnosis of novel coronavirus pneumonia: A review. Neural Computing and Applications, pages 1-19, 2022.
[11) S. Depeweg, J.-M. Hernandez-Lobate F. Doshi-Velez and S. Udluft. Decomposition of uncertainty in bayesian deep learning for efficient and risk-sensitive learning. In International Conference Machine Learning, pages 1184-1193. PMLR, 2018.
[12] P. Dhariwal and A. Nichol. Diffusion models beat gans image synthesis. Advances in neural information processing systems, 34:8780-879 8794, 2021.
[13] C. Ekmekci and M. Cetin. Uncertainty quantification for deep unrolling based computational imaging. IEEE Transactions on Computational Imaging, 8: (195-129,, 2022.
[14] C. Ekmekci and M. Cetin. Quantifying generative model uncertainty posterior sampling methods for computational imaging. In NeurlPS 2023 Workshop Deep Learning and Inverse Problems, 2023.
[15] S. Elfwing, E. Uchibe, and K. Doya. Sigmoid-weighted linear units for neural network function approximation in reinforcement learning. Neural networks, 107:3-11 11, 2018.
[16] B. T. Feng, J. Smith, M. Rubinstein, H. Chang, K. L. Bouman, and W. T. Freeman. Score-based diffusion models principled priors for inverse imaging. In International Conference Computer Vision (ICCV). IEEE, 2023.
[17] Y. Gal. Uncertainty in Deep Learning. PhD thesis, University of Cambridge, 2016.
[18] Y. Gal and Z. Ghahramani. Dropout bayesian approximation: Representing model uncer- tainty in deep learning. international conference on machine learning, pages 1050-1059. PMLR, 2016.

---

[19] A. Graves. Practical variational inference for neural networks. Advances in neural information processing systems, 24, 2011.
[20] D. Ha, A. Dai, and Q. Le. Hypernetorrks, 2016.
[21] K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-
778, 2016.
[22] H. Hersbach, Bell, Berrisford, S. Hirahara, A. Horányi, J. Muñoz-Sabater, Nicolas, C. Peubey, R. Radu, D. Schepers, et al. The era5 global reanalysis. Quarterly Journal of the Royal Meteorological Society, 146(730):1999-204, 2020.
[23] Ho, A. Jain, and P. Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems,
2020.
[24] A. Horé and D. Ziou. Image quality metries: Psnr vs. ssim. In 2010 20th International Conference on Pattern Recognition, pages 2366-2369, 2010.
[25] G. N. Hounsfield. Computerized transverse axial scanning (tomography): Part 1. description of system. The British journal of radiology, 46(552) 1973.
[26] E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, and W. Chen. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations, 2022.
[27] E. Hüllermeier and W. Waegeman. Aleatoric and epistemic uncertainty in machine learning:
An introduction to concepts and methods. Machine Learning, 110:457-506, 2021.
[28] Huszár. Variational inference using implicit distributions. arXiv preprint arXiv: 170.00230 2017.
[29] J. Jumper, R. Evans, A. Pritzel, T. Green, M. Figurnov, O. Ronneherger, K. Tunyasuvunakool, R. Bates, A. Žídek, A. Potapenko, et al. Highly accurate protein structure prediction with alphafold. Nature, 596(7873):583-588) 2021.
[30] A. Kendall and Y. Gal. What uncertainties do we need in bayesian deep learning for computer vision? Advances in neural information processing systems, 30, 2017.
[31] D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. arXiv preprint 1412. 29800, 2014.
[32] R. Koenker and G. Bassett Jr. Regression quantiles. Econometrica: journal of the Econometric Society, pages 33-50, 1978.
[33] A. Kristiadi, S. Däubener, and A. Fischer. Predictive uncertainty quantification with compound density networks. arXiv preprint arXiv: 010000000000......00000.. 2019.
[34] D. Krueger, C.-W. Huang, R. Islam, R. Turner, A. Lacoste, and A. Courville. Bayesian hypernetworks. arXiv preprint 17Xiv:111.00459) 2017.
[35] B. Lakshminarayanan, A. Pritzel, and C. Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. Advances in neural information processing systems, 30, 2017.
[36] R. Lam, A. Ewalds, Z. Eaton-Rosen, W. Hu, al. Learning skillful medium-range global weather forecasting, Science, page eadi2336, 2023.
M. Willson, P. Wirnsergee, M. Fortunato, F. Alet, S. Ravuri,
[37] S. Lee, H. Kim, and J. Lee. Graddiv: Adversarial robustness of randomized neural networks via gradient diversity regularization. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022.
[38] J. Liu, J. Paisley, M.-A. Kioumourtzoglou, and B. Coull. Accurate uncertainty estimation and decomposition ensemble learning. Advances in neural information processing systems, 32, 2019.
12

---

[39] Q. Liu, J. D. Lee, and M. I. Jordan. A kemelized stein discrepancy for goodness-of tests and model evaluation, 2016.
[40] D. J. MacKay, A practical bayesian framework for backpropagation networks. Neural computa- lion, 4(3):448447,, 1992.
[41] J. E. Matheson and R. L. Winkler. Scoring rules for continuous probability distributions. Management science, 22(10):0087-199, 1976.
[42] M. T. McCann, H. Chung, J. C. Ye, and M. L. Klasky. Score-based diffusion models for bayesian image reconstruction. arXiv preprint arXiv:2305. 2023.
[43] Mukhoti, A. Kirsch, van Amersfoort, H. Torr, and Y. Gal. Deep deterministic uncertainty:
A new simple baseline. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 24384 11884-2494, 2023.
[44] R. M. Neal. Bayesian learning for neural networks, volume 118. Springer Science & Business Media, 2012.
[45] O. Oktay, J. Schlemper, L. Folgoe, M. Lee, M. Heinrich, K. Misawa, K. Mori, S. McDonagh, N. Y. Hammerla, B. Kainz, B. Glocker, and D. Rueckert, Attention u-net: Learning where to look for the pancreas, 2018.
[46] Pathak, S. Subramanian, Harrington, S. Raja, Chattopadhyay, M. Mardani, T. Kurth, D. Hall, Z. Li, K. Azizzadenesheli, P. Hassanzadeh, K. Kashinath, and A. Anandkumar. Four- castnet: A global data-driven high-resolution weather model using adaptive fourier neural operators. arXiv preprint arXiv:2202. 212122222.12214.. 2022.
[47] N. Pawlowski, A. Brock, M. C. Lee, M. Rajchl, and B. Glocker. Implicit weight uncertainty in neural networks. arXiv preprint 17Xiv:1711.0129)) 2017.
[48] T. Pearce, A. Brintrup, M. Zaki, and A. Neely. High-quality prediction intervals for deep learning: A distribution--ree, ensembled approach. In International conference on machine learning, pages 4075-4084, PMLR, 2018.
[49] F. Rosenblatt. The perceptrom: probabilistic model for information storage and organization in the brain. Psychological review, 65(6))386, 1958.
[50] S. Schreck, D. J. Gagne II, C. Becker, W. E. Chapman, K. Elmore, G. Gantos, E. Kim, D. Kimpara, T. Martin, M. J. Molina, et al. Evidential deep learning: Enhancing predictive uncertainty estimation for earth system science applications. arXiv preprint arXiv:2309.13277, 2023.
[51] A. A. A. Setio, A. Traverso. T. De Bel, M. S. Berens, C. Van Den Bogaard, Cerello, H. Chen, Q. Dou, M. E. Fantacci, B. Geurts, et al. Validation, comparison, and combination of algorithms for automatic detection of pulmonary nodules in computed tomography images: the lunal6 challenge. Medical image analysis, 42:1-13, 2017.
[52] J. Sohl-Dickstein, E. Weiss, N. Maheswaranathan, and S. Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. International conference on machine learning, pages 2256-2265. PMLR, 2015.
[53] J. Song, C. Meng, and S. Ermon. Denoising diffusion implicit models. arXiv:2010.02502. October 2020.
[54] Y. Song, P. Dhariwal, M. Chen, and I. Sutskever. Consistency models. arXiv preprint 2:X1::2303.01469, 2023.
[55] Y. Song and S. Ermon. Generative modeling by estimating gradients of the data distribution. Advances in neural information processing systems, 32, 2019.
[56) Y. Song, J. Sohl-Dickstein, D. P. Kingma, A. Kumar, S. Ermon, and B. Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:201 13456, 2020.
13

---

[57] N. Srivastava, G. Hinton, A. Krizhevsky, I. Sutskever, and R. Salakhutdinov. Dropout: simple way to prevent neural networks from overfitting. The journal of machine learning research, 15()):1929-195, 2014.
[58] N. Tagasovska and D. Lopez-Paz. Single-model uncertainties for deep learning. Advances in neural information processing systems, 32, 2019.
[59] M. E. Tipping. Bayesian inference: An introduction to principles and practice in machine learning. In Summer School on Machine Learning, pages 41-62. Springer, 2003.
[60] M. Valdenegro-Toro and D. S. Mori. A deeper look into aleatoric and epistemic uncertainty disentanglement. In 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pages 1508-1516. IEEE, 2022.
[61] von Oswald, Henning, B. Grewe, and Sacramento. Continual learning with hypernet- works. In International Conference on Learning Representations, 2020.
[62] Z. Wang, A. C. Bovik, H. R. Sheikh, and E. P. Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE transactions on image processing, 13(4):600-612, 2004.
[63] L. Wu, B. L. Trippe, C. A. Naesseth, D. M. Blei, and P. Cunningham. Practical and asymptotically exact conditional sampling in diffusion models. arXiv preprini 17777 2023.
[64] Y. Wu and K. He. Group normalization. In Proceedings of the European conference on computer vision (ECCV), pages 3-19. 2018.
[65] C. Zhang. J. Bütepage, H. Kjellström, and Mandt. Advances in variational inference. IEEE fransactions on pattern analysis and machine intelligence, 2088) 200022026666 2018.
14

---

A Training Details
All baselines are trained a single NVIDIA RTX A6000 using batch size of 32, an Adam optimizer, and a learning rate of I x 10-4. Training is run over 500 epochs in our initial experiment and 400 epochs in our CT and weather experiments. DMs are trained using Markov chain of T = 100 timesteps.
A.1 Network Architecture
The backbone architecture for all baselines (i.e., BNN, DPS-UQ, HyperDM) in the toy experiment from Section 5.1 is a multi-layer perceptron (MLP) 49 with five linear layers and rectified linear
unit (ReLU) activation functions. For experiments described in Sections (.. and [.3 we scale
the DM's backbone architecture up to an Attention U-Net [45] for baselines. The U-Net consists of initial 2D convolutional layer, followed by four 2x downsampling ResNet [21] blocks, two middle ResNet blocks, four 2x upsampling ResNet blocks, and a final 2D convolutional layer. Each ResNet block consists of two 2D convolutional layers-with group normalization (64 and Sigmoid Linear Units (SiLU) 15] activation function-ar as well an additional attention layer.
A.2 Loss Functions
The training procedure for HyperDM is identical to that of a standard DM, except that the DM's weights are sampled from BHN hg. For each training pair (æz, y). we sample DM weights by first sampling random noise z (0, of) R and then computing 6 ho(2). We manually set the DM weights equal to & and compute the loss function
LilypenDM ||C s(x(1) fly, ho (2))11²
(14)
where NN0 o is the noise added to time step t and s(-1o) represents the DM. In general, we found HyperDM training to be stable across a variety of training hyper-parameters and did not encounter any over-fitting issues.
We follow 37 and train our BNN baseline b( ()) by minimizing the loss function
CBNN | b(ul)) AKL(((0) p(o)D))
(15)
which consists of data fidelity term and an additional Kullback- (KL) divergence term between the true posterior p() and the implicit distribution -( approximated using Bayes by Backprop [8]. The weights each BNN layer are sampled from zero-mean normal distribution with standard deviation 0.1, and the KL component of the loss term is down- weighted by À 0.01.
The training procedure for DPS-UQ and MC-Dropout are identical in that they share the same loss objective
CDPS-UQ LMC-Dropour ||e s(x(1), Uy, 3)
(16)
where the weights é are randomly initialized at the start training and updated via backpropagation. However, train M separate DM instances for DPS-UQ, whereas only single DM is trained for MC-Dropout.
Median
0.0175
0.0175
0.0150
0.01.50
0.025
0.0125
0..125
0.020
0.0log
0.0100
0.0075
0.0030
0.0000
Figure Aggregation of ensemble predictions. Ensemble predictions aggregated using conventional methods (e.g.. mean, median, mode). Mean and median aggregation results are similar, while mode aggregation results noticeably more noisy.
15

---

Table 3: Reconstruction quality of different ensemble aggregation methods. HyperDM recon- struction results on the ERA5 test set are shown for three different ensemble aggregation strategies: mean, median, and mode. Best scores are highlighted in red and second best scores are highlighted in blue.
AGGREGATION
SSIM f
PSNR (DB) f
L1
CRPS 4
MEAN
0.9455
32.93
0.018
0.01292
MEDIAN
0.9452
33.06
0.017
0.01294
MODE
0.6690
25.54
0.044
0.01293
To build predictive distribution of size M N with MC-Dropout, first seed pseudo-random number generator (RNG), which we use to deterninistically sample dropout masks from a Bernoulli distribution. These masks are used zero-out input tensor elements at each network layer. We continually
then reset the RNG using the same initial seed-fixing the drop-out configuration sample from the DM until obtain N predictions for that seed. This process repeated across M different seeds for total of M N predictions. In all experiments, we train and test MC-Dropout with dropout probability p 0.3.
B
Ablation Studies
B.1
Sampling Rates
HyperDM provides flexibility inference time to arbitrarily choose the number of network weights M to ampleeanalogous to the number of ensemble members in deep ensemble-and the number of predictions N to generate per sampled weight. In this study, we examine the effect of sampling rates M, on EU and AU on our OOD experiment from Section 5.3
In our first test, estimate EU
OOD measurement for fixed N = 100 and variable M 6 indicate that under-sampling weights (i.e., M S 4) leads to
{2, 4,8, 16]. Results in Figure uncertainty maps which underestimate uncertainty around OOD features and overestimate uncertainty around in-distribution features. However, as we continue to sample additional network weights, observe increased uncertainty in areas around the abnormal feature and suppressed uncertainty around nn-distribution fcatures. This result indicates the importance of large ensembles in correctly isolating OOD features from in-distribution features for EU estimation.
In our second test, we repeat the same process but instead fix M 10 and sample N {2, 4, 16) predictions from the DM. Examining the results shown in Figure 7 we observe irregular peaks in the predicted AU at low sampling rates N < 4. However, as we sample more from the DM and the sample mean converges, AÚ becomes more uniformly spread across the entire continental landmass. This result suggests the importance of sampling large number of predictions for adequately capturing the characteristics of the true likelihood distribution.
M=2
M=4
M=8
M=16
Figure 6: Efffet of sampling more weights on epistemic uncertainty. As we increase the number M of sampled weights from the hyper-network, uncertainty around out-of-distribution features (i.e., the hot spot in the upper-right) grows and uncertainty around in-distribution features (i.e., everything else the image) shrinks.
16

---

N=
N=4 N=8
N=16
Figure 7: Effect of sampling more predictions on aleatoric uncertainty. As we increase the number N of sampled predictions from the diffusion model, aleatoric uncertainty predictions smooth out more evenly.
Table 4: Ensemble prediction quality versus BNNs. When trained on four datasets of various sizes, we observe that HyperDM produces more accurate mean predictions indicated by PSNR scores) and higher quality predictive distributions (as indicated by CRPS) than BNNs, except in the extreme low data regime. Highest scores displayed boldface.
DATASET SIZE
100
200
400
800
100
200
400
800
METHOD
PSNR (DB)
CRPS 4
BNN HYPERDM
10.34
11.09
13.53
0.20
0.18
0.14
0.13
8.47
18.43
20.28
20.44
0.23
0.09
0.07
0.07
AU
EU
TU
L2
Tec
S2005
Dropout
22015
3.033
82008
Doone
AC0030
00125
poolo
DPS UQ
0,007
SC0CE
51002
322025
000000
310
HyperDM
100
3.0885
S50
2.000
6.00
52465
Figure 8: Uncertainty decomposition temperature data. Total uncertainty and its decomposition into epistemic and alcatoric components is shown in the left three columns. The L2 error between the aggregated mean ensemble prediction and the ground truth shown in the rightmost column. We observe higher total uncertainty around the North American continent, which corresponds with the increased L2 errors around the same
17

---

EU
TU
L2
GIMZZ
60000
COOTO
11
Dropont
0000
0.004
DPS-UQ
1.75
0009
0005
1.35
HANK
HyperDM
60003
690
0092
0X00
0001
Figure 9: Uncertainty decomposition on CT data. Total uncertainty is high near strong features such the spine and lining of the thoracic cavity, which corresponds to the noisy spots in L2 error map around those areas.
Uncertainty
Epistemic
45
Ecgend
Legeud
30-
12 200
10 200
25-
(4) DPS-CQ
20-
30-
0.0
0.0
6/1
0.5
0.2
Legend
Lagrodt
27-1300
log 100
HyperDM
150
uV-230
(P0-0000
Duaiity
30-
-0.01
-0.03
0.10
Varanse
Figure 10: Varying epistemic uncertainty. Aleatoric and epistemic uncertainty estimates predicted by (a) DPS-UQ and (b) HyperDM when trained dataset sizes [D]
[100, 200, 400, 800].
18

---

Aleatoric Uncertainty
Episternic Uncertainty
2,0
6.0
0.16
(0) DPS-UQ
12.5
0.54
1.25
1.00
7.--
5.-
0.00
-0.4
Lagend
Tagand
of 0.01
1101
HyperDM
Dkcrabte
150
300
5-
0.0
02
0.5
0.6
0.7
-0.04
-0.03
0.00
0.02
0.0c
Strtine
Mron
Figure 11: Varying aleatoric uncertainty. Aleatoric and epistemic uncertainty estimates predicted by (a) DPS-UQ and (b) HyperDM when trained on noisy datasets of [0.01, 0.04, 0.16, 0.64).
19

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
Our claims accurately reflect our contributions.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Please see Section
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
20

---

Section lists assumptions and provides relevant references where proofs can be found.
Justification:
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: Our method is fully described. Anonymized code that can replicate our method is provided in the supplementary materials.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
21

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification: Anonymized code that can replicate our method is provided in the supplemen- tary materials.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips... cc/
oliicy for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
//nips.cc/pblle pwwppppguuuiiieeeep hetpc///uuudess/eeeuuuuiiiii SooodeSubiissi.e ssiniiolccyy for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Details can be found in Section
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: Not all our experiments include errors bars.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi-
dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for
example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
22

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful figures symmetric error bars that would yield results that are out of range (e.g. negative rates).
show in tables or
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Our hardware resources are described in Section A
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't It make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification:
Our research is compliant.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- cration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We discuss the positive societal benefits of our work in the introduction and elsewhere. We do not foresee any negative societal impact.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
23

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our uncertainty estimation method does present obvious opportunities for misuse.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
We created all figures ourselves. We used PyTorch, which is disclosed.
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

---

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
batasets.papersstt
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is available online, the authors encouraged to reach out the asset's s creators.
13. New Assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
The anonymized code in the supplement includes readme and requirement
list.
Guidelines:
The answer NA means that the paper does not release assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: humans subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [No]
Justification:
No IRB required. No research with humans.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
25

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
26


Reviews:
[{"review_id": "ND0H17N5Z3", "reviewer": "Reviewer_pwCt", "summary": "Diffusion has been applied in many domains beyond image generation, e.g. weather forecasting and CT reconstruction. Many of these applications are safety-critical as such the model should be capable of expressing uncertainty. As such the submission proposes a method based on the idea of hypernet for uncertainty quantification: The method would train a network that maps from noise to the a set of weights of the U-net of the diffusion, then generated weight is then treated a sample from the \"posterior\" and later used to estimate the epistemic uncertainty. In order to estimate aleatoric uncertainty, the method uses MC estimation to estimate var(y | x, phi) for a fixed model weight and then average the variance across all weights generated by the hypernet. The paper then evaluates the method on weather forecasting problem and CT image reconstruction problem: In both setting, the method demonstrates a nice decomposition of aleatoric and epistemic uncertainty.\n\nOverall I find the application interesting and paper is nicely written and presentation is clear. However I find the technical contribution a little limited although the empirical results seem to be promising.", "questions": "- What's the training overhead for the hyper-network? \n\n- How should one set the complexity of the hyper network? If the network is too large / small, would the uncertainty vanish?\n\n- How does the number of samples, i.e. M and N, affect the performance?\n\n- What does BNN mean in the main text? Does it refer to MC-dropout or DPS-UQ or both?", "limitations": "- To estimate epistemic uncertainty, although the method only needs a single model (in contrast to BNNs that requires multiple copies/samples of a model), the method still needs **multiple forward propagation** to estimate the uncertainty, so the part of the computational cost at test time still exists", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The problem studied, uncertainty quantification for diffusion, is important.\n\n- The method proposed is technical sound and it works well in empirical evaluation. \n\n- The experiment and hyper-parameter setting are clearly presented in the appendix.", "weaknesses": "- Error bar / standard deviation is not provided.\n\n- It would be nice if a summarization of runtime comparison can be provided, i.e. total training time, total inference time.\n\n- The technical novelty is not very outstanding (though the application seems to be novel), see the bullet point below for more detail.\n\n- The contribution of the paper is not described in a very precise way: From my perspective, the paper applies existing methods, i.e. Bayesian hyper-network, in a new setting: **diffusion models**. However this is not highlighted in the abstract. The abstract gives readers the impression that the paper is proposing uncertainty quantification methods for generic problems.\n\n- (Minor) Lack of baseline methods: Some strong BNN baselines such cyclical SGLD or linearized Laplace, SWAG are not included. MC-dropout used in the paper is a pretty out-of-date technique for uncertainty quantification problem in my opinion.\n\n- (Minor) In Fig. 1, hyper-network is written as hypernetwork in the figure body (a), it would be better if the term can be spelled in a consistent way.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful feedback. We have summarized and responded to your questions and concerns below. Please let us know if you have any additional questions or comments and we will do our best to respond promptly.\n\n**1. Provide error bars / standard deviations.**\n\nWe have trained five sets of ensembles and computed the variance across the EU and AU estimates. The uncertainty estimates are quite consistent, but due to space constraints we are not able to fit these figures in our response. We will include them in the revised text. We have also provided RMSE errors and variances (i.e., total uncertainty) for our experimental results in Figures 8 and 9 of the supplement.\n\n**2. Provide a summary figure comparing the computational overhead of baselines and the proposed method during training and evaluation.**\n\nWe have added a figure in our global response that shows the computational runtime required by all methods for training and evaluation.\n\n**3. The technical novelty of the paper is not described precisely. The abstract should highlight the application of existing methods in a new setting.**\n\nThank you for the suggestion. We have described the technical novelty and specific contributions of our paper in Question 2 of the global response.\n\n\n**4. What's the training overhead for the hyper-network? How should one set the complexity of the hyper network? If the network is too large / small, would the uncertainty vanish?**\n\nWe chose to set the hyper-network just large enough that it produces reasonable weights for the diffusion model. In doing so, we avoid adding a significant increase in the number of trainable parameters. Specifically, this amounted to around a 10.05% increase in training time (i.e., from 44.1 min to 48.5 min) for our weather forecasting and CT reconstruction experiments. If the hyper-network architecture is too small, we found that it outputs poor quality weights for the diffusion model. This text will be incorporated into the revised paper.\n\n**5. How does the number of samples, i.e. M and N, affect performance?**\n\nIn general, we found that under-sampling M (i.e., the number of samples from the implicit weight distribution) leads to uncertainty maps which underestimate uncertainty around out-of-distribution features and overestimate uncertainty around in-distribution features. As we continually sample network weights, we observe increased uncertainty in areas around the abnormal feature and suppressed uncertainty around in-distribution features. Similarly, under-sampling N (i.e., the number of samples from the likelihood) leads to irregular peaks in the predicted aleatoric uncertainty maps. As we sample more from the diffusion model and the sample mean converges, the aleatoric uncertainty becomes more uniform. Please see our ablation study in Section B.1 of the supplement for additional figures and details. \n\n**6. What does BNN mean in the main text? Does it refer to MC-dropout or DPS-UQ or both?**\n\nBNN stands for Bayesian neural network. This acronym is specified in Section 2.1, and the network model for the BNN used in our experiment is described in Section A of the supplement. We have attempted to clarify this by placing a reference in the toy experiment which points to the definition of BNNs in Section 2.1. \n\n**7. Provide stronger BNN baselines (e.g., cyclical SGLD, linearized Laplace, SWAG).**\n\nIn our non-toy experiments, we did not include a BNN baseline because we wanted to focus on evaluating our technique against methods which are able to predict both AU and EU for a more direct comparison. We will include SQR [B] and Kendall et al. [A] as additional baselines in our benchmark. \n\n\n**8. In Figure 1, hyper-network is written as “hypernetwork” in the figure body. The term should be spelled in a consistent way.**\n\nThank you for bringing this to our attention. We have corrected the text in Figure 1 to more consistently reflect the spelling of the term “hyper-network” throughout the paper.\n\n---\n\n**References**\n\n[A] Kendall, Alex, and Yarin Gal. \"What uncertainties do we need in bayesian deep learning for computer vision?.\" Advances in neural information processing systems 30 (2017).\n\n[B] Tagasovska, Natasa, and David Lopez-Paz. \"Single-model uncertainties for deep learning.\" Advances in neural information processing systems 32 (2019).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "hC5zlf3nib", "reviewer": "Reviewer_YZoM", "summary": "This paper proposes HyperDDPM, a novel uncertainty quantification method for generative tasks that uses a hypernetwork and a denoising diffusion probabilistic model (DDPM) and outputs both aleatoric and epistemic uncertainty estimates. HyperDDPM is evaluated on a toy task with available ground-truth uncertainties as well as two real-life tasks, CT reconstruction and weather forecasting. The authors find that HyperDDPM gives accurate uncertainty estimates that are on par with deep ensembles.", "questions": "- Why should one trust their aleatoric uncertainty estimates on OOD data (Fig. 2)? Usually, a two-step procedure is carried out (see Fig. 3 in Mukhoti et al., 2023; CVPR) where the AU estimates are only considered if the EU estimates are below a certain threshold. It also seems that at OOD regions, the AU and EU estimates are highly correlated.", "limitations": "The authors address the limitations of their work adequately.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- Uncertainty quantification in generative models is a high-impact research area.\n- The paper is easy to read.\n- HyperDDPM is a single-model uncertainty estimator that performs on par with ensembles that are SotA in many uncertainty application domains.\n- The use of hypernetworks for uncertainty quantification is an interesting avenue.\n- The experiments are extensive and clearly explained. The toy experiment allows for fine-grained control and comparison with ground-truth quantities, whereas the real-world tasks of CT reconstruction and weather forecasting are important application areas. The authors provide both quantitative and qualitative evaluation.", "weaknesses": "- The paragraph starting at L29 is (i) misleading and (ii) vague. (i) A useful uncertainty estimate does not necessarily have to differentiate between aleatoric and epistemic uncertainty. A total/predictive uncertainty estimate subsumes both of these sources of uncertainty and can be excellent at predicting the underlying model's correctness on an input $x$ (e.g., see Fig. 6 in (Mukhoti et al., 2023; CVPR)). (ii) The \"valuable insights into the strengths and weaknesses of a predictive model\" are not elaborated, neither are the pathways these insights offer \"towards improving [the model's] performance\". I _do_ agree that uncertainty disentanglement is an important and interesting research direction but this paragraph doesn't quite answer _why_. Consider providing concrete use cases for the separate aleatoric and epistemic estimates and how they can improve the model's performance.\n- A more complex model class generally does not lead to a decrease of epistemic uncertainty (Section 3.2). Larger models introduce more degrees of freedom into the learning problem, leading to a larger set of models consistent with the training data (as captured by the Bayesian posterior $p(\\phi \\mid \\mathcal{D})$).\n- The claim that \"This estimate converges to the true AU of the inverse problem as $N \\to \\infty$\" is imprecise and unproven. The statement assumes that (i) the score estimator $s_\\phi$ is expressive enough to model the true score of the generative process perfectly, (ii) $s_\\phi$ is trained on the entire data distribution (i.e., the expectation in Eq. (5) is optimized), and (iii) the global minimum is reached during optimization. Only then can one argue that the true AU of the inverse problem is reached at an _in-distribution_ input $x$ when one samples infinitely many samples from the implicit distribution.\n\n**Minor Issues:**\n- Consider making the figures vector-graphical. The current PNGs are quite pixelated.\n- The precise formulation of Eq. (7) would be $\\mathbb{E}_{(x, y) \\sim \\mathcal{D}}\\left[\\mathbb{E}_{z \\sim \\mathcal{N}(0, \\sigma^2)}\\left[\\mathcal{L}(f(y \\mid \\phi(z)), x)\\right]\\right]$. The sampling is not captured by the original formula.\n- Consider moving Appendix C into the main paper in the final version.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful feedback. We have summarized and responded to your questions and concerns below. Please let us know if you have any additional questions or comments and we will do our best to respond promptly.\n\n**1. The paragraph starting at L29 is misleading and vague. A useful uncertainty estimate does not necessarily have to differentiate between aleatoric and epistemic uncertainty. Also, consider providing concrete use cases for the separate aleatoric and epistemic estimates and how they can improve the model's performance.**\n\nThis statement was indeed too strong. We have changed “To be useful…” to “To be most useful…” and added the following motivating text to the paper: “In applications such as weather forecasting, epistemic uncertainty can be used to inform the optimal placement of new weather stations. Additionally, in medical imaging, decomposition of uncertainty into its aleatoric and epistemic components is important for identifying out-of-distribution measurements where model predictions should be verified by trained experts.”\n\n**2. More complex models introduce more degrees of freedom into the learning problem, leading to a larger set of models consistent with the training data and an increase, not decrease, of epistemic uncertainty.**\n\nWe agree with the reviewer’s statement and will remove the referenced statement from the main text. \n\n\n**3. The claim that the “estimate converges to the true aleatoric AU of the inverse problem” is imprecise. This is only true under the assumption that (i) the score estimator is expressive enough to model the true score of the generative process perfectly, (ii) is trained on the entire data distribution, and (iii) the global minimum is reached during optimization.**\n\nThank you for pointing this out. We will specify the exact assumptions under which our statement is true, as listed by the reviewer.\n\n**4. Why should we trust AU estimates when EU is high? Prior work suggests that we should only consider AU when EU is below a threshold.**\n\nOur results illustrate that our single-model ensembling approach produces similar estimates of AU and EU as multi-model deep ensembling approaches. It's true$\\textemdash$and important to acknowledge$\\textemdash$that AU estimates are only meaningful when there's sufficient data that EU estimates are low. We will highlight this point in the revision.\n\n**5. The precise formulation of Eq. (7) would be $\\mathbb{E}\\_{(x, y) \\sim \\mathcal{D}}\\left[\\mathbb{E}\\_{z \\sim \\mathcal{N}(0, \\sigma^2)}\\left[\\mathcal{L}(f(y \\mid \\phi(z)), x)\\right]\\right]$. The sampling is not captured by the original formula.**\n\nThank you for the correction. As suggested, we have updated the formula in Equation 7 to more accurately denote the sampling of noise vectors $z\\sim\\mathcal{N}(0,\\sigma^2)$. \n\n**6. Move Appendix C into the main paper for the final version.**\n\nThanks for the suggestion. We have integrated Appendix C into the revised paper. \n\n\n**7. Make figures vector-graphical.**\n\nThank you for the suggestion. We have updated all figures in the revision to use vector graphics.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "WVSFaXWMGb", "reviewer": "Reviewer_cmte", "summary": "The paper proposes a approach, HyperDDPM, to estimate both aleatoric and epistemic uncertainty with an ability to disentangle them. This result is achieved by combining diffusion networks (DDPM) to sample predictions from single set of model weights, and Bayesian Hyper Network to generate sets of weights for main DDPM model. Using synthetic data the authors show, that proposed approach can estimate aleatoric and epistemic components of predictive uncertainty, and provide evidence for real-world applicability on medical imaging and weather forecasting tasks.", "questions": "1. In explanation of eq. 1 the authors claim that likelihood under the integration contains information about aleatoric uncertainty, which can be argued as true, but it seems that marginal distribution on the left side of the equation should provide a better estimate of aleatoric component, given that it's marginalized over model weights distribution, and thus epistemic component is eliminated. This can be further confirmed by the way the authors approach estimating aleatoric uncertainty in practice, described in eq. 10 - here the authors take expectation over the model weights in the outer operator.\n\n2. In Appendix A authors describe training of BHN as:\n\t- compute an L2 loss, and back-propagate the gradients to update the hyper-network weights.\n   But in the next paragraph the following is said:\n    - The BHN is trained with the same L2 loss and an additional Kullback-Leibler (KL) divergence loss\n\n   How exactly was BHN training performed? It is also quite interesting why BHN training did not collapse to an optimal mode with very low variance over produced model weights. How often was input noise vector for BHN changed during training? On per-batch level, per-input, or using some other strategy?\n\n3. Toy experiment on synthetic data is only performed to estimate AU and EU when corresponding uncertainty component is changing. It seems to not fully support claim that proposed approach can disentangle AU from EU. It would be much more demonstrative of claim, if the experiment was done by independently varying noise level and train dataset size, and plotting results on AU and EU axes.\n\n4. Real world experiments only show base performance of proposed architecture, but do not provide reasonably in-depth analysis of UE capabilities on such task, apart from figures with uncertainty maps over generated images. The paper would benefit greatly from some form of analysis of how uncertainty estimates improve prediction/reconstruction quality when filtering out low-confidence inputs (i.e. rejection verification).\n\n5. Was training of base model of MC-Dropout baseline performed using active dropout masks in the same way as during inference? Some degree of performance hit that MC-Dropout model suffered could be explained by difference between training and evaluation dropout regimes.", "limitations": "The authors have adequately addressed limitations of their work.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The paper presents a new and potentially useful way of producing ensembled predictions. \n2. On synthetic data it shows that this approach is sensitive to both epistemic and aleatoric uncertainty components when ground truth influence of such components is isolated.\n3. It shows on real world tasks, like medical image reconstruction and meteorological forecasting, that proposed architecture can outperform some, and perform on par with the other baseline methods.\n4. The authors provide codebase used to train and evaluate models, which allows for better transparency and reproducibility.\n\n[After rebuttal comment] I increased my score from 3 to 5. I am still not convinced that uncertainty is treated in coherent way as the paper jumps from the Bayesian predictive distribution to the law of total variance without any connection betweeen them. Additionally, real world experiments do not evaluate uncertainty numerically.", "weaknesses": "Here I summarize weaknesses of the paper, some details on the weaknesses are clarified by the Questions section of the review.\n0. The paper lacks a Contributions section, clearly and concisely stating which claims and results within the paper are new.\n1. The experimental section does not provide sufficient evidence of the ability of the proposed approach to disentangle epistemic and aleatoric components of uncertainty.\n2. Theoretic justifications do not fully align nicely with the engineering approach taken for experiments, and contain questionable claims.\n3. Some details on training procedure are unclear.\n4. The paper lacks a straightforward comparison of computational overhead of baselines and proposed method, during training and evaluation. Some of these are present scattered within paper text, but the paper would benefit from having them collected in one place, given that both baselines and proposed method incur significant overhead.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful feedback. We have summarized and responded to your questions and concerns below. Please let us know if you have any additional questions or comments and we will do our best to respond promptly.\n\n**1. The paper doesn’t have a dedicated contributions section.**\n\nThank you for the suggestion. We have described the technical novelty and specific contributions of our paper in bullet #2 of our global response.\n\n**2. The paper lacks a straightforward comparison of computational overhead of baselines and the proposed method during training and evaluation.**\n\nWe have added a table in our global response that shows the computational runtime required by all methods for training and evaluation. This result will be added to the main paper.\n\n**3. In Equation 1, the marginal distribution on the left side of the equation should provide a better estimate of the aleatoric component, given that it's marginalized over model weights distribution. This can be further confirmed by the way the authors approach estimating aleatoric uncertainty in practice, described in Equation 10.**\n\nWe acknowledge that future work should explore a better estimate of aleatoric uncertainty in Equation 1 and will mention this in the conclusion. However, in the main paper, we derive Equation 1 from Ekmekci et al. [9], which is supported by numerous works [24, A, B], in which the predictive distribution contains both an aleatoric and epistemic component. Likewise, our derivation for Equation 10 stems from prior works [49, C, D] which apply the law of total variance to disentangle epistemic and aleatoric uncertainties.\n\n\n**4. In Appendix A, authors describe training of BHN as: compute an L2 loss, and back-propagate the gradients to update the hyper-network weights. But in the next paragraph the following is said: \"The BHN is trained with the same L2 loss and an additional Kullback-Leibler (KL) divergence loss.\"**\n\nWe apologize for the typo in Appendix A, which has been corrected in the text: The Bayesian neural network (BNN), not Bayesian hyper-network (BHN), is trained with an additional KL divergence loss.\n\n**5. How is BHN training performed? How often is the noise vector changed during training (e.g., per-input, per-batch, etc.)? Why did the BHN not collapse to a small optimal mode?**\n\nThe BHN is trained only with an L2 loss. During training, the noise vector is changed per-batch, and for the majority of our experiments we set the batch size to 32. The BHN doesn’t collapse to a mode because there are a variety of network weights that yield plausible predictions with respect to L2 distance. \n\n**6. The toy experiment should be done by independently varying noise level and training dataset size and plotting results on AU and EU axes.**\n\nThis is a great suggestion. We  have provided these figures in the general response PDF, which show that our method performs similarly to deep ensembles. We note, however, both methods struggle to fully disentangle EU and AU when one type of uncertainty is very high. In Figure R1, both DPS-UQ and HyperDDPM estimate a similar AU across different dataset sizes, indicating that the AU estimate is relatively unaffected by the increasing EU of the problem. Similarly, in Figure R2, both methods estimate a similar EU across different noise levels, except in the extreme noise scenario. Prior work [E] suggests that one should disregard the AU estimate when EU is high (i.e., the measurement is out-of-distribution). We will incorporate these figures and corresponding descriptions into the main text.\n\n**7. The paper would benefit from a more in-depth analysis on real world problems. For example, how can EU help improve reconstruction quality (e.g., by filtering out low-confidence outputs / rejection verification)?**\n\nWe will update the main text to provide more concrete examples of use cases for EU and AU estimation (e.g., active learning for weather station placement, predicting long-tailed events with high EU, and cost mitigation strategies). We note that our current experimental results demonstrate that our approach can be used for rejection verification. For instance, in both real-world experiments, we provide in-distribution and out-of-distribution measurements to the model, and we are able to identify which parts of the prediction are unreliable and should be investigated more thoroughly by a trained expert.\n\n\n**8. Were the drop-out masks kept consistent during training and evaluation of the MC Dropout baseline? Any differences could explain the performance drop.**\n\nWe generate random activation masks with the same drop-out probability $p=0.3$ for both training and evaluation. During training, the activation masks are randomly generated per-batch. At evaluation time, we use the same procedure to randomly generate $M=10$ masks and compute AU and EU according to Equations 10 and 11. We will clarify this procedure in the revised paper.\n\n---\n**References**\n\n[A] Fellaji, Mohammed, and Frédéric Pennerath. \"The Epistemic Uncertainty Hole: an issue of Bayesian Neural Networks.\" arXiv preprint arXiv:2407.01985 (2024).\n\n[B] Kwon, Yongchan, et al. \"Uncertainty quantification using Bayesian neural networks in classification: Application to biomedical image segmentation.\" Computational Statistics & Data Analysis 142 (2020): 106816.\n\n[C] Schreck, John S., et al. \"Evidential deep learning: Enhancing predictive uncertainty estimation for earth system science applications.\" arXiv preprint arXiv:2309.13207 (2023).\n\n[D] Joshi, Shalmali, Sonali Parbhoo, and Finale Doshi-Velez. \"Pre-emptive learning-to-defer for sequential medical decision-making under uncertainty.\" arXiv preprint arXiv:2109.06312 (2021).\n\n[E] Mukhoti, Jishnu, et al. \"Deep deterministic uncertainty: A new simple baseline.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "XsaL6bWATG", "reviewer": "Reviewer_zgME", "summary": "The work applies hypernetworks to estimate both aleatoric and epistemic uncertainty in the context of diffusion models. The authors leverage the distribution on weights created by the hypernetwork to estimate uncertainty via the Total Uncertainty (TU) decomposition into Aleatoric Uncertainty (AU) and Epistemic Uncertainty (EU). They validate their method on a toy dataset, lung scan dataset, CT scan dataset and weather predictions.", "questions": "How does the scalability of hypernetworks compare to that of MC Dropout? In both instances, there are challenges with scaling the processes to networks that require a lot of parameters.", "limitations": "The authors address their limitations in the appendix.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. The paper is clear and well-written.\n2. The authors provide a solution to an important topic, given the prevalence of generative models.\n3. The authors apply their method to both a straightforward toy problem and real-life datasets.", "weaknesses": "1. The authors seem to be missing a citation where previous researchers have estimated epistemic uncertainty for diffusion models.\n    - Berry, Lucas, Axel Brando, and David Meger. \"Shedding Light on Large Generative Networks: Estimating Epistemic Uncertainty in Diffusion Models.\" The 40th Conference on Uncertainty in Artificial Intelligence.\n2. There are claims regarding ensembles that appear incorrect. It is not necessary to train $M$ distinct networks; many modern methods work by ensembling certain parts of the network while sharing weights across the rest. Thus, training ensembles is not as expensive as stated.\n    - Osband, Ian, et al. \"Deep exploration via bootstrapped DQN.\" Advances in Neural Information Processing Systems 29 (2016).\n    - Berry, Lucas, and David Meger. \"Normalizing flow ensembles for rich aleatoric and epistemic uncertainty modeling.\" Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 37. No. 6. 2023.\n3. Minor writing comments:\n    - The abstract is a bit unusual with two paragraphs and could be shortened.\n    - The notation in section 4.2 seems overloaded. Specifically, the subscript on $x$ refers to both the denoising steps and the samples in the training set.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful feedback. We have summarized and responded to your questions and concerns below. Please let us know if you have any additional questions or comments and we will do our best to respond promptly.\n\n**1. Missing citation where diffusion models are used to estimate epistemic uncertainty.**\n\nThank you for bringing this to our attention. We weren’t able to include a citation to DECU [C] in our original submission since it was published after the NeurIPS submission deadline. However, we will certainly update our paper and include a reference and discussion in Section 2.3.\n\n**2. Modern methods ensemble only certain parts of the network, so training ensembles is not necessarily as expensive as stated.**\n\nWe agree with the reviewer; this statement should be further clarified. We have attempted to make this point more apparent by adding references to relevant ensembling papers [A, B] in Section 2 and updating our statements in Section 5 to state that our method achieves a reduction in training cost up to $M\\times$ that of deep ensembling methods, depending on the number of ensembled parameters.\n\n\n**3. How does scaling of the BHN compare to MC Dropout? Both have issues with scalability.**\n\nIn our experiments, we chose to set the BHN just large enough that it produces reasonable weights for the diffusion model. In doing so, we avoid adding a significant increase in the number of trainable parameters. Our BHN incurred only a moderate 10% overhead compared to a baseline diffusion model; from 44.1 min to 48.5 min. In comparison, we observed a slightly smaller training overhead of 6.64% with MC-Dropout;  from 44.1 min to 47.0 min.\n\n\n**4. The subscript on $x$ is overloaded. It refers to both the number of denoising steps and the number of samples in the training set.**\n\nThank you for pointing this out. We have clarified this notation in the main text by denoting the denoising step $t$ with a superscript and the dataset index $i$ with a subscript. Specifically, a sample $i$ from the dataset at denoising step $t$ is now expressed as $x_i^{(t)}$.\n\n---\n\n**References**\n\n[A] Osband, Ian, et al. \"Deep exploration via bootstrapped DQN.\" Advances in neural information processing systems 29 (2016).\n\n[B] Berry, Lucas, and David Meger. \"Normalizing flow ensembles for rich aleatoric and epistemic uncertainty modeling.\" Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 37. No. 6. 2023.\n\n[C] Berry, Lucas, Axel Brando, and David Meger. \"Shedding Light on Large Generative Networks: Estimating Epistemic Uncertainty in Diffusion Models.\" The 40th Conference on Uncertainty in Artificial Intelligence.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank the reviewers for their thoughtful and constructive feedback. We are glad they found our proposed method novel (YZoM, cmte), performant against the current state-of-the-art (pwCt, YZoM, cmte), easy-to-understand (pwCt, YZoM, zgME), and well-supported by experiments (pwCt, YZoM). We have summarized and responded to common questions and concerns below.

**1. (pwCt, cmte) Provide a summary figure comparing the training and inference times of the proposed method and baselines.**

The attached PDF provides a summary figure which compares the training and inference times required by our method, MC dropout, and DPS-UQ. We will add this table to the revised text. 

**2. (pwCt, cmte) Specify contributions explicitly / provide a contributions section.**

The technical novelty of our paper is application of Bayesian hyper-networks in a novel setting (i.e., diffusion models) to estimate both epistemic and aleatoric uncertainties from a single model. The abstract has been revised as follows: “estimate epistemic and aleatoric uncertainty with a single model” → “estimate epistemic and aleatoric uncertainty with a single diffusion model.” Furthermore, our contributions can be summarized as follows (this text will be added as a contribution section to the manuscript):
- We apply Bayesian hyper-networks in a novel setting, diffusion models, to estimate both epistemic and aleatoric uncertainties from a single model.
- We conduct a toy experiment with ground truth uncertainties and show that the proposed method accurately predicts both sources of uncertainty.
- We apply the proposed method on two mission-critical real-world tasks, CT reconstruction and weather forecasting, and demonstrate that our method achieves a significantly lower training overhead and better reconstruction quality compared to existing methods.
- We conduct ablation studies investigating the effects of ensemble size and the number of ensemble predictions on uncertainty quality, which show (i) that larger ensembles improve out-of-distribution detection and (ii) that additional predictions smooth out irregularities in the aleatoric uncertainty estimates.

Meta-review: This paper proposes a Hyper-network approach that applied to diffusion models. By leveraging the sampling ability of diffusion model, the proposed approach claims to separate the epistemic and aleatoric uncertainty. Although the technical contribution is limited, it shows some preliminary results. The paper will be stronger if more evidence is shown.
Decision: Accept (poster)