Distribution-Free Data Uncertainty for Neural Network Regression
Authors: Domokos M. Kelen
Ádám Jung
Péter Kersch
Andras A Benczur
Venue: ICLR 2025 Poster (2025)
Keywords: deep learning
uncertainty quantification
regression uncertainty
aleatoric uncertainty
scoring rules
continuous ranked probability score
Primary area: probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
TL;DR: We propose a distribution-free neural network regression approach that learns aleatoric uncertainty through sample-based CRPS optimization.

Abstract: Quantifying uncertainty is an essential part of predictive modeling, especially in the context of high-stakes decision-making. While classification output includes data uncertainty by design in the form of class probabilities, the regression task generally aims only to predict the expected value of the target variable. Probabilistic extensions often assume parametric distributions around the expected value, optimizing the likelihood over the resulting explicit densities. However, using parametric distributions can limit practical applicability, making it difficult for models to capture skewed, multi-modal, or otherwise complex distributions. In this paper, we propose optimizing a novel nondeterministic neural network regression architecture for loss functions derived from a sample-based approximation of the continuous ranked probability score (CRPS), enabling a truly distribution-free approach by learning to sample from the target's aleatoric distribution, rather than predicting explicit densities. Our approach allows the model to learn well-calibrated, arbitrary uni- and multivariate output distributions. We evaluate the method on a variety of synthetic and real-world tasks, including uni- and multivariate problems, function inverse approximation, and standard regression uncertainty benchmarks. Finally, we make all experiment code publicly available.

Published as a conference paper at ICLR 2025
DISTRIBUTION-FRE DATA UNCERTAINTY FOR NEURAL NETWORK REGRESSION
Domokos M. Kelen"1 Ádám Jung
Péter Kersch
András A. Benczúrit
ABSTRACT
Quantifying uncertainty is an essential part of predictive modeling, especially in the context of high-stakes decision-making. While classification output includes data uncertainty by design in the form of class probabilities, the regression task generally aims only to predict the expected value of the target variable. Prob- abilistic extensions often assume parametric distributions around the expected value, optimizing the likelihood over the resulting explicit densities. However, using parametric distributions limit practical applicability, making it difficult for models to capture skewed, multi-modal, or otherwise complex distributions. In this paper, we propose optimizing novel nondeterministic neural network regres- sion architecture for loss functions derived from sample-based approximation of the continuous ranked probability score (CRPS), enabling truly distribution- free approach by learning to sample from the target's aleatoric distribution, rather than predicting explicit densities. Our approach allows the model learn well- calibrated, arbitrary uni- and multivariate output distributions. We evaluate the method on variety of synthetic and real-world tasks, including uni- and multi- variate problems, function inverse approximation, and standard regression uncer- tainty benchmarks. Finally, we make all experiment code publicly availabld"
INTRODUCTION
With the continued development of machine learning, more and more industries consider using ad- vanced predictive modeling to influence automated or human made decisions. Modern methodolo- gies such deep learning offer unparalleled accuracy for modeling complex real-world processes, with the potential to greatly enhance efficiency in many applications. Uncertainty quantification (UQ) an essential part of predictive modeling, especially in applications where the prediction has the potential to influence high-stakes decision making. A large body of research deals with predicting the target value along with its uncertainty, such as Bayesian Neural Networks (BNN).
In prediction task, uncertainty comes from multiple different sources. BNN-s are primarily con- cerned with epistemic (or knowledge) uncertainty, the uncertainty how a model should predict the future based on the limited data we have at disposal. In use-cases where data is limited, such approaches are invaluable for acknowledging potential blind spots in the modeling process.
However, in many industry applications, data is abundant. Still, different kind of uncertainty remains even when data is plentiful: aleatoric (or data) uncertainty arises from constraints in the problem setup, where the model lacks information to make exact prediction. Most tasks involve key pieces of information missing, factors which influence the target, yet ones we cannot provide to the model inputs. Either due inherent randomness unobservable latent variables, in most real-world scenarios it is theoretically impossible to always predict the exact target value. As an inherent constraint, alcatoric uncertainty cannot be reduced with additional data of the same kind.
Classification tasks are capable of expressing aleatoric uncertainty by design in the form of class probabilities. However, the case of regression, the models traditionally only output a single value, or simple parametric distribution at best. An expectation of 10mm of rainfall could indicate relative certainty of rainy day with actual 10mm of rainfall, but could also indicate 10% chance of deadly
"Emails: [kdomokos, jungadam, benczur (anczur]@info.iiin ilab.o.i hu, peter.kersch @ericsson.com
THUN-REN SZTAKI *Ericsson Hungary 1Széchenyi University, Györ, Hungary https://gi thwb.

---

Published as a conference paper at ICLR 2025
storm with 100mm of rain. In such cases, heteroscedastic distributional regression model with a simple unimodal output lacks the nuance required for informed decision making.
In this work, propose novel nonparametric way to model alcatoric uncertainty, relaxing the usual constraints of simple parametric distributions such as unimodality, and enabling the approxi- mation of arbitrary distributions. We introduce new nondeterministic neural network architectures to generate samples from the aleatoric distribution of the target, and train the resulting models by draw- ing upon the theory of scoring rules, originally developed for evaluating probabilistic forecasts in fields such as meteorology and economics. We derive a novel, sample-bar hasple-ased loss function based on an unbiased (n log n) formulation of the empirical continuous ranked probability score (CRPS), a strictly proper scoring rule for regression, describing both unweighted and weighted variants.
The resulting method is capable of learning arbitrary probability distributions from sample data in a well-calibrated manner, no longer being restricted to simple parametric uni-modal distributions, the same time remaining conceptually simple and easy to implement. Further, generalizes for multivariate regression, and is capable of learning the joint distribution of hundreds of output values. We evaluate our approach on multiple tasks, including synthetic uni- and multivariate examples. the inverse problem of MNIST classification, and also on the standard UCI regression uncertainty benchmark. The method consistently exhibits favorable behavior in practice, making an appealing choice for modeling alcatoric uncertainty. Our contributions can be summarized as follows:
We propose multiple novel architectures of nondeterministic neural networks for producing samples from the alcatoric target distribution with efficient optimization procedure.
We derive loss function for optimizing neural networks using a novel unbiased empirical approximation of the CRPS, describing the loss for both unweighted and weighted samples.
Finally, we extensively test the resulting method, comparing it against prior work such as BNN-s Gawlikowski et al. 2023) or Mixture Density Networks (MDN) (Bishop 1994).
Scaling. Our approach scales well to larger network sizes. While part of the model is evaluated multiple times, the incurred cost scales with target rather than input complexity, see Appendix C.8 Our experiments show that our models represent wide range of complex distributions, using relatively small nondeterministic part considering modern deep learning hardware and model sizes.
BACKGROUND
2.1 TYPES OF UNCERTAINTY
In prediction tasks, different sources of uncertainty can be distinguished. This section briefly sum- marizes the two main types we focus on; for a general overview, see (Gawlikowski et al. (023).
Generally, the data we record and collect comes from real-world process that is only partially observed. In regression, the target value (i.e., the value that we aim to predict) can be modeled as random variable Y Ω + P. where elementary events E Q are assumed to represent the true state the world. Although we are unable to directly observe w, we can observe a number of feature variables X Ω - R which give partial information about w. The classic regression task is then defined as approximating the conditional expectation E|y X = x, D), where the data D is sample of past values of (X, Y). In multivariate regression, Y takes on values from R* Uncertainty quantification extends the regression task so that we are no longer simply interested in the conditional expectation, but rather the entire conditional distribution of y, i.e., p(y X x. D).
Alcatoric uncertainty (Kendall & Gal 2017) arises from X giving us limited information about w. We can express alcatoric uncertainty through the conditional cumulative distribution function
Fy x (vo) = / H(Y(W) < yor p(w X)
Many constraints can be modeled through aleatoric uncertainty, for example measurement error, or uncertainty due to unobservable latent variables that affect Y. This kind of uncertainty cannor be reduced with more data (i.e., larger sample D), as the uncertainty arises from constraints in the problem setup. In other words, aleatoric uncertainty is the uncertainty that remains even if we assume to have perfect knowledge of the relationships between X, Y, and w.
2

---

Published as a conference paper at ICLR 2025
Epistemic uncertainty the other hand stems the fact that we have no perfect knowledge of the relationship of X, Y, and w. Equation ① is described in idealistic terms, and in fact neither term of the integration is known in practice, our only information about X and Y comes from the sample D. Epistemic uncertainty is then modeled as the uncertainty of selecting f such that / (v) ~ p(y X = xt, D), with larger dataset D implying less uncertainty. The majority of uncertainty research is focused on epistemic uncertainty, for example the research of BNN-s, see Section 2.2
In classification, aleatoric uncertainty is expressed through class probabilities, while regression tra- ditionally focuses on point estimates like the expectation. In probabilistic regression, aleatoric re- gression uncertainty is often modeled using parametric distributions, with the normal distribution being particularly common choice (Jospin et al. 2022] Abdar et al., 2021] Magris & losifidis 2023). However, parametric distributions are limited their expressive power, and the wrong choice of distribution may undermine the entire learning process. Nevertheless, little attention is usually paid to the representation of the distribution of Y, with most uncertainty-related work either simply assuming the existence of likelihood function or explicitly predicting the parameters of normal distribution. This oversight persists despite the fact that that arbitrary-shaped aleatoric distributions can arguably be learned even when only single value is recorded for each input X, by leveraging the same implicit assumptions of smoothness that supervised learning generally relies on.
2.2 BAYESIAN NEURAL NETWORKS
Bayesian neural networks (or BNN-s) (Blundell et al. 2015) are nondeterministic neural networks whose stated goal is to quantify epistemic uncertainty. Using the terminology of Section [2.1] BNN-s model the conditional density of Y through neural network parameterized by A E θ, i.e., fo(y) ~ ply X = x. D). Epistemic uncertainty is then expressed as the uncertainty of O, i.e.,
ply X = x. D) p(y 0, X x(0 D),
(2)
where the first term being integrated is the aleatoric likelihood, in practice output by the neural network after having sampled according to the second term. A large body of research deals with the theory of the optimization of BNN-s, see (Gawlikowski et al. 2023) for an overview.
Each evaluation
BNN yields a separate aleatoric uncertainty distribution, while epistemic un- certainty is captured by the nondeterminism of the network, with subsequent runs yielding different aleatoric uncertainty distributions sampled from the epistemic uncertainty distribution. In other words, epistemic uncertainty can be interpreted as the uncertainty of aleatoric uncertainty. Accord- ingly, they are sometimes referred to as first- and second-order uncertainties (Bengs et al. 2023).
Generally, aleatoric uncertainty is expressed in the form of a parametric distribution, while epistemic uncertainty is usually treated as nonparametric: while the distribution of the weights @ is individually assumed to be parametric, the resulting distribution of the output of the whole network can take arbitrary shapes. In this paper, we propose modeling both kinds as nonparametric distributions.
2.3 PROPER SCORING RULES
When model outputs distribution, scoring rules be used to quantify the accuracy of predic- tions based on a sample from the target, i.e., they are analogous to metrics like RMSE and AUC but specifically for probabilistic predictions. Generally, a scoring rule takes a predicted distribution P and sample from the target, and assigns a score S(P, v). Denoting Ey~q [S(P, Y)] as S)P. a scoring rule proper if S(Q.Q) < S(P. Q) for all P. and strictly proper if equality implies P= and vice versa. Strictly proper scoring rules encourage predictors to provide honest predictive dis- tributions that accurately reflect the target's distribution (Gneiting & Raftery, 2007). An example log((())) where f is the density
the logarithmic score (Good. 1952). defined as LogS(F,y) of F, more commonly known in machine learning literature as the negative log- likelihood.
A strictly proper scoring rule for regression is the continuous ranked probability score (CRPS):
CRPS(F. = { (F(x)) 1ly 2})² dz,
(3)
where F is the cumulative distribution function (CDF) of the predicted distribution and y is the target sample. The CRPS can be analytically computed for a number of parametric distributions, see Jordan et al. (2019) for comprehensive list.
3

---

Published as a conference paper at ICLR 2025
2.4 EMPIRICAL CRPS
To approximate Equation pirical CDF of sample. As described by (Gneiting & Raftery 2007). equivalent formulation
3 in practice, we can compute the formula substituting F for the em-
[Baringhaus& & Franz 2004 Székely & Rizzo 2003) of Equation
3 assuming E[F] < co, is
CRPS(F,)) B| yl E|Y Y'I, where Y, F, YLLY'.
(4)
When evaluating Equation 4 for the empirical CDF (FFCDF) (Krüger et al. 2021 of a (Úi...., (.i,....9))))
.9n)
sample distributed according to F, the resulting empirical score CRPS(TFEDDF y) is given by
Σ (1 -) (ni(u= < (10 - i - 2/) (5)
91
91 أ,ة
2n
nt 1=1
While the naive formula on the left hand side is ( complexity, the right hand side gives a relatively well-known (Murphy 1970 Jordan 2016 Laio & Tamea 2007 Jordan et al., (2019) alge-
braically equivalent On n) formula, with (1()) ................................ (()) denoting the ordered sample statistic.
2.5 MIXTURE DENSITY NETWORKS
While most eepertaintt-relatee work does not investigate distribution-fin -ree approaches for aleatoric regression uncertainty, one exception is Mixture Density Networks (MDN) (Bishop) 1994). MDN expresses the density mixture of fixed number Gaussians. As the number of components tends to infinity, a Gaussian mixture is capable of approximating any distribution, therefore we consider MDN one of the strongest prior art in this field, also belonging to the class of models capable of learning aleatoric uncertainty in a distribution-free manner. However, MDN models often have practical limitations. In our experiments, using our own robust implementation we achieve much stronger scores for MDN then previously reported, see Appendix C.2 and the source code for details.
3 RELATED WORK
The literature of uncertainty quantification is vast, with the majority of works focusing on quan- tifying epistemic uncertainty, for example the BNN (Blundell et al., 2015). or its many enhance- ments (Sun et al., 2017, Gawlikowski et al.) 2023). The UCI Benchmark (Hernández--Lobtto & Adams 2015) has become the standard benchmark to measure regression uncertainty, with many papers reporting RMSE and (negative) log- liglliikl (NLL) results along with standard error. See Appendix A. for a list. While certain nonparametric regression approaches, c.g., some variants of as Gaussian Process Regression (Rasmussen] 2003 Sendera et al. (021), are capable of modeling the output in a distribution-Fi free manner, the area of distribution- free approaches for neural network- based aleatoric regression uncertainty is relatively under-researched. Such methods, for example MDN | 1194)), seem to be under-utilized and under-reported in uncertainty-related literature, with most surveys barely mentioning omitting the topic, Appendix A.2 for small survey. MDN itself rarely appears as bascline. We are only aware of (El-Laham et al. 2023) on the UCI benchmark, however with substantially lower scores. See Appendix C.2 for more information.
Minimizing proper scoring rules to learn is common practice, either implicitly or explicitly. Both maximum likelihood for regression and cross entropy for binary classification coincide with min- imizing the logarithmic score (Good 1952), also main component of variational inference for BNN-s (Blundell et al.) 2015). Maximum likelihood estimation with an assumed Gaussian likeli- hood coincides with the Dawid-Sebastian score (Dawid & Sehastiani] 1999 Gneiting & Raftery) 2007 Czado et al.) 2009). Examples of explicitly using scoring rules to learn include Lakshmi-
narayanan et al. (2017); Bengs et al. (2023): Vahidi et al.) (2023); Bouchacourt et al. (2016). Prior
work using the CRPS for learning often does so to learn combination weights (Thorey et al. 2017
Berrisch & Ziel 2023 van der Meer et al. 2024). An analytic expression of the CRPS for the
Gaussian is used for optimization in Rasp & Lerch 2018); Gebetsberger et al. (2018).
The biased sample-bass based On lo CRPS formula is well-known (Murphy 1970 Jordan 2016
Laio & Tamea 2007: Jordan et al. (2019). ) and the bias of the O(n²) formula has also been noted
before (Ferro et al. 2008 Thooey et al. 2017), however knowledge we the first to describe
4

---

Published as a conference paper at ICLR 2025
the unbiased (n log n) formula. Weighted CRPS versions exist (Gneiting & Ranjan 201 Allen 2023 Taggart 2023: Pic et al. 2023), however in the sense of weights over the range of to em- phasize certain target values. In contrast, we introduce weighted- weighted-sample--beee sample-based approximation for the CRPS and, to our knowledge, are first to describe both the (nn²) and the (n log n) formulas.
Non-gaussian output distributions beyond MDN have been studied before, e.g., the method of (De- peweg et al. 2018) is capable of representing complex distributions, including multi-modal ones. In the context of generative modeling, related concepts are, e.g., Normalizing Flows (Ardizzone al. 2018 and Maximum Mean Discrepancy (MMD) (Li et al. 2015 Dziugaite et al. 2015; Du et al.,
2023 Hertrich al. 2024 Cui al.) 2020), which has close connection to kernel scoring rules
(such as the CRPS) Zawadzki & Labaie 2015 Schölkopi 2000 Sejdinovic et 2012)).
Closest to our approach is BBBuchaccoutt al. (2016) (DISCO Nets), a probabilistic method for hand- pose estimation. They also use injected random noise for nondeterminism (as in Section 4.1. and derive a dissimilarity coefficient (Rao) (982) based loss function that reduces to the Energy Score as special case (see Section 4.5), which the authors also highlight. Developed independently, our work focuses on optimizing the CRPS, whose multivariate generalization also coincides with the Energy Score. Our work extends beyond DISCO Nets through an O log loss formula instead of O(n²), weighted and multi-head architectures, explicit handling of epistemic uncertainty, and further adap- tations for traditional regression. We believe the connections between the two approaches serve to further emphasize the broader applicability of the shared ideas behind both.
4 LEARNING TO SAMPLE FROM THE ALEATORIC DISTRIBUTION
We propose optimizing neural networks using CRPS y) from Equation (3) as a loss function. By taking multiple samples from a nondeterministic neural network, we leverage automatic differ- entiation to backpropagate the loss across multiple evaluations at the same time. Note that since the ordered sample is sufficient statistic, the sort operation does not need to be backpropagated.
The effects of Equation (5) interpreted as a loss function are quite intuitive. The first term is equiv- alent to the mean absolute error (MAE), forcing the network to center its predictions around the median of the target distribution. The second term is being maximized due to its negative sign, forc- ing the distribution spread out, increasing its variance. With the coefficient, balance is found exactly when the predicted distribution follows the target distribution (Gineiting & Raftery 2007).
4.1 FAST UNBIASED SAMPLE-BASE CRPS LOSS
The formula for the On log n) sample-based CRPS of Equation (5) relatively well-known. Less widely known is the fact that the formulas of Equation 55) are biased, not unlike the naive estimate for empirical variance. As described by Ferro et al. (2008), Equation (5) underestimates the second term because the diagonal is always constant zero, unlike in the original of Equation (4) As result, the naive formula underestimates prediction variance, making the model underconfident. An unbiased version (Ferro et al.) 2008 Thorey et al. (017) can be calculated by using the coefficient instead of 2n2
We derive the corresponding modification of the On log n.) formula as
2n(n-1)
nnn - Σ (wo) - v) ( < 01) - i+1)
CRPS(( v)
(6)
where CRPS' denotes the unbiased version of the empirical CRPS, see Appendix B.4 for proof.
4.2 OPTIMIZING NEURAL. NETWORKS FOR THE EMPIRICAL CRPS
While any kind of nondeterministic neural network with sufficient expressive power should work in theory, we see no point in, c.g., sampling different weights for the network with each evaluation, as BNN-s do. Instead, propose injecting the randomness through independent standard normal random samples ε concatenated to the input or a hidden layer activation, see Figure Ta Exploring other nondeterminism injection methods is interesting research topic, however is out of scope.
Using the terminology of Section 2.1 the resulting network can be interpreted as follows. First, the network transforms X into latent representation Wj revealing partial information about w.
5

---

Published as a conference paper at ICLR 2025
lost
X
lovi X
Inpurt
bdart repr
Werl repr
Iddert repr
Iotent rope
Soandben)
Stu Mh
(a) Simple
(b) Weighted
(c) Multiple heads
(d) Layered heads
Figure 1: Illustration of nondeterministic neural network architectures. The top (empty) part of each
network is evaluated once, responsible for transforming the input. The yellow (solid) parts of cach network are evaluated multiple times, with being randomly sampled each time.
Second, we sample in place of the unobservable latent variables that, together with X, are assumed to identify w with certainty. Finally, the second part learns the deterministic mapping (w,,e) H> y which draws samples from the target distribution, with random values sampled in each run. In practice, only the second part needs to be evaluated multiple times. For further efficiency, the network can be implemented compute multiple samples in single batch, with output of shape [b x n x k).. where b is batch size, n is the number of samples, and & is the dimension of Y.
In theory, such a network can learn to produce samples from arbitrary probability distributions, and CRPS being a strictly proper scoring rule guarantees that the global optimum corresponds to the real aleatoric distribution under ideal conditions (i.e., assuming infinite data and MLP-s to be universal function approximators). Ultimately, the described setup provides a truly nonparametric way to learn aleatoric uncertainty for neural network regression.
4.3 WEIGHTED CRPS LOSS
The simple architecture described in Section 4.2 needs to model distributions by not only covering the full range of possible outputs depending on the injected randomness, but also making sure that potential outputs get sampled proportionally from the target distribution. Consequently, output re- gions with small probability get sampled infrequently during training, possibly leading to the model being unable to learn them in detail. Further, neural networks have an inherent bias towards learning smooth mappings from input to output, leading to the learned distribution possibly becoming overly smoothed when the network is limited in its capacity cannot be trained until convergence (i.e., carly stopping is used), both of which are common in practical applications.
An implicit bias towards smooth functions can be useful in certain applications, however it is also a limitation of the simple unweighted architecture. To improve the expressivity of the network, we introduce the saapheed---mple--eee architecture which produces a weighted sample as its output. By allowing a weighted sample, we are essentially allowing the model to draw samples (fx) from distribution Q' instead of Q, along with the change-of -fange-o--mmauure measure coefficients w .Q''
CRPS loss (weighted CRPS for short), along with modified
aQ (also known as likelihood-rait ratio or importance weight).
We derive the unbiased approximation of the CRPS in Appendix B.I from weighted sample as
1 n Σ w.ly Vel
CRPS (F v)
2n(n 1) w. 1ū. فzو.
4Q (.). and FEEE denotes the empirical CDF of the weighted sample (D., w.). We where w; aq"
also derive an algebraically equivalent unbiased (n logn) formula, analogous to Equation (6) as
v0nn((( CRPS' (FrEDe y) n(n w() (Din) uu) (n- (n- -
W n w +1
(8)
2
denoting W = E' (() and 8; = EjEn " See Appendix B.2 and B.3 for proof.
4.4 WEIGHTED SAMPLING ARCHITECTURES
While the weighted loss may in itself enhance model expressivity, its more significant impact is enabling advanced network architectures. A simple architecture must evaluate the non-deterministic

---

Published as a conference paper at ICLR 2025
network component to generate new samples, requiring non-negligible compute time for larger sam- ple sizes during both training and prediction. The weighted formula improves sampling efficiency by allowing the network to produce multiple weighted samples in each evaluation (see Figure [e].. Without weighting, such a network would be problematic, as all samples from the same evaluation would share the same frequency in the overall sample.
The weighted formula can leverage multiple such sampling heads while still ultimately being capa- ble of representing arbitrary distributions. This increases expressivity, each head samples from distinct parts of the output distribution, modeling simpler distributions individually. However, early stopping can cause transitions between heads to non- seamless... A convenient solution is to apply the loss function separately to multiple sets of heads (see Figure [d]. resulting in heads with over- lapping ranges. This approach resembles an in-network ensemble, however to avoid confusion with traditional ensembling, we refer to it layered sampling heads. The synthetic task in Section 5.1 serves good visual illustration of the effects of different sampling architectures.
4.5 MULTIVARIATE REGRESSION
The formulas described so far only apply to univariate targets. Fortunately, a multivariate general- ization of the CRPS exists, called the energy score (Gneiting & Raftery 2007 Jordan et al. 2019)
ES((F
Σ ll U112
2n ΣΣ
1b ūs 12,
with the same caveats as Equation regarding complexity and bias. While unbiased and weighted versions can be caleulated analogously, the On no formula is only valid for the univariate case. Potential ways to speed up the calculation do exist, see for example Hertrich et al. (2024). However, investigating the multivariate case in detail is out of scope for this work. Nevertheless, we do conduct experiments on multivariate problems using Equation (9) in its quadratic form, see Section 5.2
4.6 EPISTEMIC UNCERTAINTY
The primary goal of our approach is to learn the aleatoric uncertainty of the target variable Y in frequentist manner, without the use of Bayesian techniques. Although our method could be extended by mixing samples from a prior distribution into the training data as a way to mimic Bayesian priors, as it is sometimes done for example with out-off ddistribbution data (Malinin 2 Lakshminarayanan al. 2017: Chen et al.] 2018 Hafner et al.] 2020), we do not attempt such approaches in this paper. Still, the benchmark used in Section 5.3 involves a large number of relatively small datasets, making it essential to account for epistemic uncertainty. Fortunately, the ensemble-based method of Lakshminarayanan et al. (2017) combines well with our proposed approach, the output of multiple independently trained models can be mixed by simply pooling the samples.
4.7 LIKELIHOOD ESTIMATION
Since our method represents the output distribution through producing samples, it is very easy to estimate, e.g., percentiles. On the other hand, without an explicit density, is nontrivial to evaluate likelihood given point. However, still need to do so compare against prior work, which generally evaluate methods using the negative leglliilldood metric (see Section 5).
In general, sample-based unbiased estimator for the likelihood can exist (Rosenblatt 1956). However, the literature on nomparametric density estimation (Izenman 1991) is vast, with many methods not only guaranteeing convergence, but providing useful estimates smaller sample sizes. Given the ease of sampling in our case, the lack of explicit density in theory becomes a non-issue. In practice, we found Wang Wang (2007) to work well together with out method, see Appendix C.I
EXPERIMENTS
5.1 UNIVARIATE REGRESSION ON SYNTHETIC TASKS
In theory. both MDN and our new CRPS-based networks can learn to represent arbitrary univari- ate distributions. To test their expressiveness visually, train both methods learn a synthetic
7

---

Published as a conference paper at ICLR 2025
Original
(b) MDN
Simple
(d) Weighted
(c) MH
LMH
Figure 2: Learning the Mona Lisa as output distribution after 40 epochs (top) and after 2000 epochs (bottom). Each row of pixels the images displays the heatmap of a predicted density based on the row number as input. MH stands for multiple heads, while LMH stands for layered multiple heads.
WCRPS
3.29
8
3
D52
BAS
08
6/2
80
DSS
EAS
9
O
3:25
35
052 C
3
2
0.00
341
55
052
LAT
339
011
3.22
2
2
3
3
3
3
LST
0.
5/1
MDN1O
MON10D
4
4
5
S
7
3.44
0.14
0.50
A31
8
2
8
9
9
1
(a) Inverting
y
(b) Modeling the distribution of MNIST pixel data from the label.
Figure 3: Multivariate tasks. Image (a) shows 2D densities, while (b) displays random samples.
distribution x + p(y). where we sample (x,, y) points based on the luminance of an image of the Mona Lisa with row, and y column index. As result, the methods required learn many different, but continuously changing distributions, for each row of the image. Note that during training. the models only ever see separate (x, 3) pairs a time, i.c., not (æ; 91. ........ Vn.) samples.
We train multiple models of the same size, including an MDN model with 100 components, along with multiple kinds of Figure 2 after training for 40 and 2000 epochs. The MDN model learns quickly, however ends up with an overly smooth representation. Both single-head CRPS models fail ultimately represent details of the image, with perhaps the weighted version performing slightly better. The multi-head and layered-mulle porred-muti-hhaal model both learn fast and manage to learn the baseline distribution high- detail. However, without layering. visible artifacts remain even after 2000 epochs. Ultimately, the layered-me maltimullaeae variant learns fast and ends up representing the distribution with high- accuracy at the same time. See Appendix C33 for experimental setup and more samples during training.
models, and plot the output distributions as 2D heatmaps on
5.2 MULTIVARIATE REGRESSION
For multivariate regression, run two experiments, one with synthetic and one with real-world data. For exact experimental setups, please refer to Appendix C.4 or the code repository. First, we train the networks to produce solutions to the equation 2² = - + +y² with as input and the joint distribution of (x, y) output. We plot the resulting distributions for models of the same size: simple (CRPS) and weighted (WCRPS) CRPS-based model. and MDN models with 10 (MDN10) and 100 components (MDN100), As we can observe, all models learn to invert the equation, however the CRPS-bases models provide much more robust contour of the resulting circle shrple distribution.
in Figure 3a for four
Having successfully learned the joint distribution of 2 variables, lenging task: learning the joint distribution of 784 variables. We reverse the input-output structure of the MNIST classification task (LeCCnn 1998), turning it into multivariate regression problem, with
turn to much more chal-
8

---

Published as a conference paper at ICLR 2025
the label as input and image pixels as output, similar generative modeling, however without the use of locality structures. Note that we model MNIST solely to stress-test the multivariate formula, not for actual generative modeling purposes. In Figure 3bb] we display samples from the predicted distributions. Since multivariate likelihood is hard to estimate, we instead indicate the likelihood of the sampled latent noise corresponding to the sampled outputs. Remarkably, the model is able to learn the joint distribution of the 784 output variables, producing coherent hand-written digits.
5.3 UCI REGRESSION UNCERTAINTY BENCHMARK AND BASELINES
The UCI Benchmark (Hemáánezz-oobato & Adams 2015) has become the standard benchmark to measure regression uncertainty, with many papers reporting RMSE and (negative) log-likelihodd (NLL) along with standard error. We run our experiments on the exact train-test splits defined
by Hleráández-Lobatt & Adams (2015). using the setup and code from the repository of Gal &
Ghahramani (2016). See Appendix C.5 for discussion of hyperparameters and optimization.
Strongest baselines are Sun et al. (2017) (PBP-MV) for BNN, and Gal & Ghahramani 2016)
(Dropout), covering best overall NLL from the literature (Mukhoti et al., 2018). except for boston
(2.33. EI-Laham et al.] (2023)), naval (-5.64, Lakshminarayanan et al. (017)) and yacht (1.10.
Springenberg et al. (2016)) We run our experiments with the network sizes of Sun et al. (2017) (PBP-MV), using two hidden layers with 50 neurons each. We reimplement the method of Lak- shminarayanan et al. (0077) (Ensembles) and evaluate at the same network size. For MDN, we our own robust implementation, achieving much stronger scores then previously reported,
Appendix C.2 Baselines compute NLL analytically, while CRPS-based methods are evaluated as
described in Section [.7] For more discussion of baselines, please refer to Appendix [C.6 Further
comparison against simpler models is presented in Appendix C.II We compare against Boucha- court et al. (2016) Appendix C.13]and against Depeweg al. (2018) in Appendix C.14
5.3.1 RESULTS
We benchmark multiple variants of our CRPS-based networks, including versions with unweighted, weighted, multi-head, layered-multi serreddmuuii-headd versions, with and without ensembling. The overall best- performing variant turns out to be the single-head weighted version with ensembling (WCRPS.) on Tand 2 where we present negative log-likelihood (NLL) and root mean square error (RMSE) scores along with baseline results. Scores without ensembling are also included (WCRPS). The most direct competition for WCRPS, and WCRPS are MDN and MDN respectively, i.e., non- Gaussian methods with and without techniques for epistemic uncertainty. Best scores are highlighted in bold, however one should also note the error ranges included with scores when reading the results.
Overall, it is clear that relaxing the Gaussian assumption generally helps achieve higher NLL scores, except on concrete with Dropout winning, and naval where a simple BNN dominates (-6.84, see Appendix C.I Strongest scores are mostly shared between MDNima and WCRPS,, with the latter scoring best overall. We present scores for variants of CRPS-based methods in an ablation study in Appendix C77, revealing that the method is also capable of achieving the best NLL result on the concrete dataset, with score of 2.73 (MH variant with ensembling). Multi-head variants under- perform WCRPS, on many (though not all) datasets, indicating that the single-head architecture's strong implicit bias towards smooth distributions appears to be an advantage in most cases.
The wine dataset is unique as it is in fact a classification task with 6 discrete target values (see Appendix(C.Iig MDN outperforms others by 5 orders of magnitude in likelihood by using separate mixture components with increasingly small variances on each value. Note that this approach can theoretically produce NLL values converging to -oo even while assigning equal probabilities to all outputs, thus offering no useful prediction. Hence, on discrete datasets, NLL is an unreliable metric.
In RMSE, WCRPS, dominates, with only small advantage of MDN on yacht. As an explanation for the superior performance, while CRPS includes the mean absolute error (MAE) explicitly, all other methods optimize for NLL, which only indirectly optimizes the expectation of the prediction.
Total running time depends not only on raw numeric computation, but also on factors like batch size, data access patterns, convergence speed, and evaluation speed for early stopping. To provide a sense of relative speed, we report running time on the full UCI benchmark suite, which includes datasets with varying characteristics. With MDN at just bit over 1.5 hours, WCRPS completes in about

---

Published as a conference paper at ICLR 2025
Table 1: NLL results the UCI Datasets benchmark (lower is better)
dataset
Dropout
PBP-MV
Ensembles
MDN
MDN
WCRPS
WCRPS,
boston
0.5k
2.40
1..0s
2..0
2.52 0000
2.36
2.40..0.5
2.32 =0.05
concrete
1.0k
2.94-0:2
3.04
2.97
=.13
2.99
3.09
2.95-0:05
energy
0.8k
1.21 =0.01
1.01::001
0.62
0.77 00.04
0.65
0.47=t004
0.31 -0.0
kin8nm
8.2k
-1.14
-1.28
-1.34
-1.22=0001
-1.30
-1.32
-1.38
naval
11.9k
-4.45.0000
14.85.1000
-6,49
-6.24=0004
-6.72.10.0
-6,65.1005
-6.51 0000
power
9.6k
2.81
2.78 1:000
2.69
2.65
2.62:10001
2.66:1001
2.57 0.01
protein
45.7k
2.87 00.00
2.77 4-o0t
2.66
2.00
1.96
2.22
2.05 0.001
wine
1.6k
0.93 =.001
0.97 4:0.0
0.91 #:0.1
-3.48 00003
-5.11 +:0.9
0.90
0.80 0..0
yacht
0.3k
1.25
1.64
-0.02
0.26
0.63
0.07
-0.16
Table 2: RMSE results on the UCI Datasets benchmark (lower is better)
dataset
I
Dropout
PBP-MV
Ensembles
MDN
MDN
WCRPS
WCRPS,
boston
0.5k
3.61 20:3
3.1 1::15
3.37
3.73
2.93
3.07 20:23
2.91
conerete
1.0k
5.45
5.08
5. 1.19
5.99
5.25
5.51 20:14
4.94
energy
0.8k
0.97 10.06
0,45 1.0.1
0.86
1.17 1:0.9
0.00:
0.45 1.000
:..0:10.01
kin8nm
8.2k
0.09 1:000
0.07
0.07
0,07
0.07:10.00
0.07 0.00:::::0
0.06
nava
11.9k
0.00
0.00
0.00
0.00
0.00=
0.00
0.00
power
9.6k
4. 4.18
3.91 2:004
3.73:10.03
3.93:1004
3.8622004
3.79.00.4
3.62 20.00
protein
45.7k
4,39 20:00
3.94
=.20
3.96:2002
4.09
3.76:0002
3.47 10000
wine
1.6k
0.66 Hoot
0.64
0.63 --000
0.66 H-00r
0.66##0001
0.64
0.63 1:001
yacht
0.3k
1.23 60337
0,81 4..06
0.92
1.97 4:025
0.77 0.077
0.96 0..13
0.78 1:009
Ix, Ensembles 3.4x, WCRPS in 7x, and MDNNmn 17. 17.1x time, the latter being impacted
by both slower per-epoch computation and slow convergence. See Appendix C.8 for details.
5.3.2 CALIBRATION
Calibration is measured described in Kuleshov (ct al. (2018). We take 1000 samples from the pre-
% of sampleze wew.wwwwwwoooo. prediction
near-perfect calibration
underconfident
1.00
1.00
dicted distribution, take every 10th an estimate of percentiles, and measure the ratio of test samples for
0.75
knnäm bestonn
0.75
which the ground truth falls below them. See Fig-
0.50
0.50
u with the ideal calibration as a dashed red line.
0.25
0.25
energy
yech
The method displays good calibration datasets, except yacht and naval, where it is under- confident. However, on both datasets, the under- confidence is also reflected on the train set (see Ap- pendix C.9). implying that is most likely short- coming of training, e.g., too large learning rate.
most
0.00
protein
0.00
naval
0.0
0.5
1.0
0.0
0.5
1.0
predi cted percentile
porcentile
Figure 4: Calibration of WCRPS, on the UCI Datasets benchmark.
6 CONCLUSIONS
In this paper, proposed optimizing nondeterministic neural networks using loss functions derived from the continuous ranked probability score (CRPS). We derived unbiased On log n) formulas for both unweighted and weighted sample-based approximations of the CRPS, and proposed multiple architectures for sampling from the aleatoric distribution of the target, enabling truly distribution- free learning. Through extensive experiments both univariate and multivariate synthetic and real-world datasets, our method consistently demonstrates superior performance compared to base- line models, making it an attractive choice for capturing aleatoric uncertainty. Ultimately, we believe CRPS optimization is an effective and novel way of training neural networks for predicting regres- sion uncertainty in distribution-free manner, while the same time remaining conceptually simple and easy to implement. Finally, we make our implementations and experiment code freely available.
We have exact baseline RMSE scores for noval, as prior work reports only decimals (Sun et al. 2017
10

---

Published as a conference paper at ICLR 2025
REPRODUCIBILITY
We make all code and hyperparameters used run the experiments in the paper publicly available in our source code repository at hhttps://github com proto netooochnttchhhcoooo naut
ACKNOWLEDGMENTS
Supported by the European Union project RRF-2.3.1-21-2022 within the framework of the Artificial Intelligence National Laboratory Program and and by the European Defence Fund project 101103386 FaRADAI.
REFERENCES
Moloud Abdar, Farhad Pourpanah, Sadiq Hussain, Dana Rezazadegan, Li Liu, Mohammad Ghavamzadch, Paul Fieguth, Xiaochun Cao, Abbas Khosravi, Rajendra Acharya, et al. A review of uncertainty quantification in deep learning: Techniques, applications and challenges. Information fusion, 76:243-297, 2021.
Sam Allen. Weighted scoringrules: Emphasising particular outcomes when evaluating probabilistic forecasts. arXiv preprint arXiv:2305.07331, 2023.
Alexander Amini, Wilko Schwarting, Ava Soleimany, and Daniela Rus. Deep evidential regression. Advances in neural information processing systems, 33:14927-14937, 2020.
Lynton Ardizzone, Jakob Kruse, Sebastian Wirkert, Daniel Rahner, Eric W Pellegrini, Ralf Klessen, Lena Maier-Hein, Carsten Rother, and Ullrich Köthe. Analyzing inverse problems with invertible neural networks. arXiv preprint arXiv: 047300 2018.
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint arXiv:1607.064500 2016.
Ludwig Baringhaus and Carsten Franz. On a new multivariate two-sample test. Journal of multi- variate analysis, 88(1):190-200 2004.
Viktor Bengs, Eyke Hüllermeier, and Willem Waegeman. On second-order scoring rules for epis- temic uncertainty quantification. In International Conference on Machine Learning, pp. 2078- 2091. PMLR, 2023.
Jonathan Berrisch and Florian Ziel. Crps learning. Journal of Ecomometrics, 237(2, Part C):105 2023. ISSN 0304-4076. doi: http:///oioooo////.0.oo..cooommp w/www.co www.:ccccccaccccaaaaaaaaaaaaaaap
URL https:
Christopher M Bishop. Mixture density networks. 1994.
Charles Blundell, Julien Cornebise, Koray Kavukcuoglu, and Daan Wierstra. Weight uncertainty in neural network. In International conference machine learning, pp. 1613-1622. PMLR, 2015.
Diane Bouchacourt. Pawan K Mudigonda, and Sebastian Nowozin. Disco nets: Dissimilarity coef- ficients networks. Advances in Neural Information Processing Systems, 29, 2016.
Axel Brando. Mixture density
(mdn) for distribution and uncer- hictts:://gichub..co///as pdf] Report of the Master's Thesis: Mixture Density Networks for distribution and uncertainty estimation.
networks
tainty estimation, 2017.
URL
Wenhu Chen, Yilin Shen, William Yang Wang, and Hongxia Jin. A variational dirichlet frame-
work for out-of-distribution detection. arXiv: Learning, 2018. URL. httpss//appi
Sungjoon Choi, Kyungjae Lee, Sungbin Lim, and Songhwai Oh. Uncertainty learning from
demonstration using mixture density networks with sampling-Fe variance modeling. 2018 IEEE International Conference on Robotics and Automation (ICRA), 6915-6922, 69115-92..6111-622.... IEEE, 2018.

---

Published as a conference paper at ICLR 2025
Adrian Corduneanu and Christopher M Bishop. Variational bayesian model selection for mixture distributions. In Artificial intelligence and Statistics, volume 2001, pp. 27-34. Morgan Kaufmann Waltham, MA, 2001.
Henggang Cui, Vladan Radosavljevic, Fang-Chich Chou, Tsung-Han Lin, Thi Nguyen, Tzu-Kuo Huang, Jeff Schneider, and Nemanja Djuric. Multimodal trajectory predictions for autonomous driving using deep convolutional networks. In 2019 international conference roboties and automation (icra), pp. 2090-2096. IEEE, 2019.
Peng Cui, Wenbo Hu, and Jun Zhu. Calibrated reliable regression using maximum mean discrepancy. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 77164-17177 Curran Associates, Inc., 2020. URL https::/ neurips www.eeewwr.s...aaa...e cw/paper.pppppppcepppoeeeeee...e files/
Topp:722000...cccccffffooccaaee2 pape
Joseph Curro and John Raquet. Deriving confidence from artificial neural networks for naviga- tion. In 2018 IEEE/ION Position, Location and Navigation Symposium (PLANS), pp. 1351-1361, 2018. doi: N0.1099PLANS22208.887333
Claudia Czado, Tilmann Gneiting, and Leonhard Held, Predictive model assessment for count data. Biometrics, 1254-126 2009. ISSN 0006341X, 15410420. URLLtttp://www jstor.
oxg/stable/2254000
Philip Dawid and Paola Sebastiani. Coherent dispersion criteria for optimal experimental design.
Annals of Statistics, pp. 65-81, 1999.
Bhargob Deka, Luong Ha Nguyen, and James-A. Goulet. Analytically tractable heteroscedas- tic uncertainty quantification in bayesian neural networks for regression tasks. Neuro- computing, 572:127183, 2024. ISSN 0925-2312.
doi: https//ddbiiog///110061///.eeee 2023.127183. URL https://www. http:///wwwcciccddii.cccc rect. rreet..comssceenee//rtrilleeeeee 30925233122011300
Stefan Depeweg, Jose-Miguel Hernandez-Lobato, Finale Doshi-Velez, and Steffen Udluft. Decom- position of uncertainty in bayesian deep learning for efficient and risk-sensitive learning. In International conference machine learning, pp. 1184-1193, PMLR, 2018.
Chao Du, Tianbo Li, Tianyu Pang, Shuicheng Yan, and Min Lin. Nonparametric generative model- ing with conditional sliced- wasserseenn flows, 2023.
Gintare Karolina Dziugaite, Daniel M. Roy, and Zoubin Ghahramani. Training generative neural networks via maximum mean discrepancy optimization, 2015.
Yousef El-Laham, Niccold Dalmasso, Elizabeth Fons, and Svitlana Vyetrenko. Deep gaussian mix- ture ensembles. In Uncertainty in Artificial Intelligence, pp. 549-559. PMLR, 2023.
Christopher AT Ferro, David S Richardson, and Andreas P Weigel. On the effect of ensemble size on the discrete and continuous ranked probability scores. Meteorological Applications: A journal of forecasingg, practical applications, training techniques and modelling, 15(1):19-24 2008
Yarin Gal and Zoubin Ghahramani. Dropout as bayesian approximation: Representing model un- certainty in deep learning. In Maria Florina Balcan and Kilian Q. Weinberger (eds.), Proceedings of The 33rd International Conference Machine Learning, volume 48 of Proceedings of Ma- chine Learning Research, pp. 1050-1059, New York, New York, USA, 20-22 Jun 2016. PMLR.
Jakob Gawlikowski, Cedrique Rovile Njieutcheu Tassi, Mohsin Ali, Jongseok Lee, Matthias Humt, Jianxiang Feng, Anna Kruspe, Rudolph Triebel, Peter Jung, Ribana Roscher, al. A survey of uncertainty in deep neural networks. Artificial Intelligence Review, 56(Suppl 11:1513-1589, 2023.
Manuel Gebetsberger, Jakob W. Messner, Georg J. Mayr, and Achim Zeileis. Estimation methods for nonhomogeneous regression models: Minimum continuous ranked probability score versus maximum likelihood. Monthly Weather Review, 146(12):4333 4338, 2018. doi: 10.1175/MWR-D-17-088 17-0344.. URL https::/ /tttps:/jjounnasc anetsoc org/view/ Bourna 1 wttpw////46////// mwr 0364. xm1
12

---

Published as a conference paper at ICLR 2025
Soumya Ghosh, Jiayu Yao, and Finale Velez. Model selection in bayesian neural networks via horseshoe priors. Journal of Machine Learning Research, 20(18)): 46, 2019.
Tilmann Gneiting and Adrian E Raftery. Strictly proper scoring rules, prediction, and estimation. Journal of the American statistical Association, 102(477):359-378 2007.
Tilmann Gneiting and Roopesh Ranjan. Comparing density forecasts using threshold-and quantile- weighted scoring rules. Journal of Business & Economic Statistics, 29(3):411 (153:411-422,, 2011.
Ethan Goan and Clinton Fookes. Bayesian neural networks: An introduction and survey. Case
Studies in Applied Bayesian Data Science: CIRM Jean-Morlet Chair, Fall 2018, pp. 45-87, 2020.
Irving John Good. Rational decisions. Journal of the Royal Statistical Society: Series B (Method- ological), 14(1):107-114, 1952.
James-A Goulet, Luong Ha Nguyen, and Saeid Amiri. Tractable approximate gaussian inference for bayesian neural networks. Journal of Machine Learning Research, 22(251):1-23, 2021.
Alex Graves. Generating sequences with recurrent neural networks. CoRR, URL htttp://axxxi.....///////// org/ abs/aaas..111/// 0850.
2013.
Danijar Hafner, Dustin Tran, Timothy Lillicrap, Alex Irpan, and James Davidson. Noise contrastive priors for functional uncertainty. In Uncertainty in Artificial Intelligence, pp. 905-914, PMLR, 2020.
Dan Hendrycks and Kevin Gimpel. arXiv:1606.08411 08415, 2016.
Gaussian error linear units (gelus). arXiv preprint
José Miguel Hernández-Lobatt and Ryan Adams. Probabilistic backpropagation for scalable learn- ing of bayesian neural networks. In International conference on machine learning, pp. 1861- 1869. PMLR, 2015.
Johannes Hertrich, Christan Wald, Fabian Altekrüger, and Paul Hagemann. Generative sliced MMD flows with Riesz kernels. The Twelfth International Conference on Learning Representations, 2024.
Lars U Hjorth and Ian T Nabney. Regularisation of mixture density networks. In 1999 Ninth Inter- national Conference on Artificial Neural Networks ICANN (C.nC. Publ. No. 470), volume 2, pp. 521-526. IET, 1999.
Christian Hubschneider, Robin Hutmacher, and Marius Zöllner. Calibrating uncertainty models for steering angle estimation. In 2019 IEEE intelligent transportation systems conference (ITSC), pp. 1511-1518. IEEE, 2019.
Eyke Hüllermeier and Willem Waegeman. Aleatoric and epistemic uncertainty in machine learning:
An introduction to concepts and methods. Machine learning, 110(3)4577-006 2021.
Alan Julian Izenman. Review papers: Recent developments in nonparametric density estimation.
Journal of the american statistical association, 86(413):205-2) -2-4, 1991.
Alexander Jordan, Fabian Krüger, and Sebastian Lerch. Evaluating probabilistic forecasts with scoringrules. Journal of Statistical Software, 90(12):1-37, 2019. doi: 10.186377jss.v000..1
URL htR/sttps:/ //wp:///wwwww.. www.tssoottoog/iiioooooss
Alexander Jordan. Facets of forecast evaluation. PhD thesis, Dissertation, Karlsruhe, Karlsruher
Institut für Technologie (KIT), 2016, 2016.
Laurent Valentin Jospin, Hamid Laga, Farid Boussaid, Wray Buntine, and Mohammed Bennamoun. Hands-on bayesian neural networks-a tutorial for deep learning users. IEEE Computational Intelligence Magazine, 17(2):29-44 48, 2022.
H M Dipu Kabir, Abbas Khosravi, Mohammad Anwar Hosen, and Saeid Nahavandi. Neural network-based uncertainty quantification: A survey of methodologies and applications. JEEE Access, PP, 06 2018. doi: 10.1 (099/acces.2018.883699
13

---

Published as a conference paper at ICLR 2025
Alex Kendall and Yarin Gal. What uncertainties do need in bayesian deep learning for computer vision? Proceedings of the 31st International Conference on Neural Information Processing Systems, NIPS' 17, pp. 5580-5590, Red Hook, NY, USA, 2017. Curran Associates Inc. ISBN 9781510860964.
Fabian Krüger, Sebastian Lerch, Thordis Thorarinsdottir, and Tilmann Gneiting. Predictive infer- ence based on markov chain monte carlo output. International Statistical Review, 89(2):274-301 2021.
Volodymyr Kuleshov, Nathan Fenner, and Stefano Ermon. Accurate uncertainties for deep learning using calibrated regression. In International conference machine learning, pp. 2796-2804. PMLR, 2018.
Francesco Laio and Stefania Tamea. Verification tools for probabilistic forecasts of continuous hydrological variables. Hydrology and Earth System Sciences, 1 1(44: 2007.
Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. Advances neural information processing systems, 30, 2017.
Yann LeCun. The mnist database of handwritten digits. http://yann. lecun. com/exdb/mnistt/ 1998.
Nils Lehmann, Jakob Gawlikowski, Adam J Stewart, Vytautas Jancauskas, Stefan Depeweg, Eric Nalisnick, and Nina Maria Gottschling. Lightning uq box: A comprehensive framework for uncertainty quantification in deep learning. arXiv preprint arXiv:2410.03390, 2024.
Yujia Li, Kevin Swersky, and Richard Zemel. Generative moment matching networks, 2015.
Christos Louizos and Max Welling. Structured and efficient variational deep learning with matrix gaussian posteriors. In Maria Florina Balcan and Kilian Q Weinberger (eds.), Proceedings of The 33rd International Conference on Machine Learning, volume 48 of Proceedings of Machine Learning Research, pp. 1708-1716 New York, New York, USA, 20 22 Jun 2016. PMLR.
Martin Magris and Alexandros Iosifidis. Bayesian learning for neural networks: algorithmic survey, Artificial Intelligence Review, 56(10): 2023.
Osama Makansi, Eddy llg. Ozgun Cicek, and Thomas Brox. Overcoming limitations of mixture den- sity networks: A sampling and fitting framework for multimodal future prediction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7144-7153, 2019.
Andrey Malinin. Uncertainty estimation in deep learning with application to spoken language assessment. PhD thesis, 2019.
Safa Messaoud, David Forsyth, and Alexander G Schwing. Structural consistency and controllability for diverse colorization. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 996-612, 2018.
Jishnu Mukhoti. Pontus Stenetorp, and Yarin Gal. On the importance of strong baselines in bayesian deep learning. arXiv preprint arXiv:1871.09358 2018.
Allan H Murphy. The ranked probability score and the probability score: A comparison. Monthly Weather Review, 98(12):917-924, 1970.
Thomas Muschinski, Georg Mayr, Thorsten Simon, Nikolaus Umlauf, and Achim Zeileis Cholesky-basee multivariate gaussian regression. Econometrics and Statistics, 29:261-288 2024. ISSN 2452-3062. doi: htt.://cwc......eee......aaaaaaa
URL https:
www. www.ssceeenceiiire....... www.
Markus Oberweger, Paul Wohlhart, and Vincent Lepetit. Hands deep in deep learning for hand pose estimation. arXiv preprint /502. /502006807 2015.
Romain Pic, Clément Dombry, Philippe Naveau, and Maxime Taillardat. Distributional regression and its evaluation with the crps: Bounds and convergence of the minimax risk. International Jour- nal of Forecasting, 3964):1564-1522 doi
October 2023. ISSN 0169-2070, 2070. doi: 2022. 1 11.001 1tw.11/0/....fooorsseseee.e
2022.1 12..11.000 11.01. |tt.o///...//////////o/.........
14

---

Published as a conference paper at ICLR 2025
C Radhakrishna Rao. Diversity and dissimilarity coefficients: unified approach. Theoretical population biology, 21(1):24 43, 1982.
Carl Edward Rasmussen. Gaussian processes in machine learning. In Summer school on machine learning. pp. 63-71. Springer, 2003.
Stephan Rasp and Sebastian Lerch. Neural networks for postprocessing ensemble weather forecasts. Monthly Weather Review, 146(1 1):3885-3900, 2018.
Murray Rosenblatt. Remarks on Some Nonparametric Estimates of a Density Function. The An-
nals of Mathematical Statistics, 2733) 837, 1956. doi: 10. 00.1214/amm/111772881 URL
https: //do::/doiiio /0t///./....... 1214/ aoms 1177728190)
Christian Rupprecht, Iro Laina, Robert DiPietro, Maximilian Baust, Federico Tombari, Nassir Navab, and Gregory D Hager. Learning in uncertain world: Representing ambiguity through multiple hypotheses. In Proceedings of the IEEE international conference on computer vision, pp. 3591-3600, 2017.
Bernhard Schölkopf. The kernel trick for distances. In T. Leen, T. Dietterich, and V. Tresp (eds.), Advances in Neural Information Processing Systems, volume 13. MIT Press, 2000. URL [ttpe://pproaeeiinnn.eeeess..s.s ftt1://8855...tf.//a//////aael63
fearrrfflesspppaa flles/paper/2000//
Dino Sejdinovic, Arthur Gretton, Bharath Sriperumbudur, and Kenji Fukumizu. Hypothesis testing using pairwise distances and associated kernels (with appendix), 2012.
Marcin Sendera, Jacek Tabor, Aleksandra Nowak, Andrzej Bedychaj. Massimiliano Patacchiola, Tomasz Trzcinski, Przemystaw Spurek, and Maciej Zieba. Non-gaussian gaussian processes for few-shot regression. Advances Neural Information Processing Systems, 14:10285-10298, 2021.
Bernard W Silverman. Density estimation for statistics and data analysis. Routledge, 2018.
SiriTeam. Deep Learning for Siri's Voice: On-device Deep Mixture Density Networks for Hy- brid Unit Selection Synthesis, 2017. URL http::////mmannneeeeeeeee aapple..........nne.eep.ppppeeee com/
research/siri-vol
Jost Tobias Springenherg, Aaron Klein, Stefan Falkner, and Frank Hutter. Bayesian optimization with robust bayesian neural networks. In D. Lee, M. Sugiyama, U. Luxburg, I. Guyon, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 29, Curran Asso- ciates, Inc., 2016.
Shengyang Sun, Changyou Chen, and Lawrence Carin. Learning Structured Weight Uncertainty in Bayesian Neural Networks. In Aarti Singh and Jerry Zhu (eds.), Proceedings of the 20th International Conference on Artificial Intelligence and Statistics, volume 54 Proceedings of Machine Learning Research, pp. 1283-1292. PMLR, 20-22 22 Apr 2017.
Gábor J Székely and Maria Rizzo. new test for multivariate normality. Journal of Multivariate Analysis, 93(1):58-80, 2005.
Robert Taggart. Estimation CRPS for precipitation forecasts using weighted sums of quantile scores and Brier scores. Bureau of Meteorology, 2023.
Jean Thorey, Vivien Mallet, and Paul Baudin. Online learning with the continuous ranked probability score for ensemble forecasting. Quarterly Journal of the Royal Meteorological Society, 143(702): 521-529, 2017.
the
Jonathan Tompson, Murphy Stein, Yann Lecun, and Ken Perlin. Real-time continuous pose recovery of human hands using convolutional networks. ACM Transactions on Graphics (ToG), 33(5):1- 10, 2014.
Hristos Tyralis and Georgia Papacharalampous. A review of predictive uncertainty estimation with machine learning. Artificial Intelligence Review, 57(4):94, 2024.
15

---

Published as a conference paper at ICLR 2025
Amirhossein Vahidi, Simon Schoßer, Lisa Wimmer, Yawei Li, Bernd Bischl, Eyke Hüllermeier, and Mina Rezaci. Probabilistic self-suue learning via scoring rules minimization. arXiv preprint 2.0.0:2300.02048, 2023.
Dennis van der Meer, Pierre Pinson, Simon Camal, and Georges Kariniotakis. Crps-based online learning for nonlinear probabilistic forecast combination. International Journal of Forecasting, 2024.
Bin Wang and Xiaofeng Wang. Bandwidth selection for weighted kernel density estimation. arXiv preprint arXiv:0709.1616, 2007.
Erik Zawadzki and Sebastien Lahaie. Nonparametric scoring rules. Proceedings of the AAAI Conference on Artificial Intelligence, 29(1), Mar. 2015. doi: 10. 1609/aaai http:aa.iiii9911.9.99 URL
https://ojs aaai. org/ itwpx.ndexxeee AAAI http:////p////ttttt.////////ee/ 9996
16

---

Published as a conference paper at ICLR 2025
A BACKGROUND
A.1 LIST OF PRIOR WORK EVALUATING ON THE UCI REGRESSION UNCERTAINTY BENCHMARK
The UCI Benchmark was originally defined by Hernandez-LLbato & Adams (015)). We are aware
of results being reported for the benchmark in Louizos Welling (2016); Springenberg et (206);;
Gal & Ghahramani (2016) Sun al. (2017) akshminarayanna et al. 201 Mukhoti et al. (2018):
Ghosh et al. (2019) Amini et al. (2020); Goulet et al. (2021)
El-La et (2023); Gaw iillikowsii
al. (023))
Deka al. 2024).
A.2
SMALL SURVEY OF UNCERTAINTY SURVEYS
To provide evidence for claim that the distribution-free modeling of aleatoric uncertainty for neural network regression relatively under-researched and under-reported area, we conduct a small survey, where we list uncertainty-adjacent survey papers and examine them based on four criteria:
MDN Whether the survey mentions the use of mixture models for aleatoric uncertainty in regres- sion, cites MDN (Bishop 1994), or other works using MDN
NF Whether the survey mentions the use of Normalizing Flows for alcatoric uncertainty
MMD
Whether the survey mentions the Maximum Mean Discrepancy loss in the context of learn- ing aleatoric uncertainty
ES Whether the survey mentions the Energy score / CRPS in the context of alcatoric uncer- tainty or cites DISCO Nets (Bouchacourt et al., 2016)
We summarize findings in Table To elaborate further on each:
Goan & Fookes (2020) is survey of Bayesian Neural Networks, and such primarily concerned with epistemic uncertainty. However, it does cite SiriTeam (2017) which uses MDN.
Abdar al. 2021 is general survey of uncertainty techniques in deep learning. In the main text, it describes MDN under the name Gaussian Mixture Model and cites cites|Choi et al.) (2088) which uses MDN.
Kabir et al. (2018) is survey of uncertainty quantification in deep learning. Does not men- tion or cite any of the above.
Hüllermeier & Waegeman (2021) is general introduction epistemic and aleatoric un- certainty in machine learning. Does not mention or cite any of the above.
Jospin et (2022) is tutorial on Bayesian Neural Networks, and (somewhat appropriately in this case) does not mention cite any of the above
Magris & lositidis (2023) is a survey on Bayesian learning for neural networks in general. Does not mention or any the above.
Gawlikowski et al.
2023 is survey of uncertainty in deep neaural networks. Cites Cor-
duneanu & Bishop 2001), therefore indirectly references MDN.
Tyralis & Papacharalampous (2024) is a review of predictive uncertainty in machine learn- ing. It does describe the CRPS, but mentions uses for learning only in context of ensembles or with a Gaussian assumption. also mentions the Energy Score, but does not describe
any uses. Does not cite Bouchacourt et al. (2066).
Many of these works describe using non-Gaussian distributions, however usually in the context of variational distributions, (i.e., the distribution weights a BNN). Some mention the use of MMD for quantile regression calibration, but not for distributional regression.
17

---

Published as a conference paper at ICLR 2025
Table 3: small survey of uncertainty surveys.
Paper
MDN NF MMD ES
Goan & Fookes
(0200)
Abdar et al.
2021
Kabır et al.
2018)
Hüllermeier & Wacgeman 202
Jospin et al. (2022)
Magris & Iosifidis 2023
Gawlikowsk ct al. (2023
Tyralis & Papacharalampon 2024)
B PROOFS AND DERIVATIONS
B. WEIGHTED CRPS FORMULA
The classic formula estimates the first term of Equation and calculating
by taking a samples (0)) from y ~
/ by 9) dQ(
E y
81
y ÿtl
(10)
522
If Q is absolutely continuous w.r.t. Q' (or Q < Q'), then given Y ~ Q' and its sample (v/),
[ J oldQ()) 1/1 5l 1 dQ' (j) Filyy (Y()
n Σ
s//o0(1)), (11)
where w(y/) - (2)) Doing the same for the second term:
E 81 - Y/
/ / 01 - 02 dQ(a))
(12)
/
10z
D2
12
aQ
(01) dQ' (11)do de (02) dQ' (02)
(13)
EllY (2)
v w())))
(14)
n
with Yj, ~ Q' iid. Finally,
1
CRPS(Q, v) R CRPS (FEDEP v)
04
(15)
2n
i=1
where wy further denotes w(ğt), and FECDF denotes the empirical cumulative distribution function of the weighted sample (ūt, wa). An unbiased version can be computed as in Section 4.1 by replacing the coefficient by 2m(n-1y'
B.2 FAST BIASED VERSION
First we are going to prove a formula for the biased version for the sake of simplicity, and de- scribe how to get unbiased version in Appendix B.3 The following formula can be calculated in (n log n) time by first ordering the sample to get Ict).... (()) and corresponding weights W(1): (())
Proposition B.1. Let W = Ei W(i and s, = Σj=1 WG)- Then
CRPS (FIC y)
Σ n² 1=1
( (Ur)) - y) (n11f < ((0) - -;
W w(() )
(16)
2
18

---

Published as a conference paper at ICLR 2025
Proof. Using counting method. First observe that
160
w())
M00
(17)
2n
i=
n 1i < 5} w( (()
denl Σ m 2()
(18)
n
j=1
Next, we are going to count the (0()) yl distances on Figure 5 where we visually display the summands and their weights from Equation 18) (omitting the preceding coefficient)
1122)
((4)
$(5)
(0gg
(-R)xx
Wios
XXM
"D2"667 "p""""""
1x
Figure 5: Counting method example with n 7. We counting the sum of coefficients for each (i( - y| term. On the figure, solid lines are to be counted with a coefficient of (-n) and dotted lines with coefficient of 1. Further, individual change-of-measurg coefficients each line in the form (() w(s) W())- The global coefficient - is omitted for simplicity.
indicated
Starting from the left with the example of = 1, for (y - B(1)), we have the coefficients
(1) w(1)
w
(19)
However, this way we over-count each (y - -G)) term for (j > i, (0) < <)) y) by a factor of (() (G))
which we are going to correct for when counting the coefficients for (y -()))
Accordingly, for the i-th term (y (()) in general, assuming 01e) < y. we have the coefficients
H
(i + (()
Σ
w(i)
w(s
Σ
(20)
j=it1
j=1
w())
W()) +
W. - Σ w
(21)
/=1
1=1
=(()) (W
2s; + w())
(22)
19

---

Published as a conference paper at ICLR 2025
We can proceed to count the coefficients of (b(z) y) with y ((1) by mirroring the above logic.
Denoting k; = W Σj=1 W(() = W - 8; + ((). we have the coefficients (cf. Equation (22)
wi (W n 2k; + w(a) w(()) (W 2(W 9; + m())) w(())
(23)
(1) (W 2s; m0))))
(24)
Finally, we have
CRPS
(25)
1 n
Σ Jイ ạ
(1) (W-n- -n nnn-nn--- 28; ww() (y (()) Σ Ewr) (W --nn-s;;uue (0(0) y)
(26)
W m + ((())
w v) nlly )(1)
(27)
n2a
2
B.3 FAST UNBIASED VERSION
Proposition B.2. An unbiased version of Equation (16 can also be calculated in (n log
Proof.
w (i(1) y) [nlly < D(0) W + W(()
CRPS@(PEE y)
(28)
2 1=1
2
Σ
(0 - v) (n 2s; + W +w())) + Σ w (v - ((s) (n+ 109:-WWaa W ((oo
(29)
n12
Notice that
wolv 2(a)
Σ サー
w(s) (Dt) - u) n + Σ w(a) (y - ((s))
(30)
n
and that Equation 30 appears in Equation (29). Therefore, we can modify Equation 29 to express
the unbiased coefficient for the second term of Equation 7 without affecting the first term by taking
Σ y<ūr)
w(y (Dio - v) ((n 1) 2s, + W + (()) +
(31)
n(n - 1)
+ Σ
m6)) (y - (()) ((n - 1) + 28; - W -w(3)
(32)
Dis Ky
2
W
w(() +
Σ W(() (ōte) - y) 12 8i
+
(33)
n(n 1)
2
v<3(())
W + ((() 1
+
Σ
w() (ū(e) v) (-s;+
(34)
2
Transforming back to short form and denoting the unbiased version with CRPS' w. we get
n
(e (2() v) (())))) +ss
W W +1
CRPS', (FF v) n nnn-1) 1)
Σ
(35)
20

---

Published as a conference paper at ICLR 2025
B.4 FAST UNBIASED UNWEIGHTED VERSION
Proposition B.3.
2 1 (T()) - y) ( < (0) ( -in)) CRPS((FECF. 1( 1)
(36)
Proof. Taking w()) = 1, we get W and 8; Substituting into Equation (35 the proposition immediately follows.
C IMPLEMENTATION DETAILS AND RESULTS
C.I KERNEL DENSITY ESTIMATION
As described, we use the adaptive-bandwicth kernel density estimator of Wang & Wang (2007) for likelihoodessimmation in our experiments. We implement a GPU-based version of the algorithm, based on the C++ based cpu-implementation of the aw Python library.
The method works by first creating a naive pilot likelihood estimate in each sampled point based on the full sample using traditional constant bandwidth KDE, and then setting the local bandwith to be inversely proportional to the pilot-estimate. Calculating the pilot estimates requires calculating the pairwise distance matrix for all sampled points, making the algorithm quadratic the number of points. Since calculating pairwise distances a GPU is fast, the quadratic runtime proved to be feasible in our experiments. We run the algorithm with the default parameters for the original implementation of awKDE, i.e., using Silverman's 's estimate for the global bandwidth parameter (Sil- verman 2018).
As /Wang& & Wang (2007) note, the quality of the pilot estimate is ultimately not critical for the quality of the final estimate. Accordingly, we have seen some success with using a fixed number of samples to produce the pilot-estimate, making the runtime linear instead of quadratic. However, proper investigation of such methods is out of scope for this work.
C.2 MDN IMPLEMENTATION
MDN models seem to relatively under-utilized in uncertainty-rellted literature, with some surveys barely mentioning the method (Abdar et al.) [202] Goan & Fookes 2020), and others omitting any mention at all (Gawlikowski et al. [023 Jospin et al. 2022 Magris & Iosifidis 2023 Kabir et al., 2018 Hüllemeier & Waegeman] |021 Tyralis& & Papacharalampous 2024). Further, it appears that MDN has very rarely been evaluated as baseline in prior work. We are aware a single instance where it is used baseline method (El-Laham al. 2023) for the UCI benchmark, however with the authors reporting substantially lower scores than our experiments.
MDN is sometimes also referred to simply Gaussian mixture model (GMM) (Hubschneider al. 2019 Abdar et al.] 202T). however in general these two terms do not mean the same thing, as GMM includes a larger class of methods with various modeling and optimization strategies. is also important to note that not every Gaussian mixture model is made equal: in order to approximate even simple asymmetric densities, high number of Gaussians might be required. Using methods relying on, e.g., ensembling (Jospin et al.] (202),, prohibits the use of a high number of mixture components due computational constraints.
The reason for its neglect is perhaps the fact that MDN appears to be notoriously challenging to im- plement, often suffering from numerical instability, failure to converge, and mode collapse (Makansil
et al. ctt al. 2018 Graves 2013 Hjorth & |abney] [199).. Nevertheless, we do implement and evaluate robust variant of MDN, capable of optimizing the mixture of 100 Gaussian components in our experiments.
2019 Brando| 2017: Rupprecht et al., (2017 et al. 2019 Curro & Raquet 2018 Messaoud
httpss://githuub.coo//meeee
21

---

Published as a conference paper at ICLR 2025
Our implementation takes number of steps avoid numerical instabilities and convergence issues that tend to plague implementations of MDN, this way achieving much stronger scores on the UCI benchmark then previously reported. We apply the following techniques:
We standardize both the input variables X and the target Y to have mean and a variance of 1.
We apply a linear learning rate warm-up schedule, where the learning-rate is set 0.1 of its value at the beginning and increases over 10 epochs.
We apply gradient clipping at 10 before taking step along the loss gradient.
We caleulate the log- ligeliieo in the log rog-dommin as
2
4 = y 14
In (I.VTT
(37)
2 04
W% = π.
ISF TTn)
(38)
logllikelhhodd LSE (4 + Wi...., w..)
(39)
where LSE denotes the numerically stable implementation of LogSumExp of the PyTorch framework
i.e.
LS In.) = log
(40)
1=1
During training, clamp the log- likelihhkhod loss to over - 20, where the likelihood at is effectively zero can result the experiment producing oo and values.
single evaluation
We clamp the inputs of the softplus to over -15 and the input logits of the softmax be- tween - 15 and 15, as both tend to converge to very low and hight values:
the method turns off certain components, their logits π, tend towards -oo;
the method needs represent an unimodal distribution, its logit x; tends toward co; on discrete datasets and in the case of unused components, the logit n; as well as the input of the softplus 8; tends toward -o.
See Appendix C.4 for multivariate details.
C.3 LEARNING THE MONA LISA
For training, we sample 5000 samples for each row from the distribution defined by the luminance of an 500 x 500 image of the Mona Lisa, such that the input of the network is the row index, and the expected output is the column index. We train - networks with layers of size [128, 512, 512, 1024]. GELU activation, SGD optimizer with Ir 10- and momentum value of 0.8 for 2000 epochs. See Figure 6 for distributions at 40, 100, 200, 500, 1000 and 2000 epochs. The WCRPS-bass method uses 50 heads for both multi-head variants, and 5 layers for the layered- multi-head variant.
C.4 MULTIVARIATE EXPERIMENTAL. SETUP
For the paraboloid task 2² = x2 + 22 we generate training data by drawing 2M samples from Y.Y ~ (((2222), filter them x y2 <= 4, and calculate the corresponding =² values. We train + R2 networks with layers of size [64, 128, 128), GELU activation and LayerNorm, and use an SGD optimizer with Ik epochs and 10-4,
We implement a multivariate version of MDN with full covariance matrix modeling based on the Cholesky-decomposine
based parameterization of the multivariate normal distribution Musschnn- ski et al. 2024), while also employing techniques from Appendix C.2 where applicable.
For the MNIST task, we train network with layers of size [16, 1024, 2048, 4096) with ReLU acti-
vation on the train set of MNIST for Ik epochs with Ir = 10 - and a momentum value of 0.9
22

---

Published as a conference paper at ICLR 2025
(a) Original
(b) MDN
(c) Simple
(d) Weighted
(e)
(f) LMH
Figure 6: Learning the Mona Lisa output distribution after 40, 100, 200, 500, 1000 and 2000 epochs from top bottom respectively.
23

---

Published as a conference paper at ICLR 2025
C.5 HYPERPARAMETERS ON THE UCI BENCHMARK
We do not optimize hyperparameters individually for each dataset, however do adaptive batch sizes, and larger L2 regularization for the 4 smallest datasets. Early stopping is used with 20% of the training set validation.
In our experiments we use the Adam optimizer with a learning-rate of 0.001 and a linear learning rate warm-up schedule starting with the coefficient 0.1 and ending with 1 after 10 epochs. We use early stopping with patience value of 50 on validation sets consisting of 20% of the training set 10-6 on all datasets except the four
for each split. We set L2 regularization over the weights smallest ones (boston, concrete, energy, yacht), where we use 10-a Batch size is chosen to be max( ym 5, 1) with m being the training set size, for a balance of computational efficiency and predictive accuracy.
We run Ensembles and ensembling-based variants of WCRPS using 5 networks, in the original paper of (Lakshminarayanan et al. (2017). We use 20 heads for all multi- variants of WCRPS on the UCI benchmark, and 10 layers for all mayered-multi-hee formula variants, we take 100 samples during training and 1000 samples for evaluation during early stopping. See Appendix C.12 for a comparison of different choices for the samples drawn during training.
variants. For evaluating the CRPS
C.6 BASELINES
Most prior work run experiments by using 100 instead of 50 neurons for each hidden layer for the largest (protein) dataset. For simplicity, we use 50 neurons only, as the models seem to perform well regardless.
For our reimplementation of Ensembles, we also employ techniques from Appendix C.2 where ap- plicable. Our results are generally in line with or slightly better than those reported in the original paper at the same network sizes, except for the naval dataset, where our implementation performs significantly better (-5.64 originally, -6.52 in testing for the single-layer model). Our improve- ment may be due to one or more enhancements discussed in Appendix C.2 possibly influenced by the unusual data range, as also indicated by the RMSE values in Table 2
Note, that the standard network size in most prior work is a single layer of 50 neurons. However, the best prior scores in (Sun al. (2017) (PBP-MV) use a network size of two hidden layers with 50 neurons each, which leads us to re-evaluate Ensembles at the same size. Further, we also evaluate MDN and CRPS-based models the same network size for fairness.
For the CRPS-based models we use GELU (Hendrycks & Gimpel 2016) activation together with layer normalization (Ba al. 2016), as it appears to help the network. For correctness and trans- parency, we also re-evaluate Ensembles, MDN, and MDN with the same setup, and include stronger versions in the main text of the paper in Table 4 See Tables and for pairwise com- parisons in NLL and RMSF. According to the results, we include the Rei.U versions for Ensembles (5/9 for NLL and 4/7 for RMSE), the ReLU versions for MDN (4/8 for NLL with overall strongest results and 6/8 for RMSE), and the GELU version for MDN (6/9 for NLL and 5/8 for RMSE)..
The source-code for the method Dropout is publicly available, however proper evaluation requires grid-search optimization of two separate hyperparameters over every single train-test split of datasets separately, which proved to be prohibitively expensive for the two-layer variant. There- fore, we report the original scores for the single-layer variant in case of Dropout.
Due to methodological differences, cannot compare against Amini et al. (2020). Amini et al. (020)) propose modeling aleatoric uncertainty using the Normal distribution, and epistemic un- certainty as a parametric meta-distribution over the parameters of the aleatoric normal distribution (Normal-Inverse Gamma). They report good performance on the UCI benchmark. However, we could not include their results: reproducing their experiments and investigating the code provided") reveals that
https: thub. cot.ua.iccm/mmmi.... ps2002
24

---

Published as a conference paper at ICLR 2025
Table Pairwise comparison of ReLU and GELU based baseline variants, evaluated in NLL. Pair- wise best scores are highlighted with underline. Scores that perform at least par when compared the results in Table are highlighted in bold.
dataset
Ensembles
Ensembleg
MDN
MDN
MDN
MDN ban
WCRPS,
boston
2.44:0000
2.41 00.05
2.45 10006
2.52
2.36
2.32 1000
2.32 10.0
concrete
2.97 1:00
3.14
3.16:1000 10000
3.13 1.004
2.99 10003
2.99 1:000
2,95 1:005
energy
0,62 10.08
0.75 10.15
1,09 10.06
0.77
0.65 1.002
0.58 1:000
0.31 1:000
kin8nm
t0000
-1.37=0.0
-1.15:000
-1.22:t0:0
-1.30
-1.37
-1.38
naval
66.9 +0.02
-2 0..02
-6.55-0000
-6.24
-6.72 00001
-6.38
-6.51 ::0.00
power
2.69 :0001
2.7 2.002
2.67 1:0000
2.65
2.62 00001
2.64:10.01
2.57
protein
2.66
2.64 20.05
2.02
2.00
1.96:00.01
1.97.10001
2.05
wine
0.91 +0.01
0.96-6.00
-3.75 H0.06
-3.48 +o00
-5.11 +-0.99
-3.91440.4
0,80
yacht
-0.02
-0.32 00.05
0.79
0.26+0009
0.63
0.55+.0.0
-0.16
Table 5: Pairwise comparison of ReLU and GELU based baseline variants, evaluated in RMSE. Pairwise best scores are highlighted with underline.
dataset
Ensembles
Ensembles
MDN
MDN
MDN
MDN ban
WCRPS,
boston
3.37
3.27 -0.24
3.73
2.93 20.20
3.10 3.10
2.91 1a
concrete
19
10.16
6.34
5.99:1011
5.25 :0.11
5.36
4.94
energy
0.86
0.52
1.89 10.13
1.17
0.51 ::000
0.61
0.41 1:00
kin8nm
0.07++000
0.07 -0.00
0.08 #0.00
0.07 40.00
0.07 -:000
0.06++0000
0,06
naval
4.00
0.00+6.00
0.00
0.00++000
0.00 +0:00
0.00+.000
0.00++000
power
3.73-t003
3.83320.00
3.96
3.93
3.86
3.95
3.62
protein
4,20
4.09
4.07
3.96
4.09-
4.11
3.47.002
wine
0.63
0.64
0.65 #0.0
0.66
0.66=2001
0.64-to.o
0.63
yacht
0,92 1:0.08
1.24
1.68.10 1.018
1,97.1022
0.77 0.007
0.83..0.0
0.78
25

---

Published as a conference paper at ICLR 2025
Table 6: NLL results on the UCI Datasets benchmark (lower is better). Scores that perform least on when compared to the results in Table ! are highlighted in bold.
dataset
CRPS
WCRPS
WCRPS
WCRPS
WCRPS.
WCRPS.
WCRPS
boston
2.38
2.40
2.40
2.40
2.32
2.30.000
concrete
3.11 20.0s
3.09.1001
2.97.10.00
2.89 10003
2.95 10.05
2.73:100:1
2.74
energy
0.46=10.4
0.47
0.64
0.79
0.31 20.03
0.67:t001
0.78
kin8nm
-1.32
-1.30
-1.29
-1.38 20001
-1.36+t0.0
-1.33
naval
-6.01
-6.65 4:0.s
-5.93 0:004
-5.83
-6.51
-5,95
-5.80
power
2.66
2.66::0001
2.62:20.01
2.62:10001
2.57::0.0
2.56:10000
2.60
protein
2.29
2.22::0:1
2.22
2:001
2.05
2.18
2.29 1:000
wine
0.89
0.90
-0.36 6.003
0,15
0.80
-0.19
0.11 4:002
yacht
0.07
0.07 1:0.66
0.62
0,93
-0.16 1.005
0.80
0.92
Table RMSE results on the UCI Datasets benchmark (lower better). Scores that perform least on when compared to the results in Table 2 are highlighted in bold.
dataset
CRPS
WCRPS
WCRPS
WCRPS kmb
WCRPS.
WCRPS
WCRPS
boston
3.07 +-023
3.08 3.08
3.12
2.91 #0.18
2.98
2.93+00.0
concrete
5.48 20.16
5.51 +0.14
5.11 #0.15
5,23
4.94 20.17
4.87 4:.15
4.85 #0.13
energy
0.44
0.45 1.:02
0.52
0.48
0.41 :0.01
0.45
0.43
kin8nm
0.07=n.00
0.07 ::000
0.07 -0000
0.07 :0000
20.00
0,06:
0.07 1:000
naval
0.00
0.00
0.00
0.00.10.00
0.00
0.00.10000
0.00
power
3.79
3.79
3.74
3.71::0.04
20.04
3,65
3.64
protein
3.77
3.76
3.86 2..02
3.80+0003
3.47 #0002
3.69 #oo0
3.60
wine
0.64
0.64
0.64 +0.01
0,65+oo.0
0.63 -0.01
0.63 60.01
0.64
yacht
0.95 10.11
0.96
0.89 2.009
1.05 #
0.78 0.0.9
0,99 +0.10
0.94 H0.11
1. They do not use the same train-test splits as other approaches mentioned Section 5.3 rather random ones;
2. Their code appears to validate on the test set, and reports the minimumn of the achieved NLL results over all epochs.
The setup seems consistent across all measured methods in the paper, therefore their results are inter- nally consistent (although an argument could be made that it favors methods with good predictions and large variance across training epochs). However, unfortunately their results are not comparable with the rest of the literature as presented.
Implementing proper train-validation-test setup for their model appears result in subpar per- formance in our experiments, however this could be the case for a number of reasons, such as suboptimal hyperparameter choices our part.
C.7 CRPS MODEL VARIANTS THE UCI BENCHMARK
We present NLL scores in Table 0 and RMSE scores in Table 7 for various CRPS-based model variants. A subseript of indicates ensemble models, superscript of mh and Inh indicates multi- head and layered-multi-hees variants, respectively.
C.8 RUNNING TIMES & SCALING
See Table & for running times in seconds on the UCI Benchmark. All our implementations were AMD EPYC 7F72 workstation with 2x24 CPU cores (48 cores, 96 threads in total) and 384 MiB L3 cache. We used a single NVIDIA 100-SXXM440GGB GPU per experiment, with CUDA version 12.2 and driver version 535. 183.01, ensuring consistent conditions across experiments.
To illustrate scaling, we compare the running time training 10 epochs with MDN and CRPS on the largest profein dataset for increasingly large model sizes, however leaving the expressive power
26

---

Published as a conference paper at ICLR 2025
Table 8: Running times on the full UCI Benchmark in seconds.
MDN
WCRPS
Ensembles
WCRPS
MDN bon
5594
12074
19070
39624
95902
Training 10 epochs (log-log)
Training 10 epochs
x
*
40
*
3x
crps
crps
101
runtime (5)
30
nuntmee
20
6 10
10-
10a
10"
10'
10"
0.0
0.2
0.4
0.6
1.0
1.2
number of parameters (log)
parameters
1e8
Figure Scaling CRPS badsed models compared to a deterministic model.
w.r.t. the output distribution constant (100 components for MDN and two layers of size [100, 50] for the CRPS based model). See Figure 7 for results.
At smaller model sizes, the specifics of the GPU hardware make comparisons somewhat random, as devices can execute a large amount of computation one step. However, as expected, at larger sizes, the nondeterministic part of the network incurs a constant amount of additional cost, leaving the running time approximately linear in the number of model parameters.
C.9 CALIBRATION
Please refer Figure g for calibration plots on all train and test datasets.
C.10 DATASET STATISTICS
Please refer to Table D for dataset statistics.
C.I1 PERFORMANCE AGAINST SIMPLE METHODS
While the primary concern in term of performance is comparison against advanced alternatives, such the baselines of Section 5.3 there is value in observing the improvement when compar- ing against more traditional model types. In this section we present measurements comparing our
Table 9:
Value frequencies in the UCI benchmark datasets
dataset
samples (m)
unique y
unique / m
most frequent y
most frequent / m
boston
506
229
45.26%
16
3.16%
concrete
1030
845
82.04%
6
0.58%
energy
768
586
76.30%
6
0.78%
kin8nm
8192
8191
99.99%
2
0.02%
naval
11934
51
0.43%
234
1.96%
power
9568
4836
50.54%
9
0.09%
protein
45730
15903
34.78%
272
0.59%
wine
1599
6
0.38%
681
42.59%
yacht
308
258
83.77%
3
0.97%
27

---

Published as a conference paper at ICLR 2025
seseeeees bobw predictio
1.00
1.00
1.00
0.50
0.50
0.59
0.29
0.25
0.25
0.25
0.00
0.00
0.50
0.25
0.50
3.75
1.00
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
predicted percentiee
prediitedd percentile
predicted percentile
prellctedpeeremeee
(a) boston
(b) concrete
train
test
Sear cameettte bolw prediition
train
test
1.00
1.00
0.75
0.75
0.75
0.75
0.50
0.50
0.50
0.50
0.00
0.00
0.00
0.00
predicted percentile
predicted percentile
predicted percentile
predicted percentite
energy
kin8nm
train
test
of samp tc bolow prodiction
train
best
1.00
1.00
1,000
1.00
0.75
0.75
0.25
0.25
0.00
0.00
0.00
0.00
0.50
0.25
0.50
3.25
1.00
0.00
0.25
0.50
0.75
1.00
0.30
0.25
0:50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
predicted percantile
predicted percentile
predicted percentile
percentile
(e) naval
( power
train
curcetes Eobo prediction
ooseeeees boow
0.75
0.75
0.50
0.50
0.50
0.50
0.25
0.25
0.25
3.00
0.00
0.25
0.30
0.75
1.00
0.25
0.75
1.00
0.00
0.25
prodicted percentile
presiitte percentile
prodicted percentiee
percentede
g
protein
(h) wine
wewow preddoo
train
test
1.00
1.00
0.75
0.75
0.25
0.25
0.00
0.00
0.30
0.50
1:00
0.00
0.25
0.50
0.75
percenteee
(i) yacht
Figure 8: Calibration for the WCRPS, model on all datasets separately for train and test sets
28

---

Published as a conference paper at ICLR 2025
Table 10: NLL results on the UCI Datasets benchmark (lower is better). Scores that perform least on when compared to the results in Table I are highlighted in bold.
dataset
SHR
MDN
BNN
MDN
WCRPS,
boston
2.74
2,45
2.36
2.36-0.03
2.32
concrete
3.30
3.16
2.98
2.99-0.03
2.95
energy
1.53
1.09:006
1.00
0.65
0.65
0.31
kin8nm
-1.20
-1.15
-1.28:1001
-1.30
-1.38
naval
-6.59
-6.55
-6.84
-6.72
-6.51 +:00s
power
2.78 +0002
2.67 4:.00
2.77 2.00-
2.62-0.00
2.57 H:001
protein
2.89
2.02
2.73::0002
1.96_0.01
2.05
wine
0.98
-3.75
0.89
-5.11 -0009
0.80
yacht
0.67 1.008
0.79 1:009
0.26:10007
0.63 0.00
-0.16
Table 11: RMSE results on the UCI Datasets benchmark (lower is better). Scores that perform at least par when compared the results in Table 2 are highlighted in bold.
dataset
SHR
MDN
BNN
MDN
WCRPS.
boston
3.63
3.54
3.01 2:24
2.93
2.91
concrete
6.20
6.34
5,30
5.25 2.11
1.94
energy
1.51 1:020
1.89.1.13
0,54
0.51
0.41 1:001
kin8nm
0.08 $0.00
0.08 +:000
0,07 +..00
0.07 0..0
0.06 8000000.00
naval
0.00
0.00
0,00++000
2..0000.00
0.00
power
3,9 -.944-0.0
3,96 -9.0
3.96+
3.86=0.04
3.62 10.04
protein
2..0
4.39
4.09=0.02
3.47 10002
wine
0.64
0.65
0.63
0.66
0.63 20.01
yacht
1.39 20.17
1.68 10.18
0,88 8.008
0.77 0.007
0.78 0.009
method against simple probabilistic neural-network based heteroscedastic regression model, and classic BNN trained with variational inference.
simple heteroscedastie regression model predicts the mean along with the variance of the model, and computes the loss function accordingly (SHR for Simple Heteroscedastic Regression). Note that assuming normal distribution for the noise, this model coincides with MDN when setting the number of components to 1.
Classic Bayesian neural networks also predict the parameters of the alcatoric distribution, however also assume probability distributions over the weights of the network. The resulting network can be optimized using variational inference Blundell al. (0155) (BNN). Again assuming normal distri- bution for the noise, this model coincides with BNN based version of MDN with components.
We present results for SHR, MDN, BNN, MDN and WCRPS, in Tables TO and IT As we can observe, in most cases simple methods either benefit from relaxing the Gaussian assumption, or perform similarly to the corresponding variant of MDN. some datasets, however, the Gaussian restriction does seem to benefit the method, most apparent on the yacht and naval datasets.
C.12 PERFORMANCE COMPARISON FOR THE NUMBER SAMPLES
As the CRPS-based methods represent their distribution through producing samples, the number of samples drawn for calculating the loss is an important hyperparameter during training. In Table [2] present performance well as running times for various choices for the number of samples drawn during training.
A smaller number of samples drawn results in faster per- per-batch calculation time, but in turn the training process takes longer to converge. While the running time overall seems to vary randomly to relatively large degree, still observe that smaller values result in faster running times.
29

---

Published as a conference paper at ICLR 2025
NLL results on the UCI Datasets benchmark (lower is better) for different values of samples drawn during training.
Table 12:
dataset
2
3
5
10
20
50
100
200
boston
2.33
2.30
2.40
2.39
2.35
2.33
2.40=0.05
2.40
concrete
3.02=0.03
3.04
3.06:000
3.04
3.08 2..0s
3.09
3.09=0.04
3.11 1.00s
energy
0.59
0.49 10000
0,49 1:003
0.47
0,50
0,44
0.47=0004
0.46::005
kin8nm
-..24_0.02
-1.34.0001
-1.04:1:001
-1.34:20.01
-1.04
-1.03
-1.32=0001
-1.33
naval
-6.07
-6.23 t::00
-6.39
+o.03
-6.41
-6.52+t0.44
-6.57
-6.65
-6.58
power
2.81 -0.0
2.80+aoo
2.78 +0.00
2.73
2.66-H0001
2.66++0.01
2.66=0.00
2.65 4:000
protein
2.66
2.60
2:00
2.27.20.04
2.21
2.22
2.22
2.23 20001
wine
0.93=n.02
0.91 20:0
0.92
0.91
0.92
0.92
0,90=n.03
0.87 2:000
yacht
0.23 0.05
0.13 1:0.0
0.05 1:006
-0.05
0.00
-0.08
0.07 0.066
0.08
time (h)
2.34
2.51
2.41
2.90
3.61
3.55
3.35
3.43
On the other hand, while some smaller datasets seem to favor small sample values (notably boston and concrete), larger sample sizes seem much more robust overall, especially the larger datasets in the benchmark (naval, power, protein).
C.13 PERFORMANCE COMPARISON AGAINST DISCO NETS
As described before, the closest prior approach is DISCO Nets (Bouchacourt et al. 2016). where the authors introduce multivariate distributional regression method for hand-pose estimation. Remark- ably, they also use injected random noise for nondeterminism (as in Section 4.1), and derive a loss function which, with the correct parametrization, very similar to a multivariate version of the ba- sic non-weighte weighhet version of our method, essentially reproducing the Energy Score (see Section 4.5 which the authors also highlight.
As the DISCO Nets method is not primarily formulated for univariate regression, for fairness we first compare on the original task of (Bouchacourt al. (016)). DISCO Nets is originally measured on the NYU Hand Pose Estimation dataset (Tompson et al., 2014) as preprocessed by OOeerweger al.) (2015). The processed dataset contains 72,757 training and 8254 testing frames, with the sets consisting of RGBD images of hands, taken from 3 viewpoints. However, only the front view and the depth channel is used for the experiment. For each data point, the label consist of the 3d position of 14 finger joints.
To compare, we use the original code of DISCO Net:" to run the original preprocessing pipeline to acquire the train and test sets. We also use the same network size as BBouchacourt et al. (2016). consisting of 3 convolutional layers with filters each, max pooling layers, and 3 linear layers of size 1024 with ReLU activation.
Performance is measured as in Bouchacourt et al. (2016), in terms of Energy Score (denoted as ProbLoss, smaller is better), Mean Joint Fuclidean Error (MeJEE, smaller is better). Max Joint Euclidean Error (MaJEF, smaller is better) and Fraction of Frames within distance (FF, larger is better), including standard error of the mean (SEM) for the first three. For exact definitions, please refer to the original paper and its supplement. We take scores for DISCO Nets and basclines from the original paper, and present scores for WCRPS and WCRSP, & in Table I3 As we can observe, the improved models outperform DISCO Nets, with the WCRPS, model performing really strongly in particular. While deterministic measures (MeJEE, MaJEE, FF) also show higher scores, the performance difference is most apparent in ProbLoss, hinting much better predictive distributions.
Second, we implement the method on the UCI Benchmark and evaluate performance under the same conditions as in Section 5.3.1 For hyperparameters not explicitly investigated in Bouchacour] ctt al. (2016), we adopt the same values as the original authors (e.g., 3 1). Since DISCO Nets also generates samples rather than directly modeling the density, apply the KDE approach described in Appendix I to compute the NLL.
https://github..
weee--ggrrpppoo/ S SCONets
30

---

Published as a conference paper at ICLR 2025
Table 13: Performance comparison against Bouchacourt al. (0166) and its baselines using the multivariate WCRPS and WCRPS, models on the NYU Hand Pose dataset, +EM.
Model
ProbLoss (mm)
McJEE (mm)
MaJEE (mm)
FF (80mm)
BASE 1=100=1
103.8+006
25.2++02
52.7+0022
86.040
BASE 0=1,0=5
99.3++0.6
25.5+0.2
52.9+0.3
85.773
BASE 1=
96.3+10.6
25.7
53.2:+0.3
85.664
DISCO =
925
21.6+0.11
46.0:E0.3
92.971
DISCO 1,
89.
21.2::.11
46,4
93.262
DISCO 0.5
83.8
20.9
45.1
94.438
WCRPS WCRPS.
72.1
19.5#0.1
39.0:10.2 37.6++0.2
96.510
63.7++0.5
18.8+-0.1
96.958
Table 14: NLL results on the UCI Datasets benchmark (lower is better). Scores that perform at least on par when compared to the results in Table are highlighted in bold.
dataset
dropout
CRPS
WCRPS,
DISCO T
DISCO 2=1 7 mm0.22
DISCO
BNN-LV
7=0.5
boston
2.00
2.38::00
2.32.10.0 10.05
3.44
2.86
2.34
2.68 1:002
conerete
2.04
3.11:000 1:01
2.95
4.16
::00
3.02:10.03
3.31 1:000
energy
1,21 10.01
0.46:0004
0.31
2,61 1:011
0.82
0.65 +oo0s
2.32 +0002
kin8nm
-1.14
-1.34
-1.38
0,81 +0003
-0.83 4:0.01
-1.33 4:001
-1.24+t001
naval
-4,55
1:00
-6.51
-6.59 20.03
-6.47
-6.04.10.0
-5. 5.1411001
power
2.81 =0.01
2.66
2.57
2.54
3. 2.03
2.71 1:0.0
2.83
protein
2.87=
2.29 1::00
2.05 1000
5.30
inf
2.34 1.0.01
2.59 +0:01
wine
0.93
0.89:0000
1..0
2.62
1.75 0.006
0.94 1:0.0
1.00
yacht
1.25
0.07
-0.16
0.29
0.10
0.37
2.09
the authors propose using the MEU (Maximum Expected Utility) point, conceptually similar to the geometric median, and suggest employing MEU-based RMSE for early stopping by comparing it to the ground truth. However, in our experiments, we resort to using NLL for early stopping, as MEUUbased RMSE proved ineffective; it often failed to decrease beyond certain threshold or even increased from its starting point. In a univariate regression problem, the output distribution's 's geometric median is equivalent the median, thus MEU is essentially defined as the median of the sample. Given that the median generally does not coincide with the mean which known to be optimal for RMSE, this outcome is not unexpected.
In Bouchacouur et al. (2016). al.
We test three hyperparameter configurations for DISCO Nets, as presented in Tables 14 and 15 As expected, the theoretically optimal variant (y 0.5) performs well, while other configurations yield suboptimal results. However, even the correctly parameterized version is outperformed by our best method. This outcome is again unsurprising, since with the correct hyperparameter settings, adaptations for regression benchmarking, and other enhancements we apply for training, the method essentially replicates the unweighted CRPS approach trained using small sample size for loss calculation.
C. PERFORMANCE COMPARISON AGAINST BNN-LV
The method introduced in Depeweg et al. 2018 (BNN-LV) theoretically capable of representing complex distributions, such multi-modal ones. However, the original paper primarily evaluates the method on reinforcement learning tasks and lacks detailed quantitative analysis. Therefore, we choose not benchmark against using its original datasets. Fortunately, recent uncertainty quantification framework (Lehmann et al. 2024) implements BNN-LV for regression. Given that the first author of Depeweg al. (2018) also a co-author of Lehmann et al. (202)) and was
When a ground truth point falls too far outside the predicted distribution, the likelihood function hecomes essentially 0 numerically, causing the NL.. metrie take on the value of
31

---

Published as a conference paper at ICLR 2025
Table 15: RMSE results on the UCI Datasets benchmark (lower better). Scores that perform at least on par when compared to the results in Table 2 are highlighted in bold.
dataset
dropout
CRPS
WCRPS,
DISCO 0=1
DISCO P=1 0.0.25
DISCO P=1 7=0.5
BNN-LV
boston
2..23
3.01
2..1
4.15
3,49
2.99:10.20
3.67
concrete
5,45 1.019
5,48
4.94
6,95
6.34
5.3611017
6.54
energy
0.97
0.44
0.41
2.87
0.71 1:015
0.56:10.02
3.00
kin8nm
0,09:00.0
0.07 +:000
0.06
0,08=aom
0.07 4:0,00
0.07 +0000
0.07 +:000
naval
=..0
5.00
+0000
0.00
8,00
6.00
+.00
power
4.18
2.79
3.62
2.23
4.55
3.95
4.17 10000
protein
20.02
3.77
3.00-2:000
5.75
2.70
3.96
4.79:
wine
0.66
0.64 10000
0.63
0.70
0.65 1:001
0.65 1:0.01
0.64:10.00
yacht
1.23
0.95 10.11
0.78 10.09
1.07
0.88 10010
1.05 1:010
3,38 1:0.0
substantially involved in the BNN-LV implementationf] we consider the framework of Lehmann et al. (2024) to be an authoritative reference implementation for BNN-LV.
the KDE approach outlined in Appendix C.1 and early stopping is performed based on measuring the training loss on the validation set. As the method converges very slowly, needed to increase the maximum number of allowed epochs. However, since it also requires relatively long per-epoch training times, the approach ended up taking approximately 55X long our fastest measured method MDN (or over 86 hours) on the full benchmark, mak- ing it impractical to optimize over multiple hyperparameter combinations. Instead, we employ the hyperparameters suggested in the framework's 's manual. The results are summarized in Tables [4 and 15
For NLL calculations,
On many datasets, BNN-LV produces respectable scores, often comparable classic baselines such as dropout in NLL, however often underperforms in RMSE. In general it does not produce com- petitive scores when compared the best measured con-Gaussian methods such as WCRPS, or MDN
32


Reviews:
[{"review_id": "hV5tXh2oB5", "reviewer": "Reviewer_59RD", "summary": "This paper presents an for modeling aleatoric uncertainty in neural network regression tasks using a nondeterministic architecture optimized with loss functions derived from the continuous ranked probability score (CRPS). The method enables learning arbitrary output distributions in a nonparametric way by training the network to generate samples from the target's aleatoric distribution and the training the underlying model for regression. The authors present interesting results on inverse MNIST and a set of synthetic and non-synthetic UCI datasets.", "questions": "- Isn't heteroscedatic regression also modeling aleatoric uncertainty by explicitly putting an uncertainty on the outputs? \n\n- I think it'll be useful if you also explain why are we focused on cumulative distribution and expanding section 2.4 so that it's more accessible to readers. \n\n- Can you talk about how we would scale this to larger models?\n\n- Can you also talk how about how this method would saturate with # of samples drawn?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- I like the fact that this paper focuses on aleatoric uncertainty as opposed to total uncertainty or epistemic uncertainty. Aleatoric uncertainty has been largely overlooked \n\n- The architecture choice is very interesting and intuitive and the results presented are very useful. The loss function is also described well theoretically and it's nice to see it build upon existing literature for a new task. \n\n- The paper presents extensive experiments on both synthetic and real-world datasets, including univariate and multivariate problems.", "weaknesses": "- Heteroscedatic regression where we explicitly model the prediction and the uncertainty around it is very related to this line of work. However, the authors do not describe why should one prefer that line of work versus their approach. See [1] for an example on heteroscedatic regression. \n\n- While the authors do acknowledge it, the current method can be computationally expensive. This might be problematic as we try to scale larger architectures.\n\nReferences\n\n[1] Korattikara, A., Rathod, V., Murphy, K. and Welling, M., 2015. Bayesian dark knowledge. arXiv preprint arXiv:1506.04416.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "T4WPLNdPiq", "reviewer": "Reviewer_yrBT", "summary": "The manuscript introduces 4 new architectures of nondeterministic neural networks for producing samples from the aleatoric target distribution. Furthermore, a new loss function for optimizing neural networks based on an empirical approximation of the \"continuous ranked probability score\" (CRPS) is introduced. Test over standard datasets for classification tasks as well as regressions tasks are discussed.", "questions": "- The formulation of the problem at rows 92--96 should be more clear, in particular regarding the the definition of \"small omega\".\n- at \"The described setup provides, to our knowledge, the first nonparametric way to learn aleatoric uncertainty for neural network regression\" rows 286-287. How do we know the uncertainty is aleatoric and not epistemic?\n- Eq. 3: the loss is given by the difference between the cumulative distribution F of the labels for a batch and the stochastic results created by the network?\n- the process of generating the different outcomes in figures 2 and 3 is not clear enough. \n- In the benchmarking, are only 5 ensembles used? Is this shown / clear to be an ensemble large enough?\n- could the process of generating the underlying distribution on the NLL (row 442) be briefly summarized? It is true this comes form literature, but a reminder could help the readability.", "limitations": "", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The idea is simple but rigorous and the implementation straightforward. The performance improvements, especially for the single-head weighted version with ensembling are small but consistent.", "weaknesses": "The presentation should be improved, see questions", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "SsFn48DJxA", "reviewer": "Reviewer_yUKD", "summary": "UQ is important for predictive modeling. Most existing methods assume parametric distributions around the true value, limiting their ability to capture skewed, multi-modal distributions.\nThe paper proposes a nonparametric, nondeterministic NN regression architecture that uses a loss function based on the continuous ranked probability score to capture the aleatoric uncertainty for systems with complex distributions.\nThe method is evaluated on a few benchmarks, showing better performance compared over other models.", "questions": "1. Could you clarify how injecting standard normal noise into the latent vector is expected to better capture aleatoric uncertainty?\n2. Considering the higher computational costs reported compared to other methods, would this make it challenging to scale up to real, complex problems?", "limitations": "", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 1, "contribution": 2, "strengths": "The paper gives a decent introduction on the general UQ concepts and offers a clear summary of the relevant work.\nTo the reviewer's best judge, the authors seem to have a good grasp of the theory.", "weaknesses": "1. The nondeterministic NN presented in the paper is essentially a deterministic NN with a direct noise injection to the latent representation vector. The paper doesn’t provide a strong justification for why this approach would be effective, making the idea feel somewhat arbitrary.\n2. To the best of the reviewer’s knowledge, some statements in the paper seem problematic. For example, \"In other words, epistemic uncertainty can be interpreted as the uncertainty of aleatoric uncertainty.\" (Line 139, page 4).\n3. It is unclear whether the method is practically useful. As presented in the paper, the method requires multiple inferences at each training and evaluation step, introducing additional computational cost. This cost is expected to increase for more complex target distributions, making the method less practical for complex systems with skewed, multi-modal non-Gaussian distributions—precisely the motivation behind this work.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "oSwsuKjElz", "reviewer": "Reviewer_cN1m", "summary": "The authors approach the task of modeling irreducible (aleatoric) uncertainty in deep regression models. To do this, they rely on the _continuous ranked probability score (CPRS)_ as the target loss rather than, e.g., a parametric likelihood. They introduce an efficient unbiased estimator to the CRPS with a cost of $O(n\\log(n))$ compared to a naive $O(n^2)$ approach.\nThe method is evaluated on a range of regression tasks.\n\n\n_____\n_Update: Given the changes to the submission provided during the rebuttal discussion, I am increasing my score accordingly._", "questions": "- Q1: Can the authors comment to a greater degree on the relationship between their proposal and the existing literature?\n- Q2: All networks in the paper are rather small. How do the authors expect their method to scale as the architectural structure becomes deeper?", "limitations": "", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "- The paper is well-written and easy to follow.\n- The method is well-motivated and the proposed CRPS-based approach provides a clear and straightforward approach to settings where it is beneficial to move away from more common unimodal likelihood assumptions (although see below).\n- The proposed approach is computationally scalable.\n- The appendix and provided codebase seem to ensure full replicability. (However, I didn't check the code in detail and only skimmed it.)\n- At least on the rather simple UCI regression tasks the model is well calibrated. It would have been interesting to see regression problems that require deeper nets, e.g., an evaluation of a depth-estimation data set from the computer vision literature.", "weaknesses": "- The major weakness is in the framing of the storyline. Again and again throughout the text, the authors claim that aleatoric has not been of interest to the literature or has been only _\"treated as an afterthought\"_ (l180).\nI strongly disagree with this statement. Well-modeled aleatoric uncertainty has a strong presence in the literature, as is, e.g., discussed in the cited review paper by Gawlikowski et al. (2023). \nBe it in computer vision (Kendall & Gal, 2017; Gast & Roth, 2018,...), any normal likelihood-based paper that learns a homoscedastic or heteroscedastic noise model, evidential deep learning/prior network-based approaches such as Amini et al. (2020) or Malinin et al. (2020), and many more. \nAccurately modeling both aleatoric and epistemic uncertainty has, e.g., the important application of active learning, where information should be requested about data the model is uncertain in a reducible rather than an irreducible sense and other approaches. \nAs such, aleatoric uncertainty is not an afterthought, but rather an integral component of those models. However, where I agree, and which would be a more correct phrasing of the problem, is that most of these focus on an assumption of a univariate noise structure rather than a multi-modal one as the authors allow for.\nYet, this constraint is not an inherent one or one of ignorance in the literature. Lakshminarayanan et al. (2017), e.g., explicitly mention the proposal of extending their method with MDNs if a uni-modal normal likelihood is too restrictive. \nThis restriction is also regularly lifted in the literature. See, e.g., the work of Depeweg et al. (2018), who allow for multi-modality via latent variables to the input of BNNs, or Bouchacourt et al.'s (2016) DISCO nets, who inject noise in a later part of the deterministic nets, in an architectural similarity with the proposal by the authors. \nNon-normal likelihoods have also been the focus of, e.g., Gaussian Processes (Sendera et al., 2021), Normalizing Flows (e.g., Ardizzone et al. 2019), and other probabilistic models.\nThe authors' proposal is innovative and novel, but not unique in the sense that nobody has cared about aleatoric uncertainty since Bishop (1994). \nThis last sentence is a slightly exaggerated version of how parts of the paper read. I don't claim that the authors think that, but both the discussion and experimental evaluation ignore this field almost completely.  \n- The discussion on how the method fits into the literature, and that the proposal is primarily on nonparametric vs unimodal noise structure should be clearer.\n- Given this, an experimental comparison against, e.g., a naive heteroscedastic (deterministic or Bayesian) neural net, DISCO Nets, or Depeweg et al. (2018) is necessary to evaluate the method properly.\n\n\n_____\nAdrizzone et al., Analyzing inverse problems with invertible neural networks, ICLR 2019  \nBouchacourt et al., DISCO Nets: DISSimilarity COefficient Networks, NeurIPS 2016  \nDepeweg et al., Decomposition of Uncertainty in Bayesian Deep Learning for Efficient and Risk-sensitive Learning, ICML 2018  \nGast & Roth, Lightweight Probabilistic Deep Networks, CVPR 2018  \nMalinin et al. Regression Prior Networks, 2020  \nSendera et al., Non-Gaussian Processes for Few-Shot Regression, NeurIPS 2021", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)