How Reliable is Your Regression Model's Uncertainty Under Real-World Distribution Shifts?
Authors: Fredrik K. Gustafsson
Martin Danelljan
Thomas B. Schön
Venue: Accepted by TMLR ()

Abstract: Many important computer vision applications are naturally formulated as regression problems. Within medical imaging, accurate regression models have the potential to automate various tasks, helping to lower costs and improve patient outcomes. Such safety-critical deployment does however require reliable estimation of model uncertainty, also under the wide variety of distribution shifts that might be encountered in practice. Motivated by this, we set out to investigate the reliability of regression uncertainty estimation methods under various real-world distribution shifts. To that end, we propose an extensive benchmark of 8 image-based regression datasets with different types of challenging distribution shifts. We then employ our benchmark to evaluate many of the most common uncertainty estimation methods, as well as two state-of-the-art uncertainty scores from the task of out-of-distribution detection. We find that while methods are well calibrated when there is no distribution shift, they all become highly overconfident on many of the benchmark datasets. This uncovers important limitations of current uncertainty estimation methods, and the proposed benchmark therefore serves as a challenge to the research community. We hope that our benchmark will spur more work on how to develop truly reliable regression uncertainty estimation methods.

Published in Transactions on Machine Learning Research (09/2023)
How Reliable is Your Regression Model's Uncertainty Under Real-World Distribution Shifts?
Fredrik K. Gustafsson
Deportment of Information Technology
Uppsala University, Sweden
Martin Danelljan
Computer Vision Lab
ETH Zürich, Switzerlond
Thomas B. Schön
Department of Information Technology Uppsala University, Sweden
Reviewed on OpenReview: https: /ttps://openrevve wwtpr//eepereeee...mm net/ forum? id- WJt2Pc3qtI
Abstract
Many important computer vision applications are naturally formulated as regression problems. Within medical imaging, accurate regression models have the potential to automate various tasks, helping to lower costs and improve patient outcomes. Such safety-critical deployment docs however require reliable estimation of model uncertainty, also under the wide varicty of distribution shifts that might be encountered in practice. Molivated by this, we set out to investigate the reliability of regression uncertainty estimation methods under various real- world distribution shifts. To that end, propose extensive benchmark of 8 image-based regression datasets with different types of challenging distribution shifts. We then employ our benchmark to evaluate many of the most common uncertainty estimation methods, as detection.
well as two state-of-the-ars uncertainty scores from the task of We find that while mothods are well calibrated when there is no distribution shift, they all become highly overconfident on many of the benchmark datasets. This uncovers important limitations of current uncertainty estimation methods, and the proposed benchmark therefore serves a challenge to the research community. We hope that our benchmark will spur more work how to develop truly reliable regression uncertainty estimation methods. Code is available at
wtt.ba.cc//reepeeeeeegrrssiioooo oagpp//eeerrereeesosoooocrrrrrrr uncertainty
1 Introduction
Regression is a fundamental machine learning problem with many important computer vision applications
RRothe et al. 2016 Law & Deng 2018 Xiao et al. 2018 Zhu et al. 2018 Shi et al. 2019 Lathuilière et al. 2019) In general, it entails predicting continuous targets y from given inputs x. Within medical imaging, a number of tasks are naturally formulated as regression problems, including brain age estimation (Cole et al. Jónsson et al.) 2019 Shi et al. 2020), prediction of cardiovascular volumes and risk factors (Xue el. al, Poplin et al.. 2018 and body composition analysis (Langner et al. 2020 2021). If machine learning
2017
2017
models could be deployed to automatically regress various such properties within real-world clinical practice, this would ultimately help lower costs and improve patient outcomes across the medical system (Topol 2019
Real-world deployment in medical applications, and within other safety-critical domains, does however put very high requirements on such regression models. In particular, the common approach of training deep neural network (DNN) directly output predicted regression target I = f(x not sufficient, as fails capture any measure of uncertainty in the predictions ÿ. The model is thus unable to e.g. detect inputs æ

---

Published in Transactions on Machine Learning Research (09/2023)
which are out-of-distributiin (OOD) compared to its training data. Since the predictive accuracy of DNNs
typically degrades significantly on OOD inputs (Hendrycks & Dietterich, 2019 Koh et al. 2021), this could
have potentially catastrophic consequences. Much research effort has therefore been invested into various
approaches for training uncertainty-aware DNN models [Bhndell et al.) 2015 Gal 2016 Kendall & Gal 2017
Takshminarayanan et al. 2017 Romano el al. 2019;; to explicilly estimale the uncertainly in the predictions.
These uncertainty estimates must however be accurate and reliable. Otherwise, if the model occasionally becomes overconfident and outputs highly confident yet incorrect predictions, providing uncertainty estimates might just instill false sense of security arguably making the model even less suitable for safety-critical deployment. Specifically, the uncertainty estimates must be well calibrated and properly align with the prediction errors (Gneiting et al. 2007 Guo et al.) 2017). Moreover, the uncertainty must remain well calibrated also under the wide variety of distribution shifts that might be encountered during practical deployment (Quionero- Canioll et al. 2009 Finlayson et al. (021). For example in medical applications, a model trained on data collected solely at large urban hospital the year 2020, for instance, should output well-calibrated predictions also in 2023, for patients both from urban and rural areas. While uncertainly calibration, well as general DNN robustness (Hendrycks et al. 2021a), has been evaluated under distribution shifts for classification tasks (Oadia et al. 2019), this important problem is not well-studied for regression.
Motivated by this, we set out to investigate the reliability of regression uncertainty estimation methods under various real-world distribution shifts. To that end, we propose an extensive benchmark consisting of image-based regression datasets (see Figure D with different types of distribution shifts. These are all publicly available and relatively large-scale datasets (6 592 20 614 training images), yet convenient to store and train models on (64 x 64 images with 1D regression targets). Four of the datasets are also taken from medical applications, with clinically relevant distribution shifts. We evaluate some of the most commonly used regression uncertainty estimation methods, including conformal prediction, quantile regression and what is often considered the ofate-of-the-art- - ensembling (Ovadia et al. consider the approach of selective prodiction (Geifman & Yaniv 2017), in which the regression model can abstain from outputting predictions for certain inputs. This enables us to evaluate uncertainty scores from the rich literature on OOD detection (Salehi et al.) 2022). Specifically, we evaluate two recent scores based featuro-space density (Mukhoti et al. 2021a Pleiss et al., 2020 Sun et al. 2022 Kuan & Mueller 2022
2019 Gustafsson et al. 2020b). We also
In total, we evaluate 10 different methods. Among them, we find that not a single one is close to being perfectly calibrated across all datasets. While the methods are well calibrated on baseline variants with no distribution shifts, they all become highly overconfident on many of our benchmark datasets. Also the conformal prediction methods suffer from this issue, despite their commonly promoted theoretical guarantees. This highlights the importance of always being aware of underlying assumptions, assessing whether or not they are likely to hold in practice. Methods based on the state-of-the-art OOD uncertainty scores perform well relative to other methods, but are also overconfident in many cases the absolute performance is arguably still not sufficient. Our proposed benchmark thus serves as challenge to the research community, and we hope that it will spur more work on how to develop truly reliable regression uncertainty estimation methods.
Summary of Contributions We collect a set of 8 large-scale yet convenient image-based regression datasets with different types of challenging distribution shifts. Utilizing this, we propose a benchmark for testing the reliability of regression uncertainty estimation methods under real-world distribution shifts. We then employ our benchmarkk to evaluate many of the most common uncertainty estimation methods, as well as two tate-offtheeart uncertainty scores from OOD detection. We find that all methods become highly overconfident. many of the benchmark datasets, thus uncovering limitations of current uncertainty estimation methods.
2 Background
In a regression problem, the task is to predict a target y* E y for any given input x* E X. To solve this, we are also given a train set of i.i.d. input-target pairs, Durain = ((x,, vi)) (2,, ~ p(x, y). What separates regression from classification is that the target space y continuous, y' y'= =K. In this work, we only consider the 1D case, ie.. when y = R. Moreover, the input space X' here always corresponds to the space of images.
2

---

Published in Transactions on Machine Learning Research (09/2023)
Prediction Intervals, Coverage & Calibration Given a desired miscoverage rate a, a prediction interval Ca(x*) = (x') (x()') S IR is function that maps the input x' onto an interval that should cover the true regression target y² with probability 1 - a. For any set {(,,, hi of N examples, the empirical interval coverage is the proportion of inputs for which the prodiction interval covers the corresponding target,
N not E C. (x)))
Coverage(Ca) 1
Σ N
(1)
If the coverage equals 1 a, we say that the prediction intervals are perfectly calibrated. Unless stated otherwise, we set a = 0.1 in this work. The prediction intervals should thus obtain a coverage of 90%.
2.1 Regression Uncertainty Estimation Methods
The most common approach to image-based regression is to train a DNN fo X - R that outputs predicted target V (1) for any input I, using e.g. the L2 or L1 loss (Lathuilière et al. 2019). We are interested in methods which extend this standard direct regression approach to also provide uncertainty estimates for the predictions. Specifically, we consider methods which output a prediction interval (a and a predicted target ((z) E (ar for each imput x. The uncertainty in the prediction ğ(z) is then quantified as the length of the interval C.(2) (1) (larger interval higher uncertainty). Some of the most. commonly used regression uncertainty estimation methods fall under this category, as described in more detail below.
Conformal Prediction The standard regression approach can be extended by utilizing the framework of split conformal prediction (Papadopoulos et al. 2002 Papadopoulos 2008 Romano et al., 2019). This entails splitting the train set [[r,, [i)];= into a proper train set Z1 and a calibration set Zz. The DNN fo is trained on I, and absolute residuals R = fly; - (z;)) : E are computed on the calibration set Iz. Given a new input x", a prediction interval Cx(x) ) is then constructed from the prediction fo(x") as,
Ca(x*) [Fo(x*) - QR, So(x*) + QR-a(R,
(2)
where QR, I2) is the (1-a))-h quantile of the absolute residuals R. Under the assumption of exchangeably drawn train and lest data, this prediction interval is guaranteed to satisfy Ppy" E Ca(x*)) 2 1 - a (marginal coverage guarantee). The interval (x*) has a fixed length of 2Q1-a
for all inputs x².
Quantile Regression A DNN can also be trained dircctly output prediction intervals of input-dependent length, utilizing the quantile regression approach (Koenker & Bassett Jr
1978 Romano al., 2019 Jensen
et al. 2022). This entails estimating the conditional quantile function qd (x) inf R : Fyix (y)=) 2 a),
where Fyix is the conditional cumulative distribution function. Specifically, a DNN is trained to output estimates of the lower and upper quantiles n (xx,, (x) at ag a/2 and Cup = 1 a/2. Given new input x*, a prediction interval Ca (x*) can then be directly formed as,
Ca (xx)) [16
1 (x*),
28 Oup (x*)).
(3)
The estimated quantiles do ale (x*)
Os dup (x*) can be output by single DNN fe, trained using the pinball loss (teinwarrt & Christmann 2011). A prediction ((**) can also be extracted as the center point of C. (x*).
Probabilistic Regression Another approach is to explicitly model the conditional distribution p(y [r), for example using a Gaussian model po(y)z)= = (1: (z)),, oz))) (Chua et al. 2018 Gustafsson et al. 2020ah. A single DNN fo can be trained to output both the mean (o( and variance (z()) by minimizing the corresponding negative ligeliiel For a given input x", a prediction interval can then be constructed as,
Ca (x*) = [ua(=*) - -x(x=)00000000000) - (1 a/2), no(x*) + oo(x=) '(1 a/2)],
(4)
where & is the CDF of the standard normal distribution. The mean (0(x") ) is also taken as a prediction ÿ. Epistemic Uncertainty From the Bayesian perspective, quantile regression and Gaussian models capture alcatoric (inherent data noise) but not epistemic uncertainty, which accounts for uncertainty in the model Kendall & Gal 2017). This can be estimated in principled manner via Bayesian
parameters Gaa 2016
3

---

Published in Transactions on Machine Learning Research (09/2023)
Cells- Tails
ChairAngle- Gap
AssetWealth
Ventricular Volume
Brain Tumour Pixels
SkinLesion Pixels
Histology NucleiPixels
AerialBuilding Pixels
Train
y 100
y
y 1.594
y 40.38
y
y
y
y
Test
198
y 43.8
1.314
85,93
273
=1616
y= 1156
y 433
Figure 1: We propose a benchmark consisting of imago-basee based regression datasets, testing the reliability of regression uncertainty estimation methods under real-world dist.ribution shifts. Example train (lop row) and test inputs z, along with the corresponding ground truth targets y, are here shown for each of the 8 datasets.
inference, and various approximate methods have been explored (Neal 1995
Ma et al. 2015 Blundell et al,
2015: Gal & Ghahramani 2016). In practice, it. has been shown difficult. to beat the simple approach of ensembling [Lakshmmnarayanan et al., 2017 Ovadia et al. 2019), which entails training M models J and combining their predictions. For Gaussian models, single mcan á and variance 6² can be computed as,
M
1
0²(x") = Σ (p(x')) - no, (x*))) oz (x'))
A(x*)
no, (x").
(5)
M
and then plugged into
4 construct a prediction intorval Ca Cx(") for given input xt
2.2 Selective Prediction
The framework of selective prediction has been applied both to classification (Herbei & Wegkamp) 2006 Geifnan & EJ- El-Yanivy 2017)) and regression problems (Jiang et al.) 2020). The general idea give a model the option to abstain from outputting predictions for some inputs. This is achieved by combining the prediction model with an uncortainty function Ky R. Given an input x", the prediction fo(x*) is output if the uncertainty ky(x*) S (for some user-specified threshold 7), otherwise xf is rejected and no prediction is made. The prediction rate is the proportion of imputs for which prediction is output,
N N Σ Hn) ST..
Predition Rate
In principle, if high uncertainty ky(x*) corresponds to large prediction error ly* - -(x*)] and vice versa, small errors will be achieved for all predictions which are actually output by the model. Specifically in this work, we combine selective prediction with the regression methods from Section 2.1 A prediction interval Ca(x") and predicted target f(x*) are thus output and only (iff) Ky (x*)) S r. Our aim is for this improve the calibration (interval coverage closer to a:) of the output prediction intervals.
For the uncertainty function Ky (x), the variance 8² of Gaussian ensemble 5 could be used, for example. One could also some of the various uncertainty scores employed in the rich OOD detection literature (Salehi ct al. 2022). In OOD detection, the task is to distinguish in-distribution inputs inputs which are similar to those of the train set {(r,, W from out-of-distribution inpuls. A principled approach to OOD detection would be to a model of p(œ) on the train set. Inputs z for which p(z) is small are then deemed OOD (Sorrà et al. 2020). In our considered case where inputs x are images, modelling p(z) can however be quite challenging. To mitigate this, a feature extractor g : Rs can be utilized, modelling p(z) indirectly by fitting simple model to the feature vectors g(z). In the classification setting, Mukhoti al. (2021a) fit.
4

---

Published in Transactions on Machine Learning Research (09/2023)
Gaussian mixture model (GMM) to the feature vectors {9(2;) li=1 of the train set. Given an input 2", it is then deemed OOD if the GMM density (MM(g(2*) is small. Pleiss et al. (2020) apply this approach also to regression problems. Instead of fitting a GMM to the feature vectors and evaluating its density, (Sun et al.) 2022
Kuan & Mueller 2022) compute the distance kNN(g(x*)) between g(ø") and its k nearest neighbors in the train set. (g(r)));=1 The input x is then deemed OOD this kNN distance is large.
3 Proposed Benchmark
We propose an extensive benchmark for testing the reliability of regression uncertainty estimation methods under real-world distribution shifts. The benchmark consists of 8 publicly available image- regression datasets, which are described in detail in Section 3.1 Our complete evaluation procedure, evaluating uncertainty estimation methods mainly in terms of prediction interval coverage, is then described in Section 3.2
3.1 Datasets
In an attempt to create a standard benchmark for image-based regression under distribution shifts, we collect and modify datasets from the literature. Two of them contain synthetic images while the remaining six are real-world datasets, four of which are taken from medical applications. Examples from each of the 8 datasets are shown in Figure ! We create two additional variants of each of the synthetic datasets, thus resulting in 12 datasets in total. They are all relatively large-scale (6 59222- 20 614 train images) and contain input images v of size 64 x 64 along with 1D regression largets Descriptions of all 12 datasets are given below (further details are also provided in Appendix A] starting with the two synthetic cltasets and their variants.
Cells Given a synthetic fluorescence microscopy image x, the task is to predict the number of cells y in the image. We utilize the Cell-200 dalaset from Ding et al. (2021 2020), consisting of 200 000 grayscale images of size 64 x 64. We randomly draw 10 000 train images, 2 000 val images and 10000 test images. Thus, there is distribution shift between train/ val and test. We therefore use this as baseline dataset.
Cells-Tails A variant of CELLS with a clear distribution shift between train/val /val and test.. For train/ val, the regression targets y are limited to |50, 150). For test, the targets instoad lie in the original range [1, 200]
Cells-Gap Another variant of CELLS with clear distribution shift between train /val and test. For train train/val, /aal, the regression targets y are limited to [1, 5)51 200). For test, the targets instead lie in the original [1, [00).
ChairAngle Given a synthetic image x of chair, the task is to predict the yaw angle y of the chair. We utilize the RC-49 dataset (Ding et al. 2021 2020), which contains 64 x 64 images of different chair models rendered at yaw angles ranging from 0.1º to 89.9". We randomly split their training set and obtain 17 640 train images and 4410 val images. By sub-sampling their test set we also get 11 225 test images. There is no clear distribution shift between train/val /vll and test, and we therefore use this as a second baseline dataset.
ChairAngler Tails A variant of CHAIRANGLE with clear distribution shift between train/val /val and test. For train/val, we limit the regression targets y to |15, 75]. For test, the targets instead lie in the original Jo, 90
Chair AlairAngle-Coe Gag Another variant of CHAIRANGLE with a clear distribution shift. For train/ val, the regression targets are limited to J0, 30]J660, 90[. For test, the targets instead lie in the original 90
Asset Wealth Given a satellite image x, the task is to predict the asset wealth index y of the region. We utilize the Poverty Wilds dataset from (Koh et al. 2021), which is a variant of the dataset collected by Ych et al. (2020). We use the training, validation-ID and test-OOD subsets of the data, giving us 9 797 train images, 100 val images and 3 3633 test images. We resize the images from size 224 224 to 64 64. Train/val and test contain satellite images from disjoint sets of African countries, creating distribution shift.
VentricularVolun Given an echocardiogram image x of human heart, the task is to predict the volume of the left ventricle. We utilize the EchoNel-Dynamic dataset (Ouyang et al.) 2020), which contains 10030 echocardiogram videos. Each video captures a complete cardiac cycle and is labeled with the left ventricular volume at two separate time points, representing end-systolc (at the end of contraction) and end-diastole (just before contraction). For each video, we extract just one of these volume measurements along with the corresponding video frame. To create a clear distribution shift between train vrainvaal /va and test, we select the end
5

---

Published in Transactions on Machine Learning Research (09/2023)
systolic volume (smaller volume) for train and val, but the end diastolic volume (larger volume) for test. We utilize the provided dataset splits, giving us 7460 train images, 1 288 val images and 1 276 test images.
BrainTumouPPixels Given an image slice I of a brain MRI scan, the task is to predict the number of pixels y in the image which are labeled as brain tumour. We utilize the brain tumour dataset of the medical segmentation decathlon (Simpson et al. 2019 Antonelli et al., 2022), which a subset of the data used in the 2016 and 2017 BraTS challenges (Bakas et al. 2018 2017 Menze al.) 2014). The dataset contains 484 brain MRI scans with corresponding tumour segmentation masks. We split these scans 016/22%%/20 into train, val and test sets. The scans are 3D volumes of size 240 x 240 X 155. We convert each scan into 155 image slices of size 240 x 240, and create regression target for cach image by counting the number of labeled brain tumour pixels. This gives us 614 train images, 616 val images and 6 652 test images.
SkinLesionPixels Given a dermatoscopic image z of a pigmented skin lesion, the task is to predict the number of pixels y in the image which are labeled as lcsion. We utilize the HAM10000 datasct by [sccand]] et al. 2018), , which contains 10015 dernaloscopic images with corresponding skin lesion segmentation masks. HAM10000 consists of four different sub-datasets, three of which were collected in Austria, while the fourth sub-dataset was collected in Australia. To create clear distribution shift between train /val and test, we use the Australian sub-dataset as our test set. After randomly splitting the remaining images 85% /15% into train and val sets, we obtain 6592 train images, 1 164 val images and 2 259 test images.
HistologyNNucliPiixels Given an H&E stained histology image I, the task is to predict the number of pixels y in the image which are labeled as nuclei. We utilize the CoNSeP dataset by Graham et al. (2019). along with the pre-processed versions they provide of the Kumar (Kumar et al. 2017) and TNBC Naylor et al., 2018) datasets. The datasets contain large H&E stained image tiles, with corresponding nuclear segmentation masks. The three datasets were collected at different hospitals/instiiuu /isttiitiiiiins with differing procedures for specimen preservation and staining. By using CoNSeP and Kumar for train/val /val and TNBC for test, we thus obtain a clear distribution shift. From the large image tiles, we extract 64 x 64 patches via regular gridding, and create a regression target for each image patch by counting the number of labeled nuclei pixels. In the end, we obtain 10 808 train images, 702 val images and 267 test images.
AcrialBuildingPixcck Given an acrial image x, the task is to prodict the number of pixcls y in the image which are labeled as building. We utilize the Inria acrial image labeling dataset (Maggiori et al. 2017 which contains 180 large aerial images with corresponding building segmentation masks. The images are captured at five different geographical locations. We use the images from two densely populated American cities for arain/val, and the images from a more rural European area for test, thus obtaining a clear distribution shift. A After preprocessing, we obtain 11 184 train images, 797 val images and 890 test images.
Constructing these custom datasets is one of our main contributions. This is what enables us propose extensive benchmark of large-scale yet convenient datasets (which are all publicly available), containing different types of challenging distribution shifts, specifically for imsge-based regression.
3.2 Evaluation
We propose to evaluate regression uncertainty estimation methods mainly in terms of prediction interval coverage [ if a method outputs a prediction (xx) and a 90% prediction interval (o.1()) for each input x, does the method actually achieve 90% coverage on the test set? Le., are the prediction intervals calibrated?
Motivation Regression uncertainty estimation methods can be evaluated using various approaches. One alternative sparsification (llg et al., 2018), measuring how well the uncertainty can be used to sort predictions from worst to best. Perfect sparsification can however be achieved even if the absolute scale of the uncertainty is consistently underestimated. Therefore, lot of previous work has instead evaluated methods in terms of calibration (Gnciting et al. 2007 Guo et al. 2017). Specifically for regression, common form of calibration is based on quantiles (Kuleshov el al.) 2018 8 Cui el al. 2020 Chung et al.) (201)). Essentially a model is there said to be calibrated if the interval coverage (1) equals 1 - a for all miscoverage rates a e|0, 1[. This is 1 Σn |CoverageCaa. ) (1 - a3)], aj ~ U(0, 1). Our proposed evaluation metric is thus a special case of this approach, considering just one specific miscoverage rate 0.1. We argue that this results in simpler and more interpretable metric, which also is motivated
mcasured by the expected calibration error, ECE-
m
6

---

Published in Transactions on Machine Learning Research (09/2023)
by how prediction intervals actually are used in real-world applications. There, one particular value of a is selected (ax = 0.1 is a common choice), and the corresponding intervals C. are then expected to achieve a coverage of 1 - a on unseen test data. Recent alternative calibration metrics directly measure how well the uncertainty aligns with the prediction errors (Levi et al. 2022 Pickering & Sapsis 2022 While these enable relative comparisons of different methods, they are not easily interpretable in terms of absolute performance.
Implementation Details For each dataset and method, we first train a DNN on the train set Purain- Then, run the method on the val set D resulting in a prediction interval (x) = [L, (2t), (x)) for each input (a = 0.1). Importantly, we then calibrate these prediction intervals on val using the procedure in (Romano et al. 2019). This gives calibrated prediction intervals Ca (z), for which the interval coverage on val exactly equals a. Specifically, C. (x) is constructed from the original interval (x = (x), (x())),
Ca(x) = [Z)(z) - Q1-a Dval,, (z) + Q1-a Dval)
where E [max (La())) Di- Ut-Ua()))) : 3 are conformity scores computed on Pval, and Q1-a (E, Dyai) is the (1 a)-th quantile of these scores. Finally, we then run the method the test set. Drest, outputting calibrated prediction interval Ca(x*) for each input z², Ideally, the interval coverage of C. (xx) does not change from val to test, i.e. Coverage(Ca, = 1 - should be true also on test. If Coverage(C (Ca) & a, a a) is preferred compared to an overconfident method (< - a). For methods based on selective prediction (Section 2.2), the only difference is that prediction intervals (x²") are output only for some test inputs x² (iff ky(x*) S T). The interval coverage is thus computed only on this subset of test. This is similar to the notion of "selective calibration" " discussed for the classification setting by Fisch et al. 2022). We set a 0.1 since this is commonly used miscoverage rate in practice.
conservative method (> 1 a)
Secondary Metrics We also evaluate methods in terms of mean absolute error (MAE) and average interval length on the val set. This measures the quality of the prediction j(x) and the prediction interval C. (z), respectively (Gneiting et al. 2007). The average interval length is a natural secondary metrie, since a method that achieves a coverage close to 1- - but outputs extremely large intervals for all imputs x, not would be particularly useful in practice. Moreover, two different methods both are perfectly calibrated, i.e. a, the method producing smaller prediction intervals would be preferred. For methods Coverage(Ca)) prediction (which output predictions only for certain inputs z). the proportion of inputs based selective for which a prediction actually is output is another natural secondary metric. For these methods, we thus also evaluate in terms of the prediction rate 6 on test. Tf a coverage close to 1 - a is achieved with a very low prediction rate, the mod might still not be practically useful in certain applications. For two perfectly calibrated methods, one with a higher prediction rate would be preferred.
4 Evaluated Methods
We evaluate five common regression uncertainty estimation methods from Section 2.1 which all output a prodiction interval (z) and a predicted target ((x) E Cz(z) for cach input x. Two of these we also combine with selective prediction (Section 2.2 utilizing four different uncertainty functions kx(x)). In total, we evaluate 10 different methods. For all methods, backbone DNN. This architecture is chosen because of its simplicity and widespread use. The RosNet takes an image & as input and outputs a feature vector g(x) E 12122 Below we specify and provide implementa atinntaaiionn details for each of the 10 evaluated methods, while we refer back to Section 2 for more general descriptions.
train models based on ResNet34 (He et al.) 2016)
Conformal Prediction We create a standard direct regression model by feeding the ResNet feature vector g(x) into a network head of two fully-connected layers, outputting a scalar prediction fe(x). The model is trained using the T.2 loss. We then utilize conformal prediction to create prediction intervals (x) according to (2) Instead of splitting the train images into Ii and I2, we use the val images as the calibration set Iz.
Ensemble We train an ensemble (fe....., } of M = 5 direct regression models and compute the ensemble mean p(x) 1 EM, for (x) and ensemble variance (x) = 3 EM, (A(x) fo. (z))² By inserting these into equation prediction intervals Ca (1) of input-dependent length are then constructed.
7

---

Published in Transactions on Machine Learning Research (09/2023)
Gaussian We create Gaussian model po(y]x) = NNy: po(z), oz(z))
by feeding the ResNet feature vector
g(x) into two separate network heads of two fully-connected layers. These output the mean po(x) and variance
(2)), respectively. Prediction intervals (z)r) are then constructed according to 14
Gaussian Ensemble We train an ensemble of M Gaussian models, compute a single mean ((x) and
variance (x) according to ( and plug these into ( to construct prediction intervals C. (x).
Quantile Regression We create quantile regression model by feeding the ResNet feature vector g(x) into two separate network heads. These output the quantiles gg (z), 9g dup (xx, directly forming prediction intervals
Ca(x) according to The model is trained by minimizing the average pinball loss of dg "g"(()) (z) and Og
Pup (x).
Gaussian + Selective GMM We combine the GAUSSIAN method with a selective prediction mechanism. After training a Gaussian model, we run it on each image in train to extract ResNet feature vectors (g( We then utilize scikit-learn (Pedregosa et al., 2011) to fit a GMM (4 components, full covariance) to these train feature vectors. To compute an uncertainty score Ky (z) for a given input x, we extract g(x) and evaluate its likelihood according the fitted GMM, (x) = (G)M() The prediction pa)(a) and corresponding prediction interval (z) of the Gaussian model are then output iff (x) S T. To set the user-specified threshold т, we compute y (z) on all images in val and pick the 95% quantile. This choice of I is motivated by the commonly reported FPR95 OOD detection metric, but could be set using other approaches.
Gaussian + Selective KNN Identical to GAUSSIAN + SELECTIVE GMM, but (x))) = KNN (g(x)).
Specifically, the uncertainty score ky(x) is computed by extracting g(x) and computing the average distance to its k 10 nearest neighbors among the train feature vectors {g(x:)], Following Kuan & Mueller 2022). we utilize the Annoy approximate neighbors library, with cosine similarity as the distance metrie.
Gaussian + Selective Variance Identical to GAUSSIAN + SELECTIVE GMM, but Ky(x) = oz(x) (the variance of the Gaussian model). This is used as a simple baseline for the two previous methods.
Gaussian Ensemble + Sclective GMM We combine GAUSSIAN ENSEMBLE with a selective prediction mechanism. After training an ensemble of M Gaussian models, we run each model on each image in train to extract M sets of ResNet feature vectors. For each model, we then fit a GMM to its set of train feature vectors. Le., we fit M different GMMs. To compute an uncertainty score K (2) for given input x, extract feature vector and evaluate its likelihood according to the corresponding GMM, for each of the M models. Finally, we compute the mean of the GMM likelihoods, ky(x)) M Σ (9.)))
Gaussian Ensemble + Selective Ensemble Variance Identical to GAUSSIAN ENSEMBLE + SELECTIVE 1
1 EM (2, (r). Hence, the variance of the GMM, but xy (z(x)= = M E. (A(z) Me, (z))² where A(z) = M ensemble means is used as the uncertainty score. This constitutes a simple bascline for the previous method.
All models are trained for 75 epochs using the ADAM optimizer (Kingma & Ba, 2014). The same hyperpa- rameters are used for all datasets, and neither the training procedure nor the models are specifically tuned for any particular dataset. All experiments are implemented using PyTror (Paszke et al. 2019), and our complete implementation is made publicly available. All models were trained on individual NVIDIA TITAN Xp GPUs On one such GPU, training 20 models on one dataset look approximately 24 hours. We chose to train all models based on a ResNet34 backbone because it is widely used across various applications, yet simple implement and quite computationally incxpensive. The proposed benchmark and the evaluated uncertainly estimation methods are however entirely independent of this specific choice of DNN backbone architecture. Exploring the use of other more powerful models and evaluating how this affects the reliability of uncertainty estimation methods is an interesting direction which we leave for future work.
5
Related Work
dft-o--dist robustness of DNNs is an active area of research (Hendrycks & Dietterich, 2019 Hendrycks 2021b a Izmailov et al. 2021 Yao et al.) 2022 Schwinn et al.) 2022 Wiles et al.) 2022 Zhang et al.)
et al.
2022). All these previous works do however focus exclusively on classification tasks, Moreover, they consider no uncertainty measures but instead evaluate only in terms of accuracy. While (Zaidi al. 2021 Hendrycks

---

Published in Transactions on Machine Learning Research (09/2023)
Lest Coverage
0.9
0.8
0.7
0.6
0.5
Cells
Cells-
Cells-Gap
ChairAugle
ChairAngle- Tails
ChairAngle-
Interval Length (1)
Gap
Cells
Cails-Tails
Cells-Gap
ChairAngle
ChairAngh
ChairAngh Gap
Tails
Conformal Prediction
Ensemble
Gaussian
Gaussian Ensemble
Quantile Regression
Figure 2: Results for the five common regression uncertainty estimation methods (which output predictions and corresponding 90% prediction intervals for all inputs), on the six synthelic datasets. Top: Results in terms our main metric test coverage. perfectly calibrated method would achieve a test coverage of exactly 90%, as indicated by the solid line. Bottom: Results in terms of average val interval length.
evaluate uncertainty calibration, they also just consider the classification setting. In contrast, evaluation of uncertainty estimation methods is main focus, and we do this specifically for regression.
al. 2022
The main sources of inspiration for our work are (Koh et al. 2021 and (Ovadia al. 2019 While Koh et al. (201)) propose an extensive benchmark with various real-world distribution shifts, it only contains a single regression dataset. Moreover, methods are evaluated solely terms of predictive performance. Ovadia et al. (2019) perform a comprehensive evaluation uncertainty estimation methods under distribution shifts, but only consider classification tasks. Inspired by this, we thus propose our benchmark for evaluating reliability of uncertainty estimation methods under real-world distribution shifts in the regression setting. Most similar to our work is that of Malinin et al. (2021). However, their benchmark contains just two regression datascts (labular weather prediction and complex vehicle motion prediction lask), they only evaluate ensemble-basee uncertainty estimation methods, and these methods are not evaluated in terms of calibration.
6 Results
We evaluate the 10 methods specified in Section 1 on all 12 datasets from Section 3.1 according to the evaluation procedure described in Section 3.2 For each method we train 20 models, randomly select of them for evaluation and report the averaged metrics. For the ensemble methods, we construct an ensemble by randomly selecting M = out the 20 trained models, evaluate the ensemble and then repeat this 5 times in total. To ensure that the results do not depend on our specific choice of a = 001,, we also evaluate methods with two alternative miscoverage rates. While the main paper only contains results for a 0.1, we repeat most of the evaluation for 0.2 and 0.05 in Appendix B observing very similar trends overall.
6.1 Common Uncertainty Estimation Methods
We start by evaluating the first five methods from Section those which output predictions and corresponding 90% prediction intervals for all inputs. The results in terms of our main metric test coverage are presented in the upper part of Figure 2 for the synthetic datasets, and in Figure for the six real-world datasets. In the

---

Published in Transactions on Machine Learning Research (09/2023)
Lest Coverage
0.9
0.8
0.7
0.6
0.5
Asset Wealth
Ventricular
BrainTumoun Pixels
SkinLesion Pixels
Histology NucleiPixels
AerialBuilding
Interval Length (1)
Volume
Pixels
Asset Wealth
Ventricular Volume
Brain'Tumuur Pixels
SkinLesion Pixels
Histology NucleiPixels
AcrialBuilding Pixels
Conformal Prediction
Ensemble
Gaussian
Gaussian Ensemble:
Quantile Regression
Figure 3: Results for the five common regression uncertainty estimation methods (which output predictions and corresponding 90% prediction intervals for all inputs), the six real-world datasets. Top: Results in terms of our main metric test coverage. Bottom: Results in terms of average val interval length.
lower parts of Figure 2 results in terms of average val interval length are presented. The complete results, including our other secondary metric val MAE, are provided in Table A1 Table A12 in the appendix. Please note that, because we utilize a new benchmark consisting of custom datasets, we are not able to directly compare the MAE of our models with that of any previous work from the literature.
In Figure 2 the test coverage results on the first synthetic dataset CELLS are found in the upper-left. As it baseline. We observe
there is no distribution shift between train/val and test for this dataset, that all five methods have almost perfectly calibrated prodiction intervals, i.c. they all obtain test coverage very close to 90%. This is exactly the desired behaviour. On CELLS-TAILS however, which we introduced clear distribution shift, we observe in Figure 2 that the test coverage drops dramatically from the desired 90% for all five mothods. Even the statc-of-the-art uncertainty estimation method GAUSSIAN ENSEMBLE here becomes highly overconfident, as its test coverage drops down to 59%. On CELLS-GAF the test. coverages are slightly closer to 90%, but five methods are still highly overconfident. On the other synthetic dataset CHAIRANGLE, we observe Figure that all five methods have almost perfectly calibrated prediction intervals. However, can observe that the test coverage once again drops dramatically for all methods.
introduce clear distribution shifts on CHAIRANGLE- TAILS and CHAIRANGLE-GAN GAP,
The results on the six real-world datasets are found in Figure 3 In the upper part, we observe that all five methods have quite well-calibrated prediction intervals on ASSETWEALTH and BRANTUMOURPIXELLS even though they all are consistently somewhat. overconfident. (test coverages of 82%-89%). On the four remaining datasets, the methods are in general more significantly overconfident. On VENTrIcuLARVOL.DR observe Lest coverages of 60%-80% for all methods, and on SKINLESION the very best coverage is 82%. On HISTOLOCYNNCLEIIIXCSS A WRRIA.BBILDINGPIIHL the very best coverage is ~ 81%. In fact, not single method actually reaches the desired 90% test coverage on any of these real-world datasets.
most methods only obtain test coverages of 55%-70% and on
In terms of average val interval length, we observe in the lower parts of Figure 2 & that ENSEMBLE consistently produces smaller prediction intervals than CONFORMAL PREDICTION. Morcover, the intervals of GAUSSIAN ENSEMBLE are usually smaller than those of GAUSSIAN. When comparing the interval lengths of QUANTILE REGRESSION and GAUSSIAN, observe clear trend that is consistent across all datasets.
10

---

Published in Transactions on Machine Learning Research (09/2023)
test Coverage
0.9
0.8
0.7
0.6
0.5
0 (Cells)
(Cells-Tails)
Distribution Shift Intensity
tost Coverage
0.9
0.8
0.7
0.6
0.5
0 (ChainAngle)
2
3
4 (ChairAngle-Gapp
Distribution Shift Intensily
Test coverage results for the five common regression uncertainty cstimation methods, on synthetic datasets with increasing degrees of distribution shifts. Top: From CELLS (no distribution shift) CELLS- TAILS (maximum distribution shift). Bottom: From CHAIRANGLE to CHARANGLE-GAP.
Figure 4:
Since the average interval lengths vary lot between different datasets, Figure & only show relative comparisons of the methods. For absolute numerical scales, see Table A1 Table A12
To further study how the test coverage performance is affected by distribution shifts, we also apply the five methods to three additional variants of the CELLS dataset. CELLS has no difference regression target range between train/val and test, whereas for CELLS-TAILS the target range is |50, 150) for train/ /val and [1, [1,200] for test. By creating three variants with intermediate target ranges, we thus obtain sequence of five datasets with increasing degrees of distribution shifts, starting with CELLS (no distribution shift) and ending with 'ELLS-TAILS (maximum distribution shift). The test coverage results on this sequence of datasets are presented in the upper part of Figure 4 We observe that as the degree of distribution shift is increased step-by-step, the test coverage also drops accordingly. The lower part of Figure 4 presents the results of a similar experiment, in which we construct a sequence of five datasets starting with CHAIRANGLE (no distribution shift) and ending with CHAIRANGLF-GE GAP (maximum distribution shift).. Also in this case, we observe that the test coverage drops step-by-step along with the increased degree of distribution shift.
A study of the relative performance of the five methods the real-world datasets, in terms of all three metrics (test coverage, average val interval length, val MAE), is finally presented in Figure A1 Figure A3 in the appendix. One can clearly observe that ENSEMBLE and GAUSSIAN ENSEMBLE achieve the best performance, thus indicating that ensembling multiple models indeed helps to improve the performance.
6.2 Selective Prediction Methods
Next, evaluate the methods with added selective prediction mechanism. We start with the three methods based on GAUSSIAN. The results in terms of test coverage and test prediction rate are available in Figure 5 for the synthetic datasets, and in Figure 0 for the six real-world datasets. While a complete evaluation of these methods also should include the average val interval length, we note that the selective prodiction mochanism does not modify the intervals of the underlying GAUSSIAN method (which already has been evaluated in terms of interval length in Section 6.1) Here, we therefore focus on the test coverage and test prediction rate. Complete numerical results are provided in Table AT Table A12 in the appendix.
11

---

Published in Transactions on Machine Learning Research (09/2023)
Lest. Coverage
0.9
0.8
0.7
0.6
0.5
Cella
Tails
Cells-Gap
ChairAugle
ChairAngle- Tails
ChairAngle-
Gap
Prediction Rale (1)
0.8
0.6
0.4
0.2
Cells
Cails-Tails
Cells-Gap
ChairAngle
ChairAngle- Tails
Ahale-ngle-
Gap
Gaussian
Gaussian Selective GMM
Gaussian + Selective kNN
Gaussian
Selective Variance
Figure 5: Results for the three selective prediction methods based on GAUSSIAN, the six synthelic datasets. Top: Results in terms of our main metric test coverage. Bottom: Results in terms of test prediction rate (the proportion of test inputs for which a prediction actually is output).
In the upper part of Figure 5 observe that selective prediction based on feature-space density significantly improves the test coverage of GAUSSIAN on the synthetic datasets. While GAUSSIAN has well-calibrated prediction intervals only on CELLS and CHAIRANGLE, which are baseline datasets without any distribution shift, GAUSSIAN + SELECTIVE GMM is almost perfectly calibrated across all six datasets. On CELLS-TAILS, for example, improves the test coverage from = 54% up 89%. GAUSSIAN + SELECTIVE KNN also significantly improves the test coverages, but not quite the same extent. In the lower part of Figure we observe that when GAUSSIAN + SELECTIVE GMM significantly improves the test coverage, there also clear drop in its test prediction rate. For example, the prediction rale drops from & 0.95 on CELLS down to By rejecting nearly 50% of all inputs as OOD in this case, GAUSSIAN + SELECTIVE GMM can thus remain well-calibrated on the subset of test it actually outputs predictions for. In Figure we also observe that GAUSSIAN + SELECTIVE VARIANCE only marginally improves the test coverage.
se 0.54 on CELLS-TAIE
While GAUSSIAN + SELECTIVE GMM signíficantly improves the test coverage of GAUSSIAN and has well- calibrated prediction intervals across the synthetic datasets, we observe in Figure t that this not true for the six real-world datasets. GAUSSIAN + SELECTIVE GMM does consistently improve the test coverage, but only marginally, and it still sulfers from significant overconfidence in many cases. On VENTRICUUAARVOOUMM for example, the test prediction rate of GAUSSIAN + SELECTIVE GMM is as low as ss 0.71, but the test coverage only improves from & 73% to 2 75% compared to GAUSSIAN.
For the two methods based on GAUSSIAN ENSEMBLE, the results are presented Figure & A5 in the appendix. Overall, we obscrve very similar trends. GAUSSIAN ENSEMBLE + SELECTIVE GMM significantly improves the test coverage of GAUSSIAN ENSEMBLE and is almost perfectly calibrated across the synthetic datasets. However, when it comes to the real-world datasets, it often remains significantly overconfident.
Finally, Figure A6 presents relative comparison of the five selective prediction methods across all 12 datascts, in terms of average (est. coverage error (absolute difference between empirical and expected interval coverage) and average test prediction rate. We observe that GAUSSIAN ENSEMBLE + SELECTIVE GMM achieves the best test coverage error, but also has the lowest test prediction rate. In fact, each improvement in terms of test coverage error also corresponds decrease in test prediction rate for these five methods, meaning that there seems to be inherent trade-off between the two metrics.
12

---

Published in Transactions on Machine Learning Research (09/2023)
Lest Coverage
0.9
0.8
0.7
0.6
0.5
Asset Wealth
Ventricular Volume
Brain Tumour Pixels
SkinLesion Pixels
Histology NucleiPixels
AerialBuilding
Pixels
Prediction Rate (f)
0.8
0.6
0.4
0.2
Asset Wealth
Ventricular Volume
Brain'Tmmurr Pixels
SkinLesion Pixels
Histology NucleiPixels
AcrialBuilding Pixels
Ganssian
Ganssian
Selective GMM
Gaussian + Selective KNN
Gaussian Selective Variance
Figure 6: Results for the three selective prediction methods based on GAUSSIAN, on the six real-world datasets.
Top: Results in terms of test coverage. Bottom: Results in terms of test prediction rate.
7
Discussion
Let us now analyze the results from Section e in more detail, and discuss what consider the most important findings and insights. First of all, can observe that among the 10 considered methods, not single close to producing perfectly calibrated prediction intervals across all 12 datasets. We thus conclude that proposed benchmark indeed is challenging and interesting. Morcover, the results in Figure 2 & demonstrate that while common uncertainty estimation methods are well calibrated when there is no distribution shift (CELLS and NGI.E) they can all break down and become highly overconfident in many realistic scenarios. This highlights the importance of employing sufficiently realistic and thus challenging benchmarks when evaluating uncertainty estimation methods. Otherwise, we might be lead to believe that methods will be more reliable during practical deployment. than they actually are.
Coverage Guarantees Might Instill False Sense of Security We also want to emphasize that CONFORMAL PREDICTION and QUANTILE REGRESSION] have theoretical coverage guarantees, but still are observed to become highly overconfident for many datasets in Figure 2 & 3 Since the guarantees depend on the assumption that all data points are exchangcable (true for i.i.d. data, for instance), which generally does not hold under distribution shifts, these results should actually not be surprising. The results are however a good reminder that we always need to be aware of the underlying assumptions, and whether or not they are likely to hold in common practical applications. Otherwise, such theoretical guarantees might just instill false sense of security, making trust methods more than actually should.
Clear Performance Differences between Synthetic and Real-Worlle Datasets We find it interesting that selective prediction based on feature-space density, in particular GAUSSIAN + SELECTIVE GMM, works almost perfectly in terms of test coverage across the synthetic datasets (Figure 5 but fails to give significant improvements on the real-world datasets (Figure 0 The results on VENTRICULARVOII interesting, as the prediction rate drops quite a lot without significantly improving the test coverage. This means that while relatively large proportion of the test inputs are deemed OOD and thus rejected by the method, the test coverage is barely improved. On the synthetic datasets, there is corresponding improvement in test coverage whenever the prediction rate drops significantly (Figure 5 It is not clear
are particularly
2iinee all prediction intervals calibrated on val, we using Conformakzeed Quantile Regression
Romano
2019
13

---

Published in Transactions on Machine Learning Research (09/2023)
why such an obvious performance difference between synthetic and real-world datasets is observed. One possible explanation is that real-world data requires better models for p(ær), i.e. that the relatively simple approaches based on feature-space density not are sufficient. Properly explaining this performance difference is an important problem, but we will here leave this as an interesting direction for future work.
OOD Uncertainty Scores Perform Well, but Not Well Enough Comparing the selective prediction methods, we observe that GAUSSIAN + SELECTIVE GMM consistently outperforms GAUSSIAN + SELECTIVE VARIANCE (Figure & 6 and that GAUSSIAN ENSEMBLE + SELECTIVE GMM outperforms GAUSSIAN ENSEMBLE + SELECTIVE ENSEMBLE VARIANCE in most cases (Figure A4 & A5). Relative to common uncertainty estimation baselines, methods based on featuro-space density thus achieve very strong performance. This is in line with the state-of-the-art OOD detection performance that has been demonstrated recently. In our results, we can however clearly observe that while feature-space methods perform well relative to common baselines, the resulting selective prediction methods are still overconfident performance is still far from perfect. Using our benchmark, we are thus able to not only compare the relative performance of different OOD uncertainty scores, but also evaluate their performance in an absolute sense.
many cases the absolute
Performance Differences among Real-Worlee Datasets are Mostly Logical When we compare the performance on the different real-world datasets in Figure 3 all methods are relatively well-calibrated on CAMANTTMMURPPPHLL and ASSET WEALTH. For PRAAINTUMOURPP created by randomly splitting the original set of MRI scans. The distribution shift between train/ /al and test is thus also fairly limited. For ASSETWEALTH (satellite images from different African countries), the shift is likely quite limited at least compared to BUIDDNGGEES (satellite images from two different continents). Finally, the results for where CONFORMAL PREDICTION clearly obtains the best test coverage. It is not clear why the methods which output prediction intervals of depen-de length struggle on this particular dataset.
the train, val and test splits were
are interesting, this is the only dataset
Finally, we should emphasize that while test coverage is our main metric, this by itself is not sufficient for a method to be said to "perform well" in a general sense. For example, a perfectly calibrated method with a low test. prediction rate might not be particularly useful in practice. While even very low lest prediction rates likely would be tolerated in many medical applications and other safety-critical domains (as long as the method stays perfectly calibrated), one can imagine more low-risk scttings where this calibration versus prediction rate trade-off is a lot less clear. Since not a single one of the evaluated methods was close to being perfectly calibrated across all 12 datasets, we did however mainly focus on analyzing the test coverage in this paper. If multiple methods had performed well in terms of test coverage, more detailed analysis and discussion of the secondary metrics performance would have been necessary.
The main actionable takeaways from our work can be summarized as follows:
All methods are well calibrated on bascline datasets with no distribution shift, but become highly overconfident in many realistic scenarios. Uncertainty estimation methods must therefore be evaluated using sufficiently challenging benchmarks. Otherwise, one might be lead to believe that methods will be more reliable during real-world deployment than they actually are.
Conformal prediction methods have commonly promoted theoretical coverage guarantees, but these depend on assumption that is unlikely to hold in many practical applications. Consequently, also these methods can become highly overconfident in realistic scenarios. If the underlying assumptions are not examined critically by practitioners, such theoretical guarantees risk instilling a false sense of security making these models even less suitable for safety-critical deployment.
The clear performance difference between synthetic and real-world datasets observed for selective prediction methods based on feature-space density is very interesting direction for future work. If the reasons for this performance gap can be understood, an uncertainty estimation method that stays well calibrated across all datasets could potentially be developed.
Selective prediction methods basod on feature-space density perform well relative to other methods (as expected based on their state-of-the-ar OOD detection performance), but are also overconfident in many cases. Only comparing the relative performance of different methods is therefore not sufficient.
14

---

Published in Transactions on Machine Learning Research (09/2023)
To track if actual progress is being made towards the ultimate goal of truly reliable uncertainty estimation methods, benchmarks must also evaluate method performance in an absolute sense.
8
Conclusion
We proposed an extensive benchmark for testing the reliability of regression uncertainty estimation methods under real-world distribution shifts. The benchmark consists of 8 publicly available image- based regression datasets with different. types of challenging distribution shifts. We employed our benchmark to evaluate many of the most common uncertainty estimation methods, well two stato-of-the-art uncertainty scores from OOD detection. We found that while all methods are well calibrated when there is no distribution shift, they become highly overconfident on many of the benchmark datasets. Methods based on the OOD uncertainty scores performed well relative to other methods, but the absolute performance is still far from perfect. This uncovers important limitations of current regression uncertainty estimation methods. Our work thus serves as challenge to the research community, to develop methods which actually produce calibrated prediction intervals across all benchmark datasets. To that end, future directions include exploring the use of more sophisticated models for p(x) within selective prediction hopefully closing the performance gap between synthetic and real-world datasets and employing alternative DNN backbone architectures. We hope that our benchmark will spur more work on how to develop truly reliable regression uncertainty estimation methods.
Acknowledgements
This research was supported by the Swedish Research Council via the projects NewLEADS New Directions in Learning Dynamical Systems (contract number: 621-2016-06079) and Deep Probabilistic Regression New Models and Learning Algorithms (contract number: 2021-04301), and by the Kjell & Märta Beijer Foundation. We also thank Ludvig Hult and Dave Zachariah for providing helpful feedback on early manuscript.
References
Michela Antonelli, Annika Reinke, Spyridon Bakas, Keyvan Farahani, Annette Kopp-Schneider, Bennett. A Landman, Geert Litjens, Bjoern Menze, Olaf Ronneberger, Ronald M Summers, et al. The medical segmentation decathlon. Nature Communications, 13(1):1-13, 2022.
Spyridon Bakas, Hamed Akbari, Aristeidis Sotiras, Michel Bilello, Martin Rozycki, Justin S Kirby, John B Freymann, Keyvan Farahani, and Christos Davatzikos. Advancing the cancer genome atlas glioma mri collections with expert segmentation labels and radiomic features. Scientific data, 4(1):1-13, 2017.
Spyridon Bakas, Mauricio Reyes, Andras Jakab, Stefan Bauer, Markus Rempfler, Alessandro Crimi, Rus- sell Takeshi Shinohara, Christoph Berger, Sung Min Ha, Martin Rozycki, et al. Identifying the best machine learning algorithms for brain tumor segmentation, progression assessment, and overall survival prediction in the brats challenge. arXiv preprint arXiv:1811.0203 2018.
Charles Blundell, Julien Cornebise, Koray Kavukeuoglu, and Daan Wierstra. Weight uncertainty in neural network. In International Conference on Machine Learning (ICML), pp. 1613-1622, 2015.
Kurtland Chua, Roberto Calandra, Rowan McAllister, and Sergey Levine. Deep reinforcement learning in a handful of trials using probabilistic dynamics models. In Advances in Neural Information Processing Systems (NewrIPS), pp. 4759-4770 2018.
Youngseog Chung, Willie Neiswanger, Ian Char, and Jeff Schneider. Beyond pinball loss: Quantile methods for calibrated uncertainty quantification. Advances in Neural Information Processing Systems (NewrIPS), 34:10971-10994 2021.
James H Cole, Rudra PK Poudel, Dimosthenis Tsagkrasoulis, Matthan WA Caan, Claire Steves, Tim Spector, and Giovanni Montana. Predicting brain age with deep learning from raw imaging data results in reliable and heritable biomarker. Neurolmage, 163:115-19 124, 2017.
15

---

Published in Transactions on Machine Learning Research (09/2023)
Peng Cui, Wenbo Hu, and Jun Zhu. Calibrated reliable regression using maximum mean discrepancy. Advances in Neural Information Processing Systems (NewrPPS), 33:17164-17175, 2020.
Xin Ding, Yongwei Wang, Zuheng Xu, William J. Welch, and Z. Jane Wang. Continuous conditional generative adversarial networks for image generation: Novel losses and label input mechanisms. arXiv preprint trXiv:2011.07466, 2020.
Xin Ding, Yongwei Wang, Zuheng Xu, William J Welch, and Z. Jane Wang. CeGAN: Continuous condi- tional generalive adversarial networks for image generation. In International Conference on Learning Representations (ICLR), 2021.
Samuel G Finlayson, Adarsh Subbaswamy, Karandeep Singh, John Bowers, Annabel Kupke, Jonathan Zittrain, Isaac $ Kohane, and Suchi Saria. The clinician and dataset shift in artificial intelligence. New England Journal of Medicine, 385(3):283-285) 2021.
Adam Fisch, Tommi S. Jaakkola, and Regina Barzilay. Calibrated selective classification. Transactions on Machine Learning Research (TMLR), 2022.
Yarin Gal. Uncerlainly in Deep Learning. PhD thesis, University of Cambridge, 2016.
Yarin Gal and Zoubin Ghahramani. Dropout as a Bayesian approximation: Representing model uncertainty deep learning. In International Conference on Machine Learning (ICMT., pp. 1050-1059, 2016.
Yonatan Geifman and Ran EI- El-Ya.iv. Selective classification for deep neural networks. Advances in Newral Information Processing Systems (NeurIPS), 30, 2017.
Tilmann Gneiting, Fadoua Balabdaoui, and Adrian E Raftery. Probabilistic forecasts, calibration and sharpness. Journal of the Royal Statistical Society: Series (Statistical Methodology). 69(2) 2007.
Simon Graham, Quoe Dang Vu, Shan E Ahmed Raza, Ayesha Azam, Yee Wah Tsang, Jin Tae Kwak, and Nasir Rajpoot. HoVer-Net: Simultaneous segmentation and classification of nuclei in multi-tissue histology images. Medical Image Analysis, 58:101563, 2019.
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. On calibration of modern neural networks. In Proceedings of the 34th International Conference on Machine Learning (ICML), pp. 1321 1330, 2017.
Fredrik K Gustafsson, Martin Danelljan, Goutam Bhat, and Thomas B Schön. Energy-based models for deep probabilistic regression. In Proceedings of the European Conference on Compuler Vision (ECCV), pp. 325 343, 2020a.
Fredrik K Gustafsson, Martin Danelljan, and Thomas B Schön. Evaluating scalable Bayesian deep learning methods for robust computer vision. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, 2020b.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770-778, 2016.
Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common corruptions and perturbations. In International Conference on Learning Representations (ICLR), 2019.
Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, et al. The many faces of robustness: A critical analysis of out-of-distribution gemeralization In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 8340-8349, 2021a.
Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardt, and Dawn Song. Natural adversarial examples. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pottern Recognation (CVPR), pp. 15262-16 11221, 2021b.
16

---

Published in Transactions on Machine Learning Research (09/2023)
Dan Hendryeks, Andy Zou, Mantas Mazeika, Leonard Tang, Bo Li, Dawn Song, and Jacob Steinhardt. Pixmix: Dreamlike pictures comprehensively improve safety measures. In Proceedings of the JEEF/CVE
Conference on Computer Vision and Pattern Recognition (CVPR), pp. 16783 16792, 2022.
Radu Herbei and Marten H Wegkamp. Classification with reject option. The Canadian Journal of Statistics/ Taa Revue Canadienne de Statistique, pp. 709 721, 2006.
Eddy Tlg, Ozgun Cicck, Silvio Galesso, Aaron Klein, Osama Makansi, Frank Hutter, and Thomas Bro. Uncertainty estimates and multi-hypotheess networks for optical flow. In Proceedings of the Europeon Conference Computer Vision (ECCV), pp. 652-667, 2018.
Pavel Izmailov, Patrick Nicholson, Sanae Lotfi, and Andrew Wilson. Dangers of bayesian model averaging under covariate shift. Advances in Neural Information Processing Systems (NeurIPS), 34:3309 3322, 2021.
Vilde Jensen, Filippo Maria Bianchi, and Stian Norman Anfinsen. Ensemble conformalized quantile regression for probabilistic time series forecasting. arXiv preprint arXiv:2202.08756, 2022.
Wenming Jiang, Ying Zhao, and Zehan Wang- Risk-controlled selective prediction for regression deep neural network models. In 2020 International Joint Conference on Newral Networks (IJCNN), pp. 1-8. IEEE, 2020.
Benedikt Atli Jónsson, Gyda Bjornsdottir, TE Thorgeirsson, Lotta María Ellingsen, G Bragi Walters, DF Gudbjartsson, Hreinn Stefansson, Kari Stefansson, and MO Ulfarsson. Brain age prediction using deep learning uncovers associated sequence variants. Nature Communications, 10(1):5409, 2019.
Alex Kendall and Yarin Gal. What uncertainties do need in Bayesian deep learning for computer vision? In Advances in Neural Information Processing Systems (NewrIPS), pp. 5574-5584, 2017.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:14122 6980, 2014.
Roger Koenker and Gilbert Bassett Jr. Regression quantiles. Econometrica: journal of the Econometric Society, pp. 33-50, 1978.
Pang Wei Koh, Shiori Sagawa, Henrik Marklund, Sang Michael Xie, Marvin Zhang, Akshay Balsubramani, Woihua Hu, Michihiro Yasunaga, Richard Lanas Phillips, Irena Gao, et al. Wilds: benchmark of in-the-wild distribution shifts. In International Conference on Machine Leurning (ICML), pp. 5637 5664. PMLR, 2021.
Johnson Kuan and Jonas Mueller. Back to the basics: Revisiting out-of-distribatiio detection baselines. arXiv preprint vcXic:2207.03066 2022.
Volodymyr Kuleshov, Nathan Fenner, and Stefano Ermon. Accurate uncertainties for deep learning using calibrated regression. In International Conference on Machine Learning (ICML), pp. 2796 2804. PMLR, 2018.
Neeraj Kumar, Ruchika Verma, Sanuj Sharma, Surabhi Bhargava, Abhishek Vahadane, and Amit Sethi. A dataset and a technique for generalized muclear segmentation for computational pathology. IEEE Transactions on Medical Imaging, 36(7):1550-1555 2017.
Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell Simple and scalable predictive uncertainty estimation using deep ensembles. In Advances in Neural Information Processing Systems (NewrIPS), pp. 6402-6413, 2017.
Taro Langner, Robin Strand, Håkan Ahlström, and Joel Kullberg. Large-scale biometry with interpretable neural network regression uk biobank body mri. Scientific Reports, 10(1):17752, 2020.
Taro Langner, Fredrik K Gustafsson, Benny Avelin, Robin Strand, Häkan Ahlström, and Joel Kullberg. Uncertainty-aware body composition analysis with deep regression ensembles on uk biobank mri. Comput- erized Medical Imaging and Graphics, 93:101994, 2021.
17

---

Published in Transactions on Machine Learning Research (09/2023)
Stéphane Lathuilière, Pablo Mesejo, Xavier Mlameda-Pinedde and Radu Horaud. A comprehensive analysis of deep regression. IEEF Transactions on Pollern Analysis and Machine Intelligence (TPAMT), 2019.
Hei Law and Jia Deng. Cornernet: Detecting objects as paired keypoints. In Proceedings of the Europeon Conference on Computer Vision (ECCV), pp. 734-750, 2018.
Dan Levi, Liran Gispan, Niv Giladi, and Ethan Fetaya. Evaluating and calibrating uncertainty prediction regression tasks. Sensors, 22(15),5540, 2022.
Yi-An Ma, Tianqi Chen, and Emily Fox. A complete recipe for stochastic gradient MCMC. In Advonces in Neural Information Processing Systems (NeurIPS), pp. 2917-2925, 2015.
Emmanuel Maggiori, Yuliya Tarabalka, Guillaume Charpiat, and Pierre Alliez. Can semantic labeling methods generalize to any city? the Inria acrial image labeling benchmark. In 2017 IEEE International Geoscience and Remole Sensing Symposium (IGARSS), pp. 3226 3229. IEEE, 2017.
Andrey Malinin, Neil Band, Yarin Gal, Mark Gales, Alexander Ganshin, German Chesnokov, Alexey Noskov, Andrey Ploskonosov, Liudmila Prokhorenkova, Ivan Provilkov, Vatsal Raina, Vyas Raina, Denis Roginskiy, Mariya Shmatova, Panagiotis Tigas, and Boris Yangel. Shifts: A dataset of real distributional shift across multiple large-scale tasks. In Thärty-fifth Conference on Neural Information Processing Systems (NeurIPS), Dalasels and Benchmarks Truck, 2021.
Bjoern H Menze, Andras Jakab, Stefan Bauer, Jayashree Selpathy-Crammee Keyvan Farahani, Justin Kirby, Yuliya Burren, Nicole Porz, Johannes Slotboom, Roland Wiest, et al. The multimodal brain tumor image segmentation benchmark (BRATS). IEEE Transactions on Medical Imaging, 34(10):1993 2024, 2014.
Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. Spectral normalization for generative adversarial networks. In International Conference on Learning Representations (ICLR), 2018.
Jishnu Mukhoti, Andreas Kirsch, Joost van Amersfoort, Philip HS Torr, and Yarin Gal. Deep deterministic uncertainty: A simple baseline. ar preprint, 2021a.
Jishnu Mukhoti, Andreas Kirsch, Joost van Amersfoort, Philip HS Torr, and Yarin Gal. Deterministic neural networks with appropriate inductive biases capture epistemic and aleatoric uncertainty. arXiv preprint arXiv:2102.11582, 2021b.
Peter Naylor, Marick Laé, Fabien Reyal, and Thomas Walter. Segmentation of nuclei in histopathology images by deep regression of the distance map. IEEE Transactions on Medical Imaging, 38(2):448 4494-499 2018.
Radford M Neal. Bayesian learning for neural networks. PhD thesis, University of Toronto, 1995.
David Ouyang, Bryan He, Amirata Ghorbani, Neal Yuan, Joseph Ebinger, Curtis P Langlotz, Paul A Heidenreich, Robert A Harrington, David H Liang, Euan A Ashley, et al. Video-based Al for beat-to-beat assessment of cardiac function. Nature, 580(7802):253) 256, 2020.
Yaniv Ovadia, Emily Fertig, Jie Ren, Zachary Nado, D. Sculley, Sebastian Nowozin, Joshua Dillon, Balaji Lalshminarayanan, and Jasper Snoek. Can you trust your model's uncertaint?? Evaluating predictive uncertainty under dataset shift. In Advances in Neural Information Processing Systems (NeurIPS), volume 32, 2019.
Harris Papadopoulos. Inductive conformal prediction: Theory and application to neural networks. In Tools in artificial intelligence. Citescer, 2008.
Harris Papadopoulos, Kostas Proedrou, Volodya Vovk, and Alex Gammerman. Inductive confidence machines for regression. In European Conference Machine Learning, pp. 345-356. Springer, 2002.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. PyTorch: An imperative style, high-performance deep learning library. In Advances in Neural Information Processing Systems (NeurIPS), pp. 8024-8035, 2019.
18

---

Published in Transactions on Machine Learning Research (09/2023)
Fabian Pedregosa, Gaäl Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, Jake Vanderplas, Alexandre Passos, David Cournapeau, Matthieu Brucher, Matthieu Perrot, and Edouard Duchesnay. Scikit-learn: Machine learning in python. Journal of Machine Learning Research (JMLR), 12(85))2825-283)) 2011.
Ethan Pickering and Themistoklis P Sapsis. Structure and distribution metric for quantifying the quality of uncertainty: Assessing gaussian processes, deep neural nets, and deep neural operators for regression. arXiv preprint arXi::203.004 04515, 2022.
Geoff Pleiss, Amauri Souza, Joseph Kim, Boyi Li, and Kilian Q Weinberger. Neural network out-of-distriibution detection for regression tasks, 2020. URL
Ryan Poplin, Avinash V Varadarajan, Katy Blumer, Yun Liu, Michael V McConnell, Greg S Corrado, Lily Peng, and Dale R Webster. Prediction of cardiovascular risk factors from retinal fundus photographs via deep learning. Nature Biomedical Engineering. 2(3):158-164 2018.
Joaquin Quionero-Candedd Masashi Sugiyama, Anton Schwaighofer, and Neil D Lawrence. Dataset shift in machine learning, 2009.
Yaniv Romano, Evan Patterson, and Emmanuel Candes. Conformalized quantile regression. Advances in Neural Information Processing Systems (NeurIPS), 32, 2019.
Rasmus Rothe, Radu Timofte, and Luc Van Gool. Deep expectation of real and apparent age from single image without facial landmarks. International Journal of Computer Vision, 126(2-4):144 157, 2016.
Mohammadreza Salehi, Hossein Mirzaei, Dan Hendrycks, Yixuan Li, Mohammad Hossein Rohban, and Mohammad Sabokrou. A unified survey anomaly, novelty, open-set, and out of-distribution detection: Solutions and future challenges. Transactions on Machine Learning Research (TMLR), 2022.
Leo Schwinn, Leon Bungert, An Nguyen, René Raab, Falk Pulsmeyer, Doina Precup, Björn Eskofier, and Dario Zanca. Improving robustness against real-world and worst-case distribution shifts through decision region quantification. In International Conference on Machine Learning (ICML), pp. 19434 19449. PMLR, 2022.
Joan Serrå, David Álvarez, Vicenç Gómez, Olga Slizovskaia, José F. Núñez, and Jordi Luque. Input complexity and distriiiistiinttonnn detection with likelihood- based generative models. In International Conference on Learning Representations (ICLR), 2020.
Shaoshuai Shi, Xiaogang Wang, and Hongsheng Li. PointRCNN: 3d object proposal generation and detection from point cloud. In Proceedings of the IEEE Conference on Compuler Vision and Pallern Recognition (CVPR), pp. 770-779, 2019.
Wen Shi, Guohui Yan, Yamin Li, Haotian Li, Tingting Liu, Cong Sun, Guangbin Wang, Yi Zhang, Yu Zou, and Dan Wu. Fetal brain age estimation and anomaly detection using attention-based deep ensembles with uncertainty. Newrolmage, 223:117316, 2020.
Amber L. Simpson, Michela Antonelli, Spyridon Bakas, Michel Bilello, Keyvan Farahani, Bram Van Ginneken, Annette Kopp-Schneider, Bennett A Landman, Geert Litjens, Bjoern Menze, et al. A large annotated medical image dataset. for the development. and evaluation of segmentation algorithms. arxiv preprint 2019.
Ingo Steinwart and Andreas Christmann. Estimating conditional quantiles with the help of the pinball loss. Bernoulli, 17(1):211-22) 2011.
Yiyou Sun, Yifci Ming, Xiaojin Zhu, and Yixuan Li. Out-of-distribation detection with deep nearest neighbors. In International Conference on Machine Learning (ICML), 2022.
Eric J Topol. perooacceoe medicine: the convergence of human and artificial intelligence. Nature Medicine, 25(1) (1):444444)) 56, 2019.
19

---

Published in Transactions on Machine Learning Research (09/2023)
Philipp Tschandl, Cliff Rosendahl, and Harald Kittler. The HAM10000 dataset, large collection of multi-source dermatoscopie images of common pigmented skin lesions. Scientific dala, 5(1):1-9, 2018.
Florian Wenzel, Andrea Dittadi, Peter Gehler, Carl-Johann SibrneGabriel, Max Horn, Dominik Zietlow, David Kernert, Chris Russell, Thomas Brox, Bernt Schiele, et al. Assaying out-of-distribation generalization in transfer learning. Advances in Neural Information Processing Systems (NewrIPS), 35:7181-7198, 2022.
Olivia Wiles, Sven Gowal, Florian Stimberg, Sylvestre-Alvise Rebuffi, Ira Ktena, Krishnamurthy Dj Dvijotham, and Ali Taylan Cemgil. fine-grained analysis on distribution shift. In International Conference on Learning Represenlations (ICLR), 2022.
Bin Xiao, Haiping Wu, and Yichen Wei. Simple baselines for human pose estimation and tracking. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 466-481, 2018.
Wufeng Xue, Ali Islam, Mousumi Bhaduri, and Shuo Li. Direct multitype cardiac indices estimation via joint representation and regression learning. IEEE Transactions on Medical Imaging, 26(10):2057-200 2017.
Huaxiu Yao, Yu Wang, Sai Li, Linjum Zhang, Weixin Liang, James Zou, and Chelsea Finn. Improving out-of-distribution robustness via selective augmentation. In International Conference Machine Learning (ICML), pp. 25407 25437. PMLR, 2022.
Christopher Yeh, Authony Perez, Anne Driscoll, George Azzari, Zhongyi Tang, David Lobell, Stefano Ermon, and Marshall Burke. Using publicly available satellite imagery and deep learning to understand economic well-being in Africa. Nature communications, (1(1):1-1,, 2020.
Sheheryar Zaidi, Arber Zela, Thomas Elsken, Chris Holmes, Frank Hutter, and Yee Teh. Neural ensemble search for uncertainty estimation and dataset shift. Advances in Neural Information Processing Systems (NeurIPS), 34:7898-7911, 2021.
Marvin Mengxin Zhang, Sergey Levine, and Chelsea Finn. MEMO: Test time robustness via adaptation and augmentation. In Advances in Neural Information Processing Systems (NearIPS), 2022.
Zheng Zhu, Qiang Wang, Bo Li, Wei Wu, Junjie Yan, and Weiming Hu. Distractor-aware siamese networks for visual object tracking. In Proceedings of the European Conference on Computer Vision (ECCV), pp- 101 117, 2018
20

---

Published in Transactions on Machine Learning Research (09/2023)
A
Dataset Details
More detailed descriptions of the 12 datasets from Section 3.1 are provided below.
Cells Given a synthetic fluorescence microscopy image x, the task is to predict the number of cells y in the image. We utilize the Cell-200 dalaset from Ding et al. (2021 2020), consisting of 200 000 grayscale images of size 64 x 64. We randomly draw 000 train images, 2 200 val images and 10 000 test images. Thus, there is no distribution shift between train/ /val and test. We therefore use this as a basoline datasct.
Cells-Tails We create a variant of CELLS with a clear distribution shift between train/ val and test. For the 10000 train images and 2 200 val images, the regression targets y are limited to the range J50, 150). For the 10 test images, the targets instcad lie in the full original range [1.).
Cells-Gap Another variant of CELLS with a clear distribution shift between train/val and test. For the 000 train images and 2 000 val images, the regression targets y are limited to [1, 50[0]150, 200]. For the 000 test images, the targets instead lie in the full original range [0..
AhairAngle Given a synthetic image x of a rendered chair model, the task to predict the yaw angle y of the chair. We utilize the RC-49 datasct from Ding et al. (2021 2020), which contains 64 x 64 RGB images of different chair models rendered at 899 yaw angles ranging from 0.1% to 89.9%, with step size 0.1% We randomly split their training set and obtaín 17 640 train images and 410 val images. By sub-sampling their test set we also get 11 225 test images. There is no obvious distribution shift between train/val /val and test, and we therefore use this as a second baseline dataset.
ChairAngles Tails We create a variant of CHAIRANGLE with a clear distribution shift between train/ vaal and test. For train and val, we limit the regression targets y to the range 115, 75|. For test, the targets instead lie in the full original range J0, 90). We obtain 11 760 train images, 2 2940 val images and 11 225 test images.
Chair Angle- Gap Another variant of CHAIRANGLE with clear distribution shift between train/val and test. For the 11 760 train images and 2940 val images, the regression targets y are limited to 30,30]U]60, 90]. For the 11 225 test images, the targets instead lie in the full original range Jo, 90
Asset Wealth Given a satellite image x (8 image channels), the task is to predict the asset wealth index y of the region. We utilize the Poverty Wilds dataset (Koh et al. 2021), which is a variant of the dataset collected by Yeh et al. (2020). We use the training, validation-ID and test- OOD subsets of the data, giving us 9 797 train images, 1 000 val images and 3 366 test images. We resize the images from size 224 x 224 to 64 x 64. Train/val and test contain satellite images from disjoint sets of African countries, creating distribution shift.
VentricularVolume Given an echocardiogram image I of a human heart, the task is to predict the volume y of the left ventricle. We utilize the EchoNet-Dynamic dataset by Ouyang et al. (2020), which contains 10 030 cchocardiogram vidcos. Each video captures a complete cardiac cyele and is labeled with measurements of the left ventricular volume at. two separate time points, represent ing end-systole (at the end of contraction smaller volume) and end-diastole (just before contraction larger volume). For each video, we extract just one of these volume measurements along with the corresponding video frame. To create a clear distribution shift between train/val and test, we select the end systolic volume (smaller volume) for train and val, but the end diastolic volume (larger volume) for test. We utilize the provided dataset splits, giving us 7660 train images, 1 288 val images and 1 277 test images. We resize the images from size 112 x 112 to 64 x 64.
Given an image slice x of a brain MRI scan, the task is to predict the number of pixels y in the image which are labeled as brain tumour. We utilize the brain tumour dataset of the medical segmentation decathlon (Simpson et al. 2019 Antonelli al., 2022), which is subset of the data used in the 2016 and 2017 BraTS challenges (Bakas et al. 2018 2017 Menze et al.) 2014). The dataset contains 484 brain MRI scans with corresponding tumour segmentation masks. We split these scans 0/6/20%/20% into train, val and test sets. The scans are 3D volumes of size 240 x 240 x 155. We convert cach scan into 155 image slices of size 240 x 240, and create a regression larget for each image by counting the number of labeled brain tumour pixels. We then also remove all images which contain no tumour pixels. The original image slices have 4 channels (FLAIR, Tiw, Tigd, T2w), but we only use the first three and convert the slices into standard RGB images. This gives us 20 614 train images, 116 val images and 6 252 test images. We also resize the images from size 240 x 240 to 64 x 64.
21

---

Published in Transactions on Machine Learning Research (09/2023)
SkinLesionPixels Given a dermatoscopic image x of pigmented skin lesion, the task is to predict the number of pixels y in the image which are labeled as lesion. We utilize the HAM10000 dataset by [sccand]] et al. (2018), ; which contains 10015 dermatoscopic images with corresponding skin lesion segmentation masks. HAM10000 consists of four different sub-datasets, three of which (ViDIR Legacy, VIDIR Current and ViDIR MoleMax) were collected in Austria, while the fourth sub-dataset (Rosendahll) was collected in Australia. To create a clear distribution shift between train/val and test, we use the Australian sub-datasct (Rosendahl) as our test set. After randomly splitting the remaining images 85%/15% into train and val sets, we obtain 6592 train images, 1 164 val images and 259 test images. We then create regression target for each image by counting the number of labeled skin lesion pixels. We also resize the images from size 450 x 600 to 64 x 64.
Histology NucleiPixels Given an H&E stained histology image 2, the task is to predict the number of pixels y in the image which are labeled as muclei. We utilize the CoNSeP dataset by Graham et al. (201)). along with the pre-processed versions they provide of the Kumar (Kumar et al., 2017) and TNBC (Naylor ct al. 2018) datascts. The datasets contain large H&E stained image tiles (of size 1 1000 x 1 000 or 512 512) at 40x objective magnification, with corresponding nuclear segmentation masks. The three datasets were with differing procedures for specimen preservation and staining. main/val /va and TNBC for test, we thus obtain clear distribution shift. From the large image tiles, we extract 64 x 64 patches via regular gridding, and create regression target for each image patch by counting the number of labeled muclei pixels. We then also remove all images which contain no nuclei pixels. In the end, we obtain 10 808 train images, 2 702 val images and 2 226 test images.
collected at different cispitals/institttiion
By using CoNSeP and Kumar for train/
AerialBuildinggixells Given an aerial image x, the task is to predict the number of pixels y in the image which are labeled as building. We utilize the Inria acrial image labeling dataset (Maggiori et al. 2017], which contains 180 large aerial images with correspouding building segmentation masks. The images are of size 5000 x 5 000, and are captured at five different geographical locations. We use the images from two densely populated American cities (Austin and Chicago) for train/val, and the images from a more rural European area (West Tyrol, Austria) for test, thus obtaining a clear distribution shift. We first resize the images to size 1 1000x x 000, and then extract 64 x 64 patches via regular gridding. We also create a regression larget for each image patch by counting the number of labeled building pixels. After removal of all images which contain no building pixcls, we finally obtain 11 184 train images, 2 797 val images and 3800 test images.
The additional variants of CELLS and CHAIRANGLE in Figure 1 are specified as follows. CELLS: no difference in regression target range between train/val /val and test. CELLS- TAILS: target range ]50, 150] for train/val, [L. 200] for test. We create three versions with intermediate target ranges (1: [37.5, 163.5]. 2: [25, 176). 3: [12.5, 188.5)) for test. CHAIRANGLE: no difference in target range between train/val and test. CHAIRANGLE- GAP: target range J0, 3660 90[ for train/ /val, J0, 90[ for test. We create three versions with intermediate target ranges (1: Jo, 33.75|u]56.225 25, 901, 2: J0, U5.5... 90f, 3: J0, 41.25[[|48875,, 90I) for test.
B Additional Results & Method Variations
Please note that the results in Table A1
Table A42 not are rounded/truncctte to only significant digits.
To ensure that the test coverage results do not depend on our specific choice of studying 90% prediction intervals = (01), we repeat most of the evaluation for two alternative miscoverage rates a. Specifically, we redo the evaluation of 6/10 methods on 9/12 datasets with 80% (a = 0.2) and 95% (a = 0.05) prediction intervals. The results for 80% prediction intervals are given Figure A7 Figure A10 and Table A13 - Tabll A21 while the results for 95% prediction intervals are given in Figure A1l Figure A14 and Table A22 Table A30 We observe very similar trends overall. For example, all methods still have almost perfectly calibrated prediction intervals on CELLS and CHAIRANCLE, i.e. they all obtain test coverage very close to 80%/95% but only GAUSSIAN + SELECTIVE GMM remains well-calibrated on CELLS-TAILS and CHAIRAGLLE-GE GAP. With the exception of BRAINTUMOURPIXESS, all methods are also significantly overconfident. on all the real-world datasets.
Figure A15 shows test coverage results for the five common regression uncortainty estimation methods on the AERIALBBLLDINGIEELL dataset, and on two versions with different test sets. For all three datasets, train/ /val contains images from Austin and Chicago. For ABRIAL.BUILDNGPIXS. test contains images from
22

---

Published in Transactions on Machine Learning Research (09/2023)
West Tyrol, Austria. For AERIALBUTLDINGPPEELSSS KITSAP, test instead contains images from Kitsap County, WA. For AARAA.BUULLDNNNGPIIISSS VIENNA, test contains images from Viemna, Austria. Intuitively, the distribution shift between train/val and test could potentially be smaller for AERAABBULDINGIIESS-KTTEE and AERIABBIILDNGPIIEESSS VIENNA than for the original AERIAL BIRRIALBCILIINPIXEEL,,,, but we observe no clear trends Figure A15
Figure A16 & A17 present study in which we aim to relate the test coverage performance quantitative measure of distribution shift ("distance" between the distributions of train/val and test), complementing qualitative discussion in the Performance Differences among Real- World Datasets are Mostly Logical paragraph of Section 7 How to quantify the level of distribution shift in real-world datascts is however far from obvious, see e.g. Appendix E.1 in (Wenzel et al., 2022). We here explore if the difference in regression accuracy (MAE) on val and test can be adopted as such a measure, extending the approach by [Wezell et al. (2022) to our regression setting. We compute the average test coverage error for the five common regression uncertainty estimation methods, whereas the val/test MAE is for the CONFORMAL PREDICTION method (standard direct regression models). The results for the six synthetic datasets are presented in Figure A 16 and for the six real-world datasets in Figure A17 We observe that, in general, larger distribution shift mcasure does indeed seem to correspond to worse test coverage performance. Among the 12 datascts, HISTOLOGYNUCLEI is the only one that quite clearly breaks this general trend.
Apart from the main comparison of the 10 methods specified in Section 4 we also evaluate a few method variations. The results for these experiments are provided Table A31 Table A42 For GAUSSIAN + SELECTIVE GMM, we vary the number of GMM mixture components from the standard k 4 k 2 and & = but observe no particularly consistent significant trends in the results. Similarly, vary the nunber of neighbors from the standard k = 10 to k = 5 and k 20 for GAUSSIAN SELECTIVE KNN, but observe no clear trends here either. For GAUSSIAN + SELECTIVE KNN, we also explore replacing the cosine similarity distance metric with L2 distance, again obtaining very similar results. Following Mukhoti et al. (2021b), we finally add spectral normalization (Miyato et al. 2018) for further feature-space regularization, but observe no significant improvements for GAUSSIAN + SELECTIVE GMM.
Average test Coverage Error (1)
0.14
0.13
0.12
0.11
0.1
1
4
Average val MAE Rank (4)
Conformal Prediction
Ensemble
Gaussian
Gaussian Ensemble
Quantile Regression
Performance comparison of the five common regression uncertainty estimation methods on the six real-world datasets, in terms of average test coverage error and average val MAE rank (the five methods are ranked 1 5 in terms of val MAE on each dataset, and then the average rank is computed). ENSEMBLE and GAUSSIAN ENSEMBLE achieve the best performance.
Figure A1:
23

---

Published in Transactions on Machine Learning Research (09/202))
Average test Coverage Error (1)
0.14
0.13
0.12
0.11
0.1
4
Average val Interval Length Rank (1)
Conformal Prediction
Ensemble
Gaussian
Gaussian Ensemble
Quantile Regression
Figure A2: Performance comparison of five common regression uncertainty estimation methods on the six real-world datasets, in terms of average test coverage error and average val interval length rank (the five methods ranked 1 - in terms of val interval length each dataset, and then the average rank is computed). ENSEMBLE and GAUSSIAN ENSEMBLE achieve the best performance.
Average val Interval Length Rank (1)
4
3
2
2
4
Average val MAE Rank (1)
Conformal Prediction
Ensemble
Gaussian
Gaussian Ensemble
Quantile Regression
Figure A3: Performance comparison of the five common regression uncertainty estimation methods on the six real-world datasets, in terms of average val interval length rank and average val MAE rank. ENSEMBLE and GAUSSIAN ENSEMBLE achieve the best performance.
24

---

Published in Transactions on Machine Learning Research (09/2023)
test. Coverage
0.9
0.8
0.7
0.6
0.5
Cclls
Cells-Tails
Colls-Gap
ChairAngle
ChairAngle- Tails
ChairAngle-
Gap
Prediction Rate (1)
0.8
0.6
0.4
0.2
Cells
Cails-Tails
Cells-Cap
ChairAngle
ChairAngle- Tails
ChairAngle- Gap
Gaussian Ensemble
Caussian Ensemble
Seloctive CMM
Gaussian Eusemble
Selective Ensemble Var
Figure A4: Results for the two selective prediction methods based GAUSSIAN ENSEMBLE, the six synthetic datasets. Top: Results in terms of test coverage. Bottom: Results in terms of test prediction rate.
Test Coverage
0.9
0.8
0.7
0.6
0.5
Wealthh
Ventricular Volume
Brain Tumour Pixels
SkinLesion Pixels
Histology NucleiPixels
AerialBuilding Pixels
Prediction Rate (1)
0.8
0.6
0.4
0.2
Asset Wealth
Ventricular Volume
Brain Tumour Pixels
SkinLesion Pixels
Histology NucleiPixcls
AcrialBuilding Pixcls
Gaussian Ensemble
Gaussian Ensemble Selective GMM
Gaussian Eusemble Selective Ensemble Var
Figure A5: Results for the two selective prediction methods based on GAUSSIAN ENSEMBLE, on the real-world
datasets. Top: Results in terms of test. coverage. Bottom: Results in terms of test. prediction rate.
25

---

Published in Transactions on Machine Learning Research (09/2023)
Average test Coverage Error (1)
0.14
0.12
0.10
0.08
0.06
0.76
0.78
0.8
0.82
0.84
0.86
0.88
Average test Prediction Rate (t)
Gaussian + Selective GMM Gaussian Ensemble + Selective GMM
Gaussian Selective kNN
Gaussian Selective Var
Gaussian Ensemble + Selective Ensemble Var
Figure A6: Performance comparison of the seleetive prediction methods across all datascts, in terms of average test coverage error and test prediction rate. GAUSSIAN ENSEMBLE SELECTIVE GMM achieves the best coverage error, but also has the lowest prediction rate. For these five methods, each improvement in terms of coverage error also corresponds to decrease in prediction rate.
test Coverage
0.9
0.8
0.7
0.6
0.5
Colls
Cells-Tails
ChairAngle
ChairAngle- Gap
Figure A7: Miscoverage rate = 0.2: Rosults in terms of test coverage for four of the common regression uncertainty estimation methods (Conformal Prediction, Gaussian, Gaussian Ensemble, Quantile Regression), on four the synthetic datasets. See Table A13
Table A16 for other metrics.
test Coverage
0.9
0.8
0.7
0.6
0.5
Ventricular Volume
BrainTumour Pixels
SkinLesion
Histology NucleiPixels
AerialBuilding Pixels
Pixels
Miscoverage rate = 0.2: Results in terms of test coverage for four of the common regression uncertainty estimation methods (Conformal Prediction, Gaussian, Gaussian Ensemble, Quantile Regression), on five of the real-world datasets. See Table A17 Table A21 for other metrics.
Figure A8:
26

---

Published in Transactions on Machine Learning Research (09/2023)
test: Coverage
0.9
0.8
0.7
0.6
0.5
Cells
Cells-Tails
ChairAngle
ChairAngle- Gap
Prediction Rate (1)
0.8
0.6
0.4
0.2
Cells
Cells-TTails
ChairAngle
ChaleAngle-
Gap
Gaussian
Gaussian Selective GMM
Gaussian Ensemble
Gaussian Ensemble Selective Ensemble Var
Figure A9: Miscoverage rate a = 0.2: Results for two of the selective prediction methods, on four of the synthetic datasets. See Table A13 Table A16 for other metrics.
test Coverage
0.9
0.8
0.7
0.6
0.5
Ventricular
Brain Tumour Pixcls
SkinLesion Pixcls
Histology NuclciPixels
AcrialBuilding Pixcls
Volume
Prediction Rate (1)
0.8
0.6
0.4
0.2
Ventricular Volume
Brain Tumour
SkinLesion Pixels
Histology NucleiPixels
AerialBuilding Pixels
Pixels
Gaussian
Gaussian
Selective GMM
Ganssian Ensemble
Gaussian Ensemble + Selective Ensemble Var
Figure A10: Miscoverage rate 0.2: Results for two of the selective prodiction methods, on five of the real-world datasets. See Table A17 Table A21 for other metrics.
27

---

Published in Transactions on Machine Learning Research (09/2023)
test Coverage
0.9
0.8
0.7
0.6
0.5
Cells
Cells-Tails
ChairAngle
ChairAngle- Gap
Figure A11: Miscoverage rate a = 0.05: Results terms of test coverage for four the common regression uncertainty estimation methods (Conformal Prediction, Gaussian, Gaussian Ensemble, Quantile Regression), four of the synthetic datasets. See Table A22 Table A25 for other metrics.
test Coverage
0.9
0.8
0.7
0.6
0.5
Ventricular Volume
BrainTTmour Pixels
SkinLesion Pixels
Histology NucleiPixels
AerialBuilding Pixels
Figure A12: Miscoverage rate a = 0.05: Results terms of test coverage for four the common regression uncertainty estimation methods (Conformal Prediction, Gaussian, Gaussian Ensemble, Quantile Regression), on five of the real-world datasets. See Table A26
Table A30 for other metrics.
test Coverage
0.9
0.8
0.7
0.6
0.5
Cella
Cells-Tails
ChairAngle
ChairAngle-
Gap
Prediction Rate (1)
0.8
0.6
0.4
0.2
Cella
Cells-Taih Tails
ChairAngle
ChairAngle-
Gap
Gaussian
Gaussian
Selective GMM
Gaussian Ensemble
Gaussian Ensemble Selective Ensemble Var
Figure A13: Miscoverage rate a 0.05: Results for two of the selective prediction methods, on four of the synthetic datasets. See Table A22 Table A25 for other metrics.
28

---

Published in Transactions on Machine Learning Research (09/2023)
test Coverage
0.9
0.8
0.7
0.6
0.5
Ventricular
Brain Tumour Pixcls
SkinLesion
Histology
AerialBuilding
Volume
Pixcls
NucleiPixell
Pixcls
Prediction Rate (1)
0.8
0.6
0.4
0.2
Ventricular Volume
BrainTTumur Pixels
SkinLesion Pixels
Histology NucleiPixels
AerialBuilding Pixels
Caussian
Caussian
Selective GMM
Caussian Ensemble
Caussian Ensemble
Selective Ensemble Var
Figure A14: Miscoverage rate 0.05: Results for two of the selective prediction methods, on five of the real-world datasets. See Table A26
Table A30 for other metrics.
tcst Coverage
0.9
0.8
0.7
0.6
0.5
0.4
AerialBuildingPixels
AerialBuildingPixels Kitsap
AerialBaildingPiells Vienna
Figure A15: Results for the five common regression uncertainly estimation methods (Conformal Prediction, En- semble, Gaussian, Gaussian Ensemble, Quantile Regression) the AERIALBULLDINGPIXELL dataset, and two versions with different test sets. For all three datascts, train/we /val contains images from Austin and Chicago. For BURIALLBULDDINGPIEESS. test contains images from West Tyrol, Austria. For AERIALBUULDINGGISEER KITSAP, test instead contains images from Kitsap County, WA. For ABRIALBUULLDINGGIII PAA.DIINNGGPPI-IIIIIIIA test. contains images from Vienna, Austria.
29

---

Published in Transactions on Machine Learning Research (09/202))
Average test Coverage Error (L)
0.3
0.2
0.1
200
400
600
800
(Lest MAE val MAF) / val MAE (%)
Cells
Tails ChairAngle-Taie
Cells-Gap
ChairAngle
ChairAngle-Cap
Figure A16: Using the difference in regression accuracy (MAE) on val and test as a quantitative measure of distribution shift in each dataset (inspired by Wenzel et al. (2022), extended to our regression setting), and comparing this to the test coverage performance. The average test coverage error is computed for the five common regression uncertainty estimation methods. The val/test MAE is for the Conformal Prediction method (standard direct regression models). Results for the six synthetic datasets, In general, larger distribution shift mcasure corresponds to worse test coverage performance.
0.20
Average test Coverage Error (1)
0.15
0.10
0.05
20
40
60
80
100
120
140
160
(Lest MAE val MAE) / val MAE (%)
Asset Wealth
Ventricular Volume
Brain'Tumone Pixess
SkinLesionPixels
Histelogy/uuceiPiieee
AerialBuildingPixel
Figure A17: Exactly the same comparison as in Figure A16 but for the six real-world datascts. larger distribution shift measure generally corresponds to worse Lest coverage performance also this case. The HnstologyNucllIPPe dataset is somewhat of an outlier.
30

---

Published in Transactions on Machine Learning Research (09/2023)
Table AI: Complete results on the CELLS dataset.
Method
val MAE (L)
vul Incerval Length (4)
test Coverage (2 0.90)
test Prediction Rate (f)
Conformal Prediction
4.03121 2.78375
18.8216 +: 11.8302
0,9117 0.00275173
1.0
Ensemble
2.22525 = 0.171309
12,2881 +: 2.78795
0.8991 0.00689986
1.0
Ganesian
3.61704 1.13624
14.5492 4.44927
0.90286
0.00594629
Gaussian Ensemble
2.78757 1.10951
17.1285 + 4.33367
0,90062
0.0027671
Quantile Regression
3.70405 0.81647
13.8023 + 2. 2.19885
0,90486 0.006771
1.0
Gaussian Selective GMM
3.61704 1.16621
14.5192 + 4.14927
0.905241
0.001183155
0.90216 O. 0.00290558
Ganssian Selective KNN
3.61704 1.13624
14.5492 +: 4.44927
0.900069
0.00677637
0.94656
0.00470472
Gaussian Selective Variance
3.61704 + 1.13624
14.5492 +: 4.44927
0.904839
0.00795449
0.95712 0.00200938
Goussian Ensemble + Selective GMM
2.78757 + 1.16951
17.1285 + 4.33367
0.899999
1.00811328
0.95066 + 0.0023105
Gaussian Ensemble + Selective Eas Var
2.78757
+ 1.19951
17.1285 + 4.33367
0.896165 + 0.0015705
0.95156 = 0.00215277
Table A2: Complete results on the CELLS-TAILS dataset.
Method
val MAE ())
val Interval Length (4)
test Coverage (> 0.90)
test Prediction Rate (1)
Conformal Prediction
3.56733 +: 1.0818
14.2128 +: 4,23582
0.55356 = 0.0299096
1.0
Ensemble
1.83161 = 0.178937
9,8697 = 1.6672
0.50078 0,007608
1.0
Ganesian
4.05446 1.38153
15.4321 4.98796
0.54402 0.0347522
1.0
Gaussian Eusemble
2.40691 0.580524
10.9696
1.70051
0.5874 + 0.0479263
Quantile Regression
3.32571 1,24578
13.0818 + 3.5-239
0,52222 0.028-1102
1.0
Gaussian Selective GMM
4.05446 1.33153
15.4321 +: 4.98796
0.889825 0.0193021
0.53654 + 0,0101012
Gaussian Selective KNN
4.05-146
1.33553
13,4321 4.98796
0.859179 0.0255173
0.56692 + 0.0127107
Goussian Selective Variance
4.05446 1.33153
15.4321 + 4.98796
0.687475 0.0659807
0.6914 0,0651196
Gaussian Ensemble + Selective GMM
2.40691 + 0.580524
10.9696 + 7U051
0.877068 + 0.0222502
0.53604 + 0.0125903
Gaussian Enseuble + Selective Eas Var
2.40691 :: 0.580524
10.9696 + 7.005
0.660212 + 0.0325832
0.74512 + 0.0438128
Table A3: Complete results on the CELLS-GAAL dataset.
Method
val MAE (4)
val Interval Length (4)
test Coverage (2 0.90)
test Prediction Rate (f)
Conformal Prediction
3.67702 + 1.111...8 1335877
17.1132 + 6
0.64002 +: 0.0620287
1.0
Ensemble
2.7261 + 0.705274
13.4061 2.40513
0.6771 + 0.0544788
1.0
Gaussian
3.53089 = 1. 1.0619
15.6396 6,23458
0.66025 = 0.114703
1.0
Gaussian Ensemble
3/46118- 0.95-129
18.707 = 4.13287
0.7066 + 0.0317638
1.0
Quantile Regression
4.75328 1.73499
18.108 4.21716
0.64488 0.112855
1.0
Gaussian + Selective GMM
3.53089 = 1.0619
15.6396 + 6.23458
0.890569 + 0.00953089
0.49372 = 0.0025926
Gauesian + Selective kNN
3.53089 = 1. 1.0619
15.6396 6,23458
0.874032 = 0.0132364
0.53646 = 0.0139864
Ganssian + Selective Variance
3.53089 1.0619
15.6996 6.23458
0.652766 0.117192
0.9748 0.000940213
Gaussian Ensemble Selective GMM
3.46118
0.95-129
18.707 = 4.13287
0.896848 = 0.0127879
0.49278 0.00192914
Gaussian Ensemble Selective Ens Var
3.46118 0.95-299
18.707 4.13287
0.68326 0.0299799
0.91 + 0.0182427
Table A4: Complete results on the CHAIRANGLE dataset.
Method
val MAE (L)
val Interval Length (4)
Lest. Coverage (2 0.90)
Lest. Prediction Rate (1)
Conformal Prediction
0.289127 + 0.081643
1.08752 + 0.202247
0.905265 + 0.00339015
1.0
Euseuble
0,189858 + 0.05448452
0,788401 4: 0.137466
0.9 0.909915 +: 0.00399101
1.0
Gaussian
0.376692 0.171928
1.37757 1: 0.382191
0.901577 0.00908472
1.0
Gaussian Eisemble
0.361834 0.165348
1.2044 0.442422
0.910414
0.00316227
Quautile Regression
0.851253 0.544717
3.19741 +:: 1.9751
0.906209
0.0020-4038
Gaussian Selective GMM
0.376692 0.1719928
1,37757 0.382191
0.902482
0.00436054
0.972739 0.00191733
Gaussiun Selective KNN
0.376692 0.171928
1.37757 0.382191
0.90222 + 0.00459694
0.975465 0.00201785
Gaussiau Selective Variance
0.376692 1: 0.171928
1.37757 J: 0.382191
0.904805 0.00765098
0.961782 :: 0.0193133
Gaussian Ensemble + Selective GMM
0.361834 1: 0.165338
1.2044 + 0.442422
0.9093 # 0.00297263
0.969033 :t 0.0001940447
Gaussian Ensemble + Selective Ein Var
0.361834
0.1653348
1.2044 + 0.442422
0.905905
0.00336481
0.951359
0.00453556
31

---

Published in Transactions on Machine Learning Research (09/2023)
Table A5: Complete results on the CHAIRANGLE- datasot.
Method
val MAE (4)
val Interval Length (4)
test Coverage (2 0.90)
test Prediction Rate (t)
Conformal Prediction
0.358162 0.168257
1.3110 +: 0.531401
0.615804 +: 0.00527695
Ensemble
0.21931 0.0596705
101074 = 0.245846
0.611706 + 0.003-15611
1.0
Guussian
0,241214 0.091736
1.09417 + 0.3829077
0.622582 + 0.00714065
1.0
Gaussian Ensemble
0.13365 = 0.01 89933
0.769172 + 0.081 4039
0.617016 + 0.0063698
1.0
Quantile Regrssson
0.820934 = 0.653268
2.9815 + 2. 3,15773
0.680953 1: 0.0379624
1.0
Gaussian + Selective GMM
0.241214 0.091796
1.09417 + 0.3829977
0.901946 + 0.00382993
0.655448 + 0.001058
Gaussian + Selective kNN
0.241214 = 0.091736
1.09417 + 0.382907
0.860311 + 0.00617031
0.703038 + 0.00691227
Gaussian + Selective Variance
0.241214 0.091196
1.09417 + 0.3829007
0.647559 :: 0.03601 79
0.924383 + 0.0751205
Gaussian Ensemble + Selective GMM
0.13365 = 0.0189933
0.769172 + 0.081-1039
0.901907 + 0.00394119
0,651314 + O. 0.0013893
Gaussian Eusembble + Selective Eus Var
0.13365 0.0189933
0.769172 + 0.0814039
0. 0.751845 0.0131765
0.772401 0.00878839
Table A6: Complete results on the CHAIRANGLE- GAP dataset.
Method
val MAE (L)
val Interval Length (4)
test Coverage (2: 0.90)
test Prediction Rate (1)
Conformal Prediction
0.35031 + 0.161819
1.35 1: 0,547089
0.659546 = 0,00916108
1.0
Eusemble
0,22896 0.0853825
1.1222 : 0.174147
0.749951 :: 0.0155327
1.0
Gaussian
0.154516 + 0.280171
2.09212 : 0. 0.933756
0.69363 0.03455876
1.0
Gaussian Ensemble
0.226332 + 0.0677:113
1.30588 0.136235
0,731065 = 0.0126216
Quantile Regression
0.639151 0.296536
3.29137 + 1-53269
0.695697 + 0.0413613
1.0
Gaussian Selective GMM
0,434516 0.280174
2.08211 + 0.9333756
0,91215 + 0.00604745
0.649332 0.00334919
Gaussian Selective kNN
0,434516 + 0.280174
2.09212 + 0,933756
0.911574 + 0.00376608
0.673761 = 0.0113-132
Gaussian + Selective Variance
0.454516 1: 0.280174
2.09212 + 0.933756
0.690436 :: 0. 0.0366025
0.981292 = 0.0152406
Gaussian Ensemble + Selective GMM
0,226352 = 0.0677/113
1.30585 + O. 136235
0,905039 + 0. 0,00229837
0.618621 + 0.000943188
Gaussian Ensemble- + Selective Eas Var
0,226352 :: 0.0677413
1.30588 + 0. 03136235
0.794531 + 4.01
0.7711 = 0.0191172
Table A7: Complete results on the ASSETWEARTE EALTH dataset.
Mechod
val MAE ())
val Interval Length (8)
Lest Coverage (2 0.90)
Lest Prediction Rate ('1)
Conformal Prediction
0.346532 + 0.00878306
1,5838 + 800318086
0.87136 + 0.0090994
1.0
Ensemble
0.320002 + 0. 0.00264220
145568 + 0.0129312
0.87348 + 0.00598963
1.0
Gaussion
0.367501 + 0.0416437
1.597 = 0,207509
0,819966 + 0.0156831
1.0
Gauseian Ensemble
0.3295 0.00783906
1.42071 0.0485497
0.866162 0.00711662
1.0
Quantile Regression
0.404279 + 0.0683226
1.58957 + 0.161689
0.823921 0.0313343
1.0
Gaussian Selective GMM
0.367501 +: 0.0416437
1.597 + 0.207589
0.850824 #: 0.047333
0.93838 0.0170443
Gaussian Selective WNN
0,367301 +: 0.0116137
1.597 + 0. 0,207599
0.852107 + 0.0474473
0.933566 + 0.02035441
Gausian + Selective Variauce
0.367501 +: 0.0416437
1.597 1 0. 1.207799
0.84698 + 0.0430247
0.926874 + 0.0159098
Gaussian Ensemble + Selective GMM
0.3295 0.00783906
1.42771 0.0485537
0.868444 1 0.00727312
0.937371 4: 0.0138213
Gaussian Ensemble- + Selective Eins Var
0.3295 0.00783906
1.42071 0.04455497
0.85047 0.00397082
0.868282 + 0.0299149
Table A8: Complete results the VENTRICULAR VOTTRIU dataset.
Mothod
val MAE (1)
val Interval Length (4)
test Coverage (> 0.90)
test Prediction Rate (t)
Conformal Prediction
11.2471 + 0.201399
47.4505 + 1.556
0.603135 + 0.0125842
1.0
Ensemble
10.2476 : 0. 113707
41.8415 1 0,750822
0. 0,707367 = 0. 0.00877743
1.0
Gaussian
12.7238 + 1.52197
51.566 + 3.52739
0.730878 + 0.06190445
1.0
Gaussian Ensemble
10.1141 : 0.180661
39.9817 +: 1.91308
0.795768 +: 0.0231243
1.0
Quantile Regression
12.4944 = 0.676265
19.0118 + 3.33314
0.710972 + 0.045171
1.0
Gaussian + Selective GMM
12.7238 + 1.52197
51.566 3.52739
0.752046 + 0.0529087
0. 0.707994 = 0. 0.0208741
Gaussian Selective kNN
12.7238 1.52197
51.566 3.52739
0.735105 + 0.0612413
0.911599 = 0. 0.01/14775
Ganesian Selective Variance
12.7238
1.52197
51.566 3.52739
0.7478688 0.0569984
0.691693 0.03440016
Gaussian Ensemble Selective GMM
10.1141 + 0.160661
39.9817 +: 1.91308
0.798094 + 0.01666744
0.6466865
O. 0.0366201
Gaussian Eusemble Selective Eas Var
10.1141 = 0.180661
39.9817 + 1.91308
0.763412 + 0.0286772
0.686207 # 0.0431835
Table A9: Complete results on the BRAINTUMOURPINSET dataset.
Method
val MAE (L)
val Incerval Length (4)
test Coverage (2 0.90)
test Prediction Rate (t)
Conformal Prediction
21.3163 + 0.45997
93.71 + 2. 2.99061
0.885925 = 0.00995743
1.0
Ensemble
21.1133 = 0.210209
90.3825 0. 0,828858
0.878183 = 0. 0.00018872
1.0
Ganesian
21.0625 0.358012
93.6284 2. 2999
0.873544 0.00871896
Gaussian Ensemble
20.5336 0.211421
87.7414 0.818979
0.868126
0.00328017
1.0
Quantile Regression
22.0348 0.697606
94.3249 + 3.07265
0.879079 0.D0080396
1.0
Gaussian Selective GMM
21.0625 0. 1.358012
93. 02.2281 +:2 29916
0.883515 0.0135279
0.973576 0.0187252
Ganssian Selective KNN
21. 01.0625-4 013550122
93.6284 2.29916
0.891264 0.D07344602
0.947185 + 0.0178434
Gaussian Selective Variance
21.0625 = 0. 0.358012
93.6284 + 2. 29916
0.878666
0.00735197
0.978791 + 0.006946622
Gaussian Ensemble + Selective GMM
20.5336 = 0. 0.211421
87. 87.7414 + 0.818979
0,873821
O. 0.00-166519
0.9853-19 + 0.00461984
Goussian Ensemble + Selective Ens. Var
20.5336 + 01.1 1421
87.7414 0.818979
0.877211 + 0.00302057
0.975368 + 0.00323397
32

---

Published in Transactions on Machine Learning Research (09/2023)
Table A10: Complete results on the SKINLESIONPIXELS dataset.
Method
val MAE (L)
val Interval Length (1)
test Coverage (2 0.90)
test Prediction Rate (t)
Conformal Prediction
107.514 + 1887661
492.922 +: 14.1805
0. 0,708012 = 0.00917777
1.0
Ensemble
99.3156 0.997867
353.842 4.73577
0.742096 0.00852236
Gaussian
105. 1.15778
535.139 +: 215.416
0.797255 0.0270731
1.0
Gonssian Ensemble
100.639 01.464183
472.14 85.2654
0.822931 + 0.01343328
1.0
Quantile Regression
113.076 +: 3.20875
405.904 + 4.75158
0.719788 +: 0.0213826
1.0
Goussian Selective GMM
105.417 + 1,15178
535.139 + 215.446
0.821515 + 0.0137705
0.849579 + 0.0206181
Gaussian Selective kNN
105.417 + 1.15178
533.139 + 215.416
0.813027 + 0.0306586
0.9270447 + 0,00830718
Gaussian + Selective Variance
105.417 + 1.151 78
535.139 + 215.446
0.799821 + 0.0150835
0.77946 + 0. 0.059169
Gaussian Ensemble + Selective GMM
100.639 : 0.464183
472.14 + 85.2654
0.826921 = 0.00719649
0.839132 # 0.0108815
Gaussian Ensemble + Selective Eas Var
100.639 = 0. 16-1181
472.14 + 85,265-1
0,819129- = 0.00910122
0. 0,782116 + 0.0109776
Table A1l: Complete results on the HistoLooGYNicc.aaiixxaa dataset.
Mothod
val MAE (4)
val Interval Length (1)
test Coverage (2 0.90)
test Prediction Rate (t)
Conformal Prediction
217.887 3.71 766
993.665 18.3282
0.841288 - 0.01
1.0
Ensemble
197.078 1.99489
853.361 14.1904
0.812704 0.00664074
1.0
Gaussian
211.795 + 10.0239
1211.83
396.946
0.388796 = 0.0912794
1.0
Gaussian Ensemble
196.785 + 2.14454
1108.53
145.034
0.549936 0.05995
1.0
Quantile Regression
227. 887.895-4 + 9.37415
914.909 + 45.995
0.684076 = 0,0428704
1.0
Gaussian Selective GMM
211.795 + 10.0239
1211.83 396.946
0. 0.59554 0.0956742
0.90569 = 0. 0.0453335
Gaussian Selective KNN
211.795 + 10.0239
1211.83 396.946
0.608929
0.0805954
0.846228 0.0482832
Gaussian + Selective Variance
211.795 + 10.0239
1211.83 + 396.946
0.588596
0.0913593
0.998765 = 0.00134954
Gaussian Ensemble + Selective GMM
196.785 + 2. 14454
1108,53 + 145.033
0.553725
O. 0.0644227
0.852488 + 0.0156395
Goussian Ensemble + Selective Ens Var
196.785 +: 2.14454
1108.53 + 145.034
0.355149
0.0620052
0.954477 +: 0.0256246
Table A12: Complete results on the ABHIAL.BUILDINNIIIEISSS dataset.
Method
val MAE (1)
val Interval Length (1)
test Coverage (2 0.90)
test Prediction Rate (t)
Conformal Prediction
235.117 + 7.16096
1038.45 51.5565
0.637581 = 0.0691102
1.0
Ensemble
199.206 2.76543
798.312 6.80986
0.772494 = 0.0227288
1.0
Gaussian
217.877 + 1.72493
929.562
47.6006
0.698766 = 0.0662263
1.0
Gaussian Ensemble
208 187 + 1.03881
885.349
27,8584
0.812339 = 0.0617386
1.0
Quantile Regression
242:284 6.0122
909.191
24.9528
0.763342
0.0902358
1.0
Gaussian Selective GMM
217.877 + 1.72493
929.562
17.6606
0.76535 0.0388677
0.652082 0. 0.0990489
Gaussian Selective KNN
217.877 + 1.724193
929.562
66,6666
0,651867 = 0.0771151
0.840103 0. 0.04639899
Gaussian Selective Variance
217.877 + 1.72493
929.562 47. 6606
0.725714 0.0779375
0.708226 + 0,103803
Gaussian Ensemble + Selective GMM
208. 487 : 1.03581
885.349 + 27. 8884
0,847101 + 0.038266
0.686787 : 0.0278702
Gaussian Ensemble Selective Ens Var
208 208.487 + 1.03581
885.349 + 27, 8584
0,838352 = O. 0.0312236
0.571928 0.0402727
Table A13: Miscoverage rate a = 0.2: Results on the CELLS dataset.
Method
val MAE (4)
val Interval Length (4)
test Coverage (> 0.80)
test Prediction Rate (t)
Conformal Prediction
4.03121 + 2, 78375
11.012 = 10.2791
0,511 + 0.0100485
1.0
Gauesian
3.61704- + 1.13624
11.544 3359513
0.808 0.00859046
1.0
Gaussian Eusemble
2.78757 + 1.16951
13.1298
3.34997
0.80044 : 0. 0.00316582
1.0
Quantile Regression
4.40758 + 1-47411
13:3314
4.39511
0.80152 : 0. 0.00667005
1.0
Gaussian + Selective GMM
3.61704 + 1.13024
11.514 = 3.59513
0.811076 0.00507358
0.95216 + 0.002800113
Gauesian Ensemble Selective Ens Var
2.78757 + 1.16951
13.1298 3.34997
0.793124 0.00576659
0.95156 0.00215277
Table A14: Miscoverage rate a = 0.2: Results on the CELLS- TAILS dataset.
Method
val MAE (1)
val Inserval Length (4)
test Coverage (> 0.80)
test Prediction Rate (t)
Conformal Prediction
3.56733 + 1.0818
11.3841 + 3.54242
0.48332 = 0.0337429
1.0
Gaussian
4.05-166 + + 1.33153
12.7056 + 1.39552
0.17106 0. 0.0319071
1.0
Gaussian Ensemble
2.40691 + 0.580524
8.62187 - 1.51
0.4768 + 0.04613347
1.0
Quantile Regression
4.20398 + 1.61554
13.1241 + 4.61204
0.49358 = 0.0341114
1.0
Goussian + Selective GMM
4.05446 + 133353
12,7086 + 4.395552
0.788228 + 0.0239896
0.5364 - 00584
Gaussian Ensemble + Selective Eas Var
2.40691 1: 0.580524
8.62187 = 1.5177
0.567734 +: 0.02706448
0.74512 + 0.0435128
33

---

Published in Transactions on Machine Learning Research (09/2023)
Table A15: Miscoverage rale a = 0.2: Results on the CHAIRANGLE dataset.
Method
MAE (4)
val Interval Length (4)
test Coverage (2 0.80)
test Prediction Rate (t)
Conformal Prediction
0.289127 0.081643
0.872808 0. 0.205526
0.80212 0.00337798
Gaussian
0.376692 0.171928
1.14193 0. 0.40832
0.806646 + 0.00563869
Gaussian Ensemble
0.361831 1.163318
1.01037 + 0.3996111
0.8131522 : 1.003297:2
1.0
Quantile Regression
0,618002 0.181932
1.87322 + 0,428961
0.807127 + 0.00380433
1.0
Gaussian Selective GMM
0.376692
0.171928
1.14193 +: 0,40832
0.807801 #: 0.00666405
0.972829 +: 0.0019256
Gaussian Ensemble Selective Eus Var
0.361831 0.165318
1.01037 + 0.399611
0.801719 + 0.00318856
0.951339 = 0.00-153556
Table A16: Miscoverage rate a = 0.2: Results on the CHAIRANGLE- GAP dataset.
Method
val MAE (L)
val Interval Length (4)
test. Coverage (2 0.80)
test. Prediction Rate (1)
Conformal Prediction
0.35034 + 0.161819
1. 1.08887 + 0-497501
0.588472 + 0.01 197619766
1.0
Gaussian
0,454516 0. 0,280174
1.69802 : 0,840083
0.613951 = 0.0358797
1.0
Gaussian Ensemble
0.226352 + 0.0677413
0.9720766 0.118477
0.6431 +: 0.0136145
Quantile Regression
0.692587 0.269476
2.299958 0.738934
0.600606 + 0.0136385
1.0
Gaussian Selective GMM
0.15-1566 + 0.280174
1.69802 0,840083
0,813412 0.00720691
0.650227 0.00138861
Gaussian Ensemble- Selective Eus Var
0,226332 +: 0.06774113
0,972076 = 0.118477
0.701301 =: 0.01 46886
0.7711 = 0.0191172
Table A17: Miscoverage rate = 0.2: Results on the VENTRICULAR dataset.
Method
val MAE (L)
val Interval Length (1)
Lest Coverage (2 0.80)
test Prediction Rate (t)
Conformal Prediction
11.2471 = 0. 13999
33.0996 + 0.991594
0.457053 = 0. 0.00583106
1.0
Gaussian
12.7238 1.52197
37.3172 +: 4.91536
0.589342 + 0.0793156
1.0
Gaussian Ensemble
10.1141 = 0. 0.180661
29.9914 +: 1.12007
0.678527 0.0277009
1.0
Quantile Regression
12.5465
0.941364
35.8073 1.5139
0.587774 0.0444887
Gaussian Selective GMM
7237238 1.52197
37.3172 + 4.91536
0.609177 0.0682058
0. 0.709875 = 0. 0.0232873
Ganesian Ensemble Selective Ens Var
10.1141 0.180661
29.9914 1.12007
0.640078 0.0277429
0.686207 0.0431835
Table A18: Miscoverage rate a = 0.2: Results on the BRAINTUMOURPPNELL dataset.
Method
val MAE (L)
val Inserval Length (4)
Lest Coverage (2 0.80)
lest Prediction Rate (t)
Conformal Prediction
21.3163 0.45997
66.146 2.75637
0.824632 0.0128289
1.0
Gaussian
21.0625 = 0. 0.358012
61,2423 = 1.9975
0.798113 + 0.0133-438
1.0
Gaussian Ensemble
20.5336 : 0.21
62.5706 1 0.713494
0.796961 = 0.00658989
1.0
Quantile Regression
21.9515 = 0.1497877
66,265-1 +: 1.33091
0.8076-16 +: 0.0165337
1.0
Gaussian + Selective GMM
21. 26.0625-4 = 0 0.358012
64.2423 1.9975
0.809477 + 0.0139729
0.973576 # 0. 0.0187252
Gaussian Ensemble + Selective Ens Var
20.5336 = 0. 0.211421
62.5706 + 0.713194
0,806998 = 0.0066115-46
0.975368 + 0.00323397
Table A19: Miscoverage rate a = 0.2: Results on the SKINTESSONN dataset.
Mothod
val MAE (L)
Interval Length (1)
test Coverage (2 0.80)
test Prediction Rate (t)
Conformal Prediction
107.514 1.87461
275.191 + 5.60123
0.553254 +: 0.0122434
1.0
Ganesian
105.417 1.151
395.661 + 174.897
0.689951 0.0406148
Gaussian Eusemble
100.639 O. 0.464183
351.961 70.4308
0.715272 0.0278757
Quantile Regression
106.382 + 2.70593
251.316 + 8.67954
0.590792 + 0.0115523
1.0
Gaussian Selective GMM
105.417 1,15778
396.661 +: 174.897
0.708767
0.0274691
0.649068 0.0189464 0. 0.782116 0.0109776
Gaussian Ensemble Selective Ens Var
100.639 0.46-1183
351.961 + 70.4308
Table A20: Miscoverage rate a 0.2:
Results on the HISTOLLOGYNUCLIIPXXIL, dataset.
Method
val MAE (1)
val Interval Length (1)
test Coverage (> 0.80)
test Prediction Rate (t)
Conformal Prediction
217.887 + 3. 3.71766
288.269 +: 13.6761
0.744155 = 0.0107073
1.0
Gaussian
211.795 + 10.0239
902.884 + 295.983
0.454345 0.09869955
1.0
Gaussian Eusemble
190.785 + 14444
840.287 1 113.005
0.412526- = 0.05-11736
1.0
Quantile Regression
221.136- = 9.7918
679.117 +: 30. 30.6805
0.580503 = O. 0.0463351
1.0
Gaussian + Selective GMM
211.795 + 10. 10.0239
902.884 1: 295.983
0. 0.162248 1: O. 0.105935
0.901985 = 0.0400294
Gaussian Ensemble + Selective Ens Var
196.785 + 2.14454
840.287 + 113.005
0.417977 = 0.0572445
0.954477 + 0.0256246
34

---

Published in Transactions on Machine Learning Research (09/2023)
Table A21: Miscoverage rate a+ = 0.2: Results on the A dataset.
Method
val MAE (1)
val Interval Length (1)
test Coverage (2 0.80)
test Prediction Rate (t)
Conformal Prediction
235.417 + 7.16096
685.829 31.3009
0.52437 0.0610316
1.0
Gauesian
217.877 +: 1.724993
685.890 + 43.1749
0.60129 0.0717427
1.0
Gaussian Ensemble
208 187 +: 1.03581
658,243
27.1206
0.683599 = 0.067-1139
1.0
Quantile Regression
229.243 6.21038
6062.104 18.5495
0.635753
0.0550343
1.0
Gauesian + Selective GMM
217. +: 1.724493
685.890
43.1749
0.679246 = 0.0462652
0.65018 = 0.0999173
Gaussian Ensemble Selective Ens Var
208.487 1.03581
658.243 27.1206
0.743714 0.0354003
0.571928 0.0402727
Table A22: Miscoverage rate a = 0.05: Results on the CELLS dataset.
Mothod
val MAE (1)
val Interval Length (1)
test. Coverage (2 0.95)
lest Prediction Rate (t)
Conformal Prediction
4.03121 + 2.78375
22.3973 + 12.7194
0.95612 + 0.00460235
1.0
Gaussian
3.61704 + 1.13624
17.0459 + 4,98381
0.95402 + 0. 0,00275928
1.0
Gaussian Ensemble
2.78757 + 1.16951
20.5445 + 5.00176
0,9484 + 0. 0,00331843
1.0
Quantile Regression
3.5478 + 1.25834
17.8244 + 3,57608
0.94858 + 0.005653346
1.0
Gaussian Selective GMM
3.61704 + 1.13624
17.0459 + 4,98381
0,955558 0.0032579
0.95212 + 0.00276362
Gaussian Ensemble Selective Ens Var
2.78757 + 1.16951
20.5445 5.00176
0.946085 + 0.00337562
0.95156 + 0.00215277
Table A23: Miscoverage rate a = 0.05: Results on the CELLS-TAILS dataset.
Method
val MAE (1)
val Interval Length (4)
test Coverage (> 0.95)
test Prediction Rate (f)
Conformal Prediction
3.56733 + 1.0818
16.3531 + 4.80611
0.59048 = 0. 0.0307226
1.0
Gaussian
4.05446 1.33153
17.5444 = 5.3978
0,59006 0. 0.0111946
1.0
Ganesian Ensemble
2.40691 + 0.580524
13.1205 + 1.81604
0.67218 0.0308246
1.0
Quantile Regression
3.34375
0.642763
14.5539 2.01592
0.5606 0.0295055
Ganesian Selective GMM
4.05446 1.33153
17.5443 5.9978
0.941551
0.01645:39
0.53636 0.0100057
Gaussian Ensemble Selective Eas Var
2.40691 0.580524
13.1205 +: 1.81604
0.720221 0.0374625
0.74512 + 0.0438128
Table A24: Miscoverage rate 0.05: Results on the CHAIRANGLE dataset.
Method
val MAE (1)
val Interval Length (1)
test Coverage (2 0.96)
test Prediction Rate (t)
Conformal Prediction
0.289127 = 0.081643
1.27437 + 0. 18759
0,953087 + 0.000729:125
1.0
Gaussian
0.376692 = 0.171928
1.59112 + 0,367248
0.919351 + 0.00143361
1.0
Ganssian Ensemble
0.3618344 + 0.165348
1.38384 0.481345
0.95633 + 0.0016841
1.0
Quantile Regression
0.888707 = 0.472522
4.04185 + 1.50356
0.951608 + 0.00184683
1.0
Gaussian + Selective GMM
0.376692 = 0,171928
1.59112 0.367248
0.9-99963 + 0. 0.00265095
0.97208 = O. 00.000666888
Gaussian Ensemble- + Selective Ens Var
0.3618344 + 0.165348
1.38384 0.481345
0.9541 6.001
0.951359 0.00453556
Table A25: Miscoverage rate a 0.05: Results on the CHAIRANGLE GAP dataset.
Method
val MAR (L)
val Interval Length (4)
test Coverage (2 0.95)
test Prediction Rate (t)
Conformal Prediction
0,35031 + 0,161819
1.56261 + 0.382951
0.697817 + 0,00918537
1.0
Gaussian
0.154516 1: 0. 0.280174
2,39892 # 1.02172
0.733595 :: 0. 0.0367327
1.0
Gaussian Ensemble
0.226352 + 0.0677413
1.59703 + 0. 159795
0.780615 + 0.0141415
1.0
Quantile Regression
1,43807 0. 0.99796
5.57113 + 1.71783
0.759929 0.0344655
1.0
Gaussian + Selective GMM
0.454516 0.280114
2,30892 = 1.02472
0,95774 + 0. 0.00281098
0.64882 = 0. 00270888
Gaussian Ensemble Selective Ens Var
0.226332 + 0.0677413
1.59703 0.159795
0.844587 + 0.0165508
0.7711 + 0.0191122
Table A26: Miscoverage rate a = 0.05: Results on the VENTRICULARVOLUME dataset.
Method
val MAE (L)
val Inserval Length (4)
test Coverage (> 0.95)
test Prediction Rate (t)
Conformal Prediction
11.2471 = 0 0.991339
70.211 1.37665
0.776176 0.00853621
1.0
Gaussian
12. 02.7238 + 1.52197
67.1562 + 6.31002
0.837931 + 0.0396691
1.0
Gaussian Ensemble
10.1141 + 0.180661
51.9065 + 3.94789
0.876646 + 0.01409944
1.0
Quantile Regression
12.0793 : 0.130032
61.623 +: 3.04835
0.843574 + 0.0243639
1.0
Goussian + Selective GMM
12,7238 + 1.52197
67. 67.1562 + 6.31002
0.854302 + 0.0279176
0.710188 + 0.0225912
Gaussian Ensemble + Selective Eas Var
10.1141 1: 0.180661
51.9065 + 3,94789
0.855053 + 0.0125036
0.686207 + 0.0131835
35

---

Published in Transactions on Machine Learning Research (09/2023)
Table A27: Miscoverage rate a = 0.05: Results on the BRAINTUMOURPHA dataset.
Mothod
val MAE (L)
val Interval Length (1)
test Coverage (2 0.95)
trst Prediction Rate (1)
Conformal Prediction
21.3163 + 0.45997
123.719 + 2.07495
0.921113 = 0.00123648
1.0
Ganesian
21 21.0625 + 0.358012
123.587 - 3.3168
0.910141 0.00537523
1.0
Gaussian Ensemble
20.5336 = 0,211421
113.592 +: 1.37398
0.905182 = O. 0.00359141
1.0
Quantile Regression
24.6897 2. 2.49358
126.788 + 7:00165
0.915995 = O. 0.009563349
1.0
Gaussian Selective GMM
21.0625 = 0. 0,358012
123.587 = 3.3168
0.9194355 +: 0.0105219
0.973672 = 0. 0.0186412
Ganesian Ensemble + Selective Ens Var
20.5336 0.211421
113.582 + 1.37398
0.910995 0.00267557
0.975368 + 0.00323397
Table A28: Miscoverage rate a 0.05: Results on the SKINLESIONPIXELS datasct.
Method
val MAE (L)
val Interval Length (1)
test Coverage (2 0.95)
test Prediction Rate (t)
Conformal Prediction
107.514 + 1.87464
845. + 31.2572
0.822753 + 0.0104666
1.0
Gaussian
105.417 + 1.15178
691.768 +: 232.071
0.866844 +: 0.0166304
1.0
Gaussian Ensemble
100.639 0. 16-11881
582.719 + 95.3488
O 0.881009- = O. 0.00686587
1.0
Quantile Regression
112.07 +: 5.13986
627.938 68.3371
0.827977 +: 0.017595
Gaussian Selective GMM
105.117 1.15178
691.768 + 232.071
0. 0.889891 = 0. 3.000910055
0.849225 + 0. 0.0194889
Ganesian Ensemble Selective Ens Var
100.639 0.4641883
582.719 95.3488
0,881749 0.005349917
0. 0.782116 0.0109776
Table A29: Miscoverage rate a = 0.05: Results on the HiSTO..cYYNCLLIIIIII datasel.
Method
val MAE (1)
val Interval Length (1)
test Coverage (2 0.95)
test Prediction Rate (t)
Conformal Prediction
217.887 + 3.71 766
1282.78 25.9915
0.896603 0.01 13207
1.0
Gaussian
211.795 + 10.0239
1491.62 472.378
0.680015 = 0.0860417
1.0
Gauesian Ensemble
196, 785 + 2.14454
1358.12 174.189
0.66749 0.0522544
1.0
Quantile Regression
251.562 12.244
1191.96
65.3122
0.754566 0.0379023
1.0
Gaussian + Selective GMM
211.795 + 10.0239
1491.62 472.378
0.68897 + 0.0895173
0.901191 0.0405698
Gauesian Ensemble Selective Ens Var
196. 755 + 2.14454
1358.12 174.189
0.673945 = 0.0533738
0.9544477 + 0.0256246
Table A30: Miscoverage rate = 0.05: Results on the AFRIAL dataset.
Method
val MAE (1)
val Interval Length (L)
test Coverage (2 0.95)
trstt Prediction Rate (t)
Conformal Prediction
235.117 + 7. 16096
1474.93 + 53.630)
0,731414 +: 0,070291
1.0
Gaussian
217.877 + 1.72493
1181.55 41.5859
0.773522 - 0.05448329
1.0
Gaussian Ensemble
208. 487 : 1.03581
1108.87
21.3551
0.88874 0.01902113
1.0
Quantile Regression
294.181 +: 19.9268
1281.37
51.1001
0.888175 0.028968
1.0
Gaussian + Selertive GMM
217.877 + 1.72493
1181.55
41. 41.5859
0.831501 - 0.0374173
0.650797 0.0987431
Gaussian Eusemble Selective Ens Var
208.487 ::2 1.03581
1108.87 21.3551
0.899835 = 0.02633409
0.571928 + 0.0402727
Table A31: Method variation results on the CELLS dataset.
Method
val MAE (4)
val Interval Length (L)
fest Coverage (2 0.90)
test. Prediction Rate (1)
Gaussian +
Selective GMM
3.61704 + 1.13624
14.5492 + 4.44927
0.905241 + 0.00448315
0.95216 + 0 0.00290558
Gaussian + Selective GMM, & ===2
3.61701 1.13621
14.5192
1.41927
0.9014 + 0. 0.001889555
0.952 0.00270259
Gaussian Selertive GMM, k
3.617044 1.13624
14.5492
4.44927
0.904928
0.00469093
0.9528 0.00331662
Gaussian Selective GMM, Spectral Norm
4.80289 1.89759
18.5596
931331
0.908 0. 0.00703097
0.95118 + 0.00220672
Ganssian Selective kNN
3.61704 + 1.13621
14.5192 + 1.44927
0,900069
0.00677637
0.94656 + 0. 0.000100122
Gaussian Selective KNN, & = 5
3.61 + 1.19624
14.5492 + 4.44927
0.900146
0.00671 1325
0.94736 + 0.003996049
Gaussian +
Selective KNN, & = 20
3.61 + 1.13624
14.5492
4.44922
0.900048
0.00691667
0.94624 + 0.00248564
Gaussia + Selective KNN, L2
3.61704 1.13624
14.5492 4.44927
0.906295 0.00579854
0.95166 1 0.00422734
Table A32: Method variation results on the CELLS-TAILS dataset.
Method
val MAE (4)
val Interval Length (L)
trst Coverage (2 0.90)
test. Prediction Rate (t)
Gaussian +
Selentive GMM
4.05446 1.33153
15.4321
4.98796
0.889825 + 0.0193021
0.53654 0.0101012
Gaussian + Selective GMM, &
1.05116 1.33153
15.1321
1.98796
0.891933 + 0.0175265
0.525 # 0.00529
Gaussian Selective GMM, &
4.05446 1.33153
15.4321
4.98796
0.893618 1 0.0182098
0.33516 1: 0.00975635
Gaussian Selective GMM, Spectral Norm
4.91114 2.46961
17.3945 +: 7.3177
0.881351 :: 0.021587
0.5331 1: 0.0137885
Gaussian Selective kNN
1.05116 + 1.33153
15.1321 + 1.98796
0.859179 + 0.0255173
0.56692 0.0127107
Gaussian + Selective KNN, A = 5
4.05146 1 1.33153
15.4321 + 4.98796
0.856387 + 0.0261636
0.57206 : 0.0129226
Gaussian + Selective KNN, & = 20
4.05446 + 1.33153
15.4321
+ 4.98796
0.862429 + 0. 0.02644602
0.56236 + 0.0122017
Gaussian + Selective KNN, L2
1.05116 1.33153
15.1321 1.98796
0.895932 = 0.00921808
0.51 + 0600
36

---

Published in Transactions on Machine Learning Research (09/2023)
Table A33: Method variation results on the CELLS-GAP datasct.
Method
val MAE (4)
val Interval Length (L)
test Coverage (2 0.90)
test Prediction Rate (1)
Gaussia +
Selective GMM
3.53089 +: 1.0619
15.6396 1: 6.23458
0.890569 0.00953089
0.49372 0.0025926
Gaussian + Selective GMM, k
3.53089 1.0619
15.6396
6,23458
0.892265 0.00867262
0.48902 4: 0.00420828
Gaussian Selective GMM, k
3,53059 = 1.0619
15.6396
6,23-158
0.86967 0.0111239
0.50196 +: 0. 0.001120011
Gaussiau Selective GMM, Spectral Norm
2.81679 1 0.42988
12.194 + 1.10907
0.8836444 + 0.0167566
0.4991 = 0.00761236
Ganssian Selective KNN
3.53089 + 1.0619
15.6396 + 6.23-158
0.874032 + 0.013236-1
0. 0,53616 0,0139861
Gaussian + Selective KNN, A = 5
3.53059 = 1.0619
15.6396 + 6.23-158
0.874822 + 0.0121219
0,5368 +: 0. 0.0137332
Gaussian + Selective kNN, & = 20
3.53059 + 1.0619
15.6396 6.23-158
0.873876 + 0.0443446
0.536 +: 0.014724
Gaussian +
Selective KNN, L2
3.53089 + 1.0619
15.6396 6.23-158
0.887052 + 0.01-44573
0.49576 + 0.00474578
Table A34: Method variation results on the CHAIRANGLE dataset.
Method
val MAE (4)
val Interval Length (4)
test Coverage (2 0.90)
test Prediction Rate (1)
Gaussian Selective GMM
0.376692 + 0.171928
1.37757 + 0:382191
0.902482 + 0.00436054
0.972739 + 0.00191733
Gaussian Selective GMM, k =
0.376682 1: 0.171928
1.37757 1 0.382191
0,902972 + 0.0054007S
0,96392 + 0.00121759
Gaussian + Selective GMM, & = 8
0.376692 1 0.171928
1.3775 1 0.382191
0,901822 1 0.00326732
0,981238 1: 0.00113892
Gaussian + Selective GMM, Spectral Norm
0.516487 + 0.227015
1.82029 + 0.809339
0.903828 + 0.00534801
0.970993 + 0.000466666
Gaussian + Selective KNN
0.376692 + 0.171928
1.37757 + 0.382191
0.90222 0.00459694
0,975-165 0.00201785
Gaussian Selective KNN, R
0.376692 + 0.171998
133757 = 0 0.382191
0,902561 + 0,003-2263
0,983801 0.00152586
Gaussian Selective WNN, A 20
0.376692 + 0.177998
1.37757 = 0.382191
0,902137 : 0.00457917
0. 0.966392 = 0,00129222
Gaussian +
Selective &NN, L2
0.376692 1 0.171928
1.37757 1 0.382191
0,903183 1 0.00504343
0.977247 1: 0.00149241
Table A35: Method variation results on the CHAIRA NGLE- TAILS dataset.
Method
val MAE (4)
val Interval Leugth (4)
Lest Coverage (2 0.90)
test Prediction Rate (1)
Gaussian +
Selective GMM
0,211214 # 0.091736
1.09417 + O. 0,382907
0.901946 + 0.00382993
0.655418 + 0. 00.001558
Gaussian + Selective GMM, 2
0.241214 1: 0.091736
109417 1 O. 0.382907
0,903353 0.00471599
0.6454152 0.00155062
Gaussian Selective GMM, == 8
0.241214 1 0.091736
1.09417 1. 0.382907
0,902103 1 0.00382668
0.660472 1. 0.00115112
Gaussian + Selective GMM, Spectral Norm
0.375118 + 0.24509
1.24343 + 0.611561
0,910147 + 0.00315697
0.65518 + 0.001 76257
Gaussian + Selective KNN
0,241214 + 0.091 7336
1.09417 + 0,382007
0.860311 + 0.00617031
0.703038 +: 0.00691222 227
Gaussian + Selective KNN.
0,241214 + 0. 0.091736
1.09:17 + O 0,382907
0.88625-# 0.00198864
0.683742 + 0.00547083
Gaussian + Selective LNN. R == 20
0.2112141 : 0. 0.091736
1.09117 : O. 3,382977
0.809351
0.00761886
0.7412287 0.0107676
Gaussian Selective KNN, L2
0,241214 + 0.091736
1.09417 + 0.3829077
0.89682 0.00698641
0.662076 + 0.00289246
Table A36: Method variation results on the CHARANGLE- GAP dataset.
Method
val MAE (1)
val Inserval Length (L)
test Coverage (2 0.90)
test Prediction Rate (1)
Gaussian + Selective GMM
0.151516 = 0,280174 74
2.09212 4 0. 0.933756
0,91215 + 0. 0.00601745
0.619372 # 0 0.00331919
Gaussian + Selective GMM, 2
0.454516 = 0,280174
2.09212 1: 0.933756
0.912708 0.00541199
0.642138 : 0. 0.00143389
Gaussian + Selective GMM, k=
0.454516 0,2801
2.09212 0.933756
0.911374
0.00475979
0.654824 0.003689799
Gaussian Selective GMM, Spectral Norm
0,294511 4 0.0861797
1.19904 I 0.1933446
0.910836
0.00442911
0.649996 + 0.00233428
Gaussian + Selective KNN
0.454516 + 0.2801 74
2.09212 + 0. 0.933756
0.91
0.00376608
0.673764 + 0--000111-33 13432
Gaussian + Selective kNN, ==5
0.154516- = 0,2901 74
2.09212 + 0 0.933756
0,91221 + O. 0.004265119
0.6706-16 # 0. 0.00691766
Gaussian + Selective KNN, A 20
0.154516 = 0,280174
2.09212 +: 0.933756
0.9075-13
0.00201721
0.680107 + 0.0162927
Ganssian Selective KNN, L2
0.454516 + 0.250174
2.09212 + 0.933756
0.913383
0.00539852
0.658744 0.00274677
Table A37: Method variation results on the ASSET WEALTH dataset.
Method
val MAE (L)
val Interval Length (L)
test Coverage (2 0.90)
test Prediction Rate (1)
Gaussion + Selective GMM
0.367501 0.0416437
1.507 à 0.207580
0.850824 : 0.047533
0.93838 + 0.0170443
Gaussian + Selective GMM, k 2
0.367501
0.0416437
1.597 : 0,2007599
0.850232
0.0469534
0,933687
0.0176348
Gaussian + Selective GMM, k 8
0.367501
0.0416437
1.507 0.207589
0.85067
0. 0.0456954
0,932879
0.0102523
Gaussian Selective GMM, Spectral Norm
0.354647 + 0.00973911
1,55994 + 0.0421589
0.872742
0.0250168
0,941812
0.0101895
Gaussian + Selective KNN
0.367501
0.0416437
1.597 1 0.20/7599
0.852107 1: 0.04744744
0,933586
0.02015541
Gaussian + Seloctive INN, A = 5
0.367501
0.0416437
1.507 1. 0.207580
0.85256 1 O. 0.0469571
0.933232
0.02149
Gaussian + Selective KNN,
0.367501
0.0416437
1.507 0.207580
0.851914 0.0460655
0.933636
0.0200087
Gaussian Selective KNN, L2
0.367501
0.0416437
1.597 1: 0,207589
0. 0.853366 0.0410864
0.897149
0.0162693
Table A38: Method variation results on the VENTRICULAR VOLUME dataset.
Method
val MAE (L)
val Interval Length (1)
test Coverage (2 0.90)
test Prediction Rate (1)
Gaussian +
Selective GMM
12.7238 + 1.52197
51.566 + 3.52739
0.752046 + 0.0629007
0.707994 + 0.1208741
Gaussian + Selective GMM, A m 2
12.7238 + 1.52197
51.566 + 3. 3.52739
0.745271 + 0.0582068
0.790752 + O 0.03076666
Gaussia + Sclective GMM, A == S
12.7238 1: 1.52197
51.566 4. 3.52739
0.747625 0.0584117
0.656583 1: 0.0258853
Gauseian Selective GMM, Spertral Norm
11,6311 4 0.483357
45.5475
0.747647
0.734907 + 0.00853342
0.719279 4: 0.0147984
Gaussian Selective KNN
12.7238 + 1.52197
51.566 + 3.52733
0.735100 0.0612413
0.91 1599 + 0.0447475
Gaussian + Selective KNN, & 5
12.7238 + 1.52197
61.566 + 3.52739
0.735857 + 0.060-6631
0.916614 + 0.0419073
Gaussian + Selective KNN, à= 20
12.7238 1: 1.52197
51.566
4: 3.52739
0.734752 4: 0.0616982
0.909404 1: 0.0469634
Gaussian + Selective KNN, 1.2
12.7238 + 1.52197
51.560 + 3-52739
0.740812 + 0.049802
0.740439 + 0.0222073
37

---

Published in Transactions on Machine Learning Research (09/2023)
Table A39: Method variation results on the BRAINTUMOURPIXELS dataset.
Meshod
val MAE (L)
val Intercall Length (4)
lest Coverage (2 0.90)
test Prediction Rate (1)
Gaussian Selective GMM
21.0625 + 0.358012
93.6284 + 2.29916
0.883515 3.079
0.973576 + 0.0187252
Gaussian Selective GMM, A 2
21.0625 + 0. 0.358012
93.6281 + 2.29916
0,881874 + 18316
0.977863 + 0.01 45103
Gaussian + Selective GMM, & = 8
21.0025 = 0.358012
93.6281 + 2.29916
0,880823 = 0.01273
0.973117 + 0.01856
Gaussian + Selective GMM, Spectral Norm
22.0729 + 1.252733
95.0605 + 3.40124
0.890506 0.01 12498
0.9738 + 0.00857U87
Gaussian Selective KNN
21,0625 + 0.358012
93.6284 + 2.29916
0.8912644 + 0 0.007341602
0,947185 + 0.01784334
Gaussian Selective KNN,
21.0625 + 0.358012
93.6281 + 2.29916
0.891103 = 0 0.00680399
0.919392 + 0.0173229
Gaussian Selective KNN, == 20
21.0625 + 0.358012
93.6281 +: 2 2.29916
0.891759 + 0.00801323
0.94453 0.0188456
Gausian Selective KNN, L2
21.0625 1 0.358012
93.6284 1: 2.29916
0.879639 +: 0.0084247
0.981862 + 0.0068585
Table A40: Method variation results on the SKINLESIONPIXELLS dataset.
Method
val MAE (4)
val Interval Length (L)
test Coverage (2 0.90)
test Prediction Rate (1)
Gaussian Selective GMM
105.117 1.15178
535.139 + 215 156
0.821515 = 0.0137705
0.819579 = 0.0206181
Gaussian Selertive GMM, k = 2
105,417 + 1.15178
535.139
215.446
0.825054 + 0.0122688
0.844799 + 0.0191042
Ganssian + Selective GMM, &t
105.417 + 1.15178
535.139
+ 215.446
0.815696 + 0.010454
0.862683 = 0.0259293
Gaussiu + Selective GMM, Spectral Narm
107.531 1: 2.11824
607.435
407.758
0.821797 + 0.0350093
0.837273 = 0.0205022
Gaussian Selective kNN
105.417 1.15178
535.139
215.446
0.8130027 + 0.0306586
0.927047 0.00830718
Gaussian Selective KNN, &
105.417 1.15178
535.139
215.446
0.812763 + 0.0309198
0.928375 + 0.0100103
Gaussiau Selective kNN, & 20
105.417 +: 1.15178
535.139
+: 215.
0.812419 + 0.0306294
0.929172 = 0.0109976
Gaussian Selective kNN, L2
105.417 +: 1.15178
535.139
+: 215.446
0.808667 + 0.01 12832
01831 784 =: 0.0181658
Table A41: Method variation results on the HISTOLOGGYNUCIIIPIXELS dataset.
Method
val MAE (4)
val Interval Leugth (L)
test Coverage (> 0.90)
lest Prediction Rate (1)
Gaussion Selective GMM
211.795 10.0239
1211.83 + 396.910
0.59551 = 0.09567112
0. 0.90569 0.0153555
Gaussian Selective GMM, A
211.795 + 10.0239
1211.83 + 336.9996
0.597998 + 0.094508
0.912836 + 0.0323147
Gaussian + Selective GMM, k =-
211.795 + 10.0239
1211.83 + 393.99966 946
0.595718 + 0.0973458
0.913419 = 0.0409511
Gaussian + Selertive GMM, Spectral Norm
206.492 7.82601
896.779
61.4921
0.65672 0.0824111
0.855668 :: 0.0632114
Gaussian + Selertive KNN
211.795 10.0239
1211.83
3961.946
0.608929 + 0.0803054
0.846228- 0.0482832
Gaussian Selective KNN, & 5
211.795 10.0239
1211.83
396.39666 9-16
0.610025 + 0.0809455
0,842258 0.046231
Gaussian Selective kNN, & = 20
211.795 10.0239
1211.85 8311.83-4
396.946
0.607876 4 0.0817183
0.858491 :: 0.0496696
Gaussian Selertive KNN, 1.2
211.795 + 10.0239
1211.83 + 396.946
0.588514 + 0.0914286
0.975562 + 0.0190356
Table A42: Method variation results on the ABRLALBUILLINGPIIELLS dataset.
Method
val MAE (4)
val Interval Leugth (L)
lest Coverage (2 0.90)
lest. Prediction Rate (f)
Gaussian Selective GMM
217.877 + 1.72493
929.562 : 47.6606
0.76335 0.0385677
0.652082 = 0.0990
Ganssian Selective GMM, &
217.877 + 1.72493
929.562 + 47.6606
0.751721 + 0.0454772
0.634602 + 0.1
Gaussian + Selective GMM, A ==
217.877 = 1.72/13
929.562 + 17. 6606
0.850213 = 0,025791
0.617738 = 0. 0.0879951
Gaussiun + Selertive GMM, Spectral Norm
220.763 7.23119
1014.77 + 164. 155
0.817517 + 0.0505544
0.638149 0.046255
Gaussian + Selective KNN
217.877 1.72493
929.562 47.6666
0.651867 + 0.0771154
0.840103 = 0.0463980
Gaussian Selective KNN, 5
217.877 1.72493
929.562
47. 47.6606
0.654256 1: 0.0771566
0.846007 :: 0.0456245
Gaussian Selertive KNN, & 20
217.877 1.72493
929.562
47.6606
0.647779 + 0.0795474
0.832596 0.0542738
Gaussian Selective KNN, L2
217.877 + 1.72493
929.562 + 47.6606
0.815075 + 0.0278439
0.641851 + 0.11347
38


Reviews:
[{"review_id": "bAzl3KTloQ", "reviewer": "Reviewer_ztGJ", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\nThe datasets and methods evaluated are well organized and diverse. The text is well written and is organized well overall.\n\n\nWeaknesses:\nIt is unclear how the model MAE performance compares to the SOTA models for these datasets. Perhaps the performance conclusions on real world datasets is because these models are not representative of models trained by experts in those domains. However this does not change the fact that the uncertainty should have been predicted higher given any performance of the model but it would be nice to present it here anyway.\n\nThere are not many specific claims made by the paper. I find it hard to enumerate the takeaways. There is a lot of discussion but the statements there are very light and hard to conclude from. For example \"feature-space methods perform well relative to common baselines, the resulting selective prediction methods are still overconfident in many cases\" maybe adding quantitative differences here can make it more actionable. Also, hypothesizing why this is the case given your experience working with these models.\n\nThe paper presents one main experiment that is repeated across different datasets. This is limiting because we just see one perspective of a potentially very dynamic space of models/methods. It might provide more concrete evidence to see the distributional shift vary based on some parameter and then look at the response of the uncertainty methods. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This work benchmarks a variety of methods that compute prediction intervals on a collection of synthetic and real-world regression tasks. The primary evaluation criteria is \"Coverage\" which is what proportion of predictions of predictions are within the prediction interval on a held-out dataset.\n\nStatements/Conclusions drawn by this work:\n> \"while all methods are well calibrated when there is no distribution shift, they become highly overconfident on many of the benchmark datasets\"\n\n> \"not a single method actually reaches the desired 90% test coverage on any of these real-world datasets\"\n", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Can you contrast the achieved performance of the models trained for this work with published SOTA numbers. Perhaps as another row in the Appendix tables that show the MAE scores. Or a scatter plot in the main text?\n\nCan you add more actionable conclusions? The paper states that these methods don't perform well under distributional shift, have the experiments yielded any more conclusions that you can enumerate in the conclusion section?\n", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "gDPjNcR5F5", "reviewer": "Reviewer_c8Ct", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength:\n \n * Novelty & Significance: This work conducted a comprehensive study for well-known regression techniques on relevant ML settings (deep regression for vision recognition under real-world distributional shift). The performance axes measured (CI coverage, CI interval width, selective prediction rate) are relatively comprehensive. The conclusion draw from the empirical studies (esp the failure mode of popular methods such as conformal / quantile regression) is worthwhile to put out there for the community.\n  * Clarity: The paper is clearly written and with experiment results neatly summarized in few key figures.\n\nWeakness:\n\n  * Experiment Evaluation: Currently, author measured the three performance axes (CI coverage, CI interval width, selective prediction rate) independently without investigating their tradeoff relationships. Furthermore, the distributional shift evaluation is done on test splits with fixed degree of shift. Therefore it is difficult to understand how method performance deteriorates as the degree of distributional shift increases. The experiment evaluation can be made much stronger by considering these aspects (please see \"Requested Chanegs\").\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This work conducts a benchmark study for (deep) regression models under realistic distributional shifts. Authors defines calibration as the coverage probability of 90% confidence intervals, and examined the calibration and selective performance of 10 variants of canonical uncertainty techniques and their combinations (conformal prediction, quantile regression, frequentest Gaussian regression, approximate Bayesian regression). Author considered 8 vision benchmarks covering medical imaging, CAD design, and remote sensing, each equipped with corresponding OOD splits. Author find that the methods such as quantile regression and conformal prediction under-perform despite their i.i.d. theoretical guarantees, and the performance of all methods (even those designed for OOD detection) deteriorate under real-world distributional shifts.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "* **Investigate the relationship between accuracy, calibration and sharpness (interval length).** In this work, authors investigated 3 aspects of method performance: accuracy (measured by MAE), calibration (measured by CI coverage) and sharpness (measured by CI length). However, as we know both intuitively and revealed in classification literature, these 3 aspects are not independent. For example, worse prediction accuracy essentially leads to a more difficulty trade-off between calibration and sharpness, in that the CI length needs to be larger to attain the correct confidence interval. Consequently, to get a deeper understanding of the model performance under distributional shift, it might be interesting to conduct some additional accuracy v.s. uncertainty studies to plot the pairwise association between the three metrics. \n\n* **Study different calibration performance at different $\\alpha$ levels**. In this work, the authors only investigated the coverage probability of 90% confidence intervals, which may not fully capture the method performance at different confidence levels. I would encourage authors to consider adding some additional results for other commonly-considered confidence level (e.g., 95%), or specify a calibration metric that measure the difference between nominal v.s. empirical coverage across a range of confidence levels. Even if all methods do not perform well at some levels (e.g., 95%), it is still a valuable message that is suitable for TMLR and worthwhile to inform the community.\n\n* **Study performance deterioration at different degrees of distributional shift**. To fully illustrate the method behavior under distributional shift, it is a standard practice of this literature (e.g., [Ovadia et al (2019)](https://proceedings.neurips.cc/paper/2019/file/8558cb408c1d76621371888657d2eb1d-Paper.pdf) and [Krishnan and Tickoo (2020)](https://proceedings.neurips.cc/paper/2020/hash/d3d9446802a44259755d38e6d163e820-Abstract.html)) to study method performance under controlled test sets of different degrees of distributional shift. Can authors consider adding at least one additional study to illustrate this? It should be easily doable for the synthetic Cells and Chair data. For real-world shifts, you can consider e.g., those remote sensing datasets by creating test sets with different degrees of geographical affinity with respect to training data.\n\nMinor:\n\n* Section 3.2. \"Secondary Metrics\".  If my understanding is correct, the secondary metrics authors proposed (interval length and prediction rate) are measuring the \"sharpness\" of the probabilistic predictions. Consequently, the evaluation framework authors proposed echoes the conceptual framework is put forward by Gneiting, Balabdaoui, and Raftery (2007, which authors already cited) \"optimizing sharpness while controlling for calibration\". It might be worthwhile to point out this connection by adding a finishing comment at the end of Section 3.2.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "MzlvwotAmU", "reviewer": "Reviewer_d2Ld", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n- Plenty of benchmark datasets can be used to facilitate the research on the uncertainty of OOD regression tasks.\n\n\nWeakness:\n\n- The metrics are confusing. Some plots compare test coverage and interval length, while some others compare prediction rates. It is unclear the motivation for using different metrics.\n\n- The comparison of test coverage under different prediction rates is tricky. It is claimed in the paper that Gaussian + Selective GMM is well-calibrated in all synthetic datasets. However, in fact, the prediction rate is quite low. It doesn't mean that it can perform well even on synthetic datasets.\n\n- There are few insights or analyses on why a certain type of distributional shift can be solved by some assumptions. More analyses on this perspective can make the benchmark datasets more representative. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes several benchmark datasets to study a key research problem of quantifying the uncertainty of out-of-domain data in regression tasks. Some baseline methods are applied, but none of them consistently achieves good performance on both synthetic and real-world datasets. ", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "I suggest the authors to better explain the evaluation metrics and conduct the experiment more systematically. Also, more insights of the distributional shift of each dataset can be interesting to readers. ", "broader_impact_concerns": "No ethical concerns. ", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents a benchmark of vision-based regression problems, with a focus on evaluating uncertainty estimation under data distribution shift.  There has been extensive work on this in the classification setting, but not as much in the regression setting, so it explores a useful void in the literature.  The initial reviews were leaning towards reject, with reviewers appreciating the contribution of the benchmark but were left wanting more in terms of analysis, discussion of related work and overall claims / contribution.  In their response, the authors have added more experiments and presented some useful and interesting take-away analysis of their results. Two of three reviewers recommended leaning reject and one leaning accept.  Also two of three found the claims were not supported by evidence.  However, one reviewer felt that the paper including the benchmarks and analysis were a useful contribution to the community.  In my own reading of the work, and in discussion with the reviewer, I tend to agree with that reviewer.   The additions to the paper after the author response period seem to help strengthen the message and the author response seems to help.  I found the take-away messages interesting and useful. Although maybe it's expected, I found it interesting to observe that conformal prediction performed very well in some scenarios but also so poorly under data distribution shift.  

Overall, there doesn't seem to be anything specifically problematic or wrong about the paper and it presents an interesting and useful contribution to the community.  Therefore I would recommend an accept but request that the authors take some time to carefully prepare the camera ready, addressing the reviewer comments.  A common criticism from the reviewers was that some of the details that should be in the main text were lost somewhat in the appendix.  Also the authors have promised to add additional experiments including at least one where the amount of distribution shift is carefully controlled.  It would be useful if the authors could integrate these into the main text for the camera ready.
Decision: Accept (poster)