Conformal Prediction via Regression-as-Classification
Authors: Etash Kumar Guha
Shlok Natarajan
Thomas Möllenhoff
Mohammad Emtiyaz Khan
Eugene Ndiaye
Venue: ICLR 2024 poster (2024)
Keywords: Conformal Prediction
Probability Function
Regression
Neural Networks
Primary area: probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
TL;DR: We transform regression into classification to make Regression Conformal Prediction more accurate and practical.

Abstract: Conformal prediction (CP) for regression can be challenging, especially when the output distribution is heteroscedastic, multimodal, or skewed. Some of the issues can be addressed by estimating a distribution over the output, but in reality, such approaches can be sensitive to estimation error and yield unstable intervals. Here, we circumvent the challenges by converting regression to a classification problem and then use CP for classification to obtain CP sets for regression. To preserve the ordering of the continuous-output space, we design a new loss function and present necessary modifications to the CP classification techniques. Empirical results on many benchmarks show that this simple approach gives surprisingly good results on many practical problems.

Published as a conference paper at ICLR 2024
CONFORMAL PREDICTION VIA RECRESSION-AS-CLASSCCCCCII
Etash Guha
Shlok Natarajan Salesforce
RIKEN Center for Al Project, SambaNova Systems
(etash.guht) @ssmmanovassstemss com
Thomas Möllenhoff
Mohammad Emtiyaz Khan
RIKEN Center for AI Project
RIKEN Center for Al Project
{thomas www.eeeeooooeeerreee.e..... jp
femtiyaz. khan]@riken, ip
Eugene Ndiaye
Apple {e (e.ndiaye)@apple.com
ABSTRACT
Conformal prediction (CP) for regression can be challenging, especially when the output distribution is heteroscedastic, multimodal, or skewed. Some of the issues can be addressed by estimating distribution over the output, but in reality, such approaches can be sensitive to estimation error and yield unstable intervals. Here, we circumvent the challenges by converting regression and then use CP for classification to obtain CP sets for regression. To preserve the ordering of the continuous-outpuu space, we design new loss function and make necessary modifications to the CP classification techniques. Empirical results on many benchmarks shows that this simple approach gives surprisingly good results many practical problems.
classification problem
INTRODUCTION
Quantifying and estimating the uncertainty of machine-learning models is an important task for many problems, especially mission-critical applications where reliable predictions are required. Conformal Prediction (CP) (Vovk et al.) (005)) has recently gained popularity and has been used successfully in applications such as breast cancer detection (Lambrou et al. 2009), stroke risk pre- diction (Lambrou et al. 2010), and drug discovery (Cortés-Ciriano & Bender] 2020). Under mild conditions, CP techniques aim to construct prediction set that, for given test inputs, is guaran- teed to contain the true (unknown) output with high probability. The set built using a conformity score, which, roughly speaking, indicates the similarity between examples. The conformal set merely gathers examples that have large conformity scores. Despite its popularity, CP for regression can be challenging, especially when the output distribution het- croscedastic, multimodal, or skewed (Lei & Wasserman 2014). The main challenge lies in the design of the conformity score. It is common use simple choice for score functions such dis- tance to mean regressor, but such choices may ignore the subtle features of the shape of the output distribution. For instance, this could lead to symmetric intervals or ignoring the heteroscedasticity. In theory, it is better to estimate the (conditional) distribution over the output, for example, by using kernel density estimation and directly using to build confidence interval. However, such esti- mation approaches are also challenging, and estimates can be sensitive to the choice of kernel and hyperparameters, which can yield unstable results.
test example and the training
We circumvent the challenges by exploiting the existing CP techniques for classification. We pro- ceed by first converting regression to a classification problem and then using CP techniques for classification to obtain conformal set. Regression-as R as-classification approaches popular for var- ious applications in computer vision and have led to more accurate training than only-regressee
training (Stewart et al.
2023 Zhang al. 2016 Fu et al. 2018 Rothe al. 2015
Van Den Oord

---

Published as a conference paper at ICLR 2024
Interval Ranges
Interval
(Ru.)) v.l Covered
(rurt)) Covezed
yt Covervd
(P% st) Not Coveewd
1:
05
14
1s
1a
L0
15
X
(a) Heteroskedastic
(b) Bimodal
Figure 1: We show two examples where the output distribution is heteroskedastic (left) and bimodal (right). In both cases, our method is able to change the interval (shaded gray region) adaptively as the imput values x are increased. Examples outside the gray regions (white dots) are deemed different from those inside it (black dots).
et al. 2016 Diaz & Marathe 2019). We leverage them to construct distribution-basse conformal set that can flexibly capture the shape of the output distribution while preserving the simplicity and efficiency of CP for classification. First, as a distinct class. Second, to preserve the ordering of the continuous output space, we design an alternative loss function that penalizes the density on bins far from the true output value but also fa- cilitates variability by using an entropy regularization. The loss design is similar in spirit to Weigend & Srivastaval [1995); Diaz & Marathe (2019). The resulting method can adapt heteroscedasticity, bimodality, or both in the label distribution. We verify this on synthetic and real datasets where we achieve the shortest intervals compared to other CP baselines. See examples in Figure
discretize the output space into bins, treating each bin
BACKGROUND ON CONFORMAL PREDICTION
Given a new input neww CP techniques aim to construct set that contains the true but unknown output Vnew with high probability. Assuming that a pair of input-output variables (z, y) has a joint density p(x, y) and conditional density p(y z)), oracle prediction sets (with joint and conditional coverage) for the output y can be constructed as
{a E R: p(z,( 2) 2 } or e p(2 x) 2 Ta,zh
(I)
where the thresholds Ta and Ta. are selected to ensure that the corresponding sets have a probability mass that meets exceeds prescribed confidence level - a (0, 1). As the ground-truth distri- bution is unknown, we rely solely on estimating these uncertainty sets using the density estimators p(z, y) and p(y æ). The latter can be inaccurate due numerous sources of errors such as model misspecification, small sample size, high optimization errors during training, and overfitting. Without stronger distribution assumption, the mample guarantee is typically not upheld.
Conformal Prediction has arisen as a method for yielding sets that do hold finite-sample guarantees. Given a partially observed instance (Inew, Vnew) where Vnew is unknown, Conformal Prediction (CP) (Vokk et al. (2055) constructs a set of values that contains Vuew with high probability without knowing the underlying data distribution. Under conformal prediction, this property is guaranteed under the mild assumption that the data satisfies exchangeability. The set is called the conformal set and is built using a conformity score, denoted by a (xx y), which measures how appropriate an output value is for given input example. There many ways to build the conformity score, but they all involve splitting the data into a training set Dur and a calibration set Deal- Often, a prediction model μa (a(x) is built using the training set, and then conformity score is obtained using this model along with the calibration set (we will shortly give an example). The conformal set merely gathers the points with larger conformity scores:
{z E R a (Fnew, 2) 2 Q1-a(Deal))
2

---

Published as a conference paper at ICLR 2024
where QD-a(Dcal) is the (1 a) quanttle of the conformity scores the calibration data. This set provably contains Yuew with probability larger than - a for any finite sample size and without assumption the ground-truth distribution.
There are many design choices for this conformity score. For example, one can choose a prediction model (ar(a)) as estimate of the conditional expectation and measure conformity the absolute value of the residual, i.e., o (z, y) = -ly - Mer (.)]. The corresponding conformal set is a single interval centered around the prediction (ar(2) (2)) and of constant length Q1-o (Paat) for any example Tnow, without taking into account its variability. However, in situations where the underlying data distribution demonstrates skewness or heteroscedasticity, we may desire a more flexible conformity score. We wish to generate an approach to approximate this density function that is versatile in the distributions it can represent and avoids the difficulties seen in prior methods. We explore established density estimation in Classification Conformal Prediction that are already performing effectively.
Since their introduction, lot of work has been done to improve the set of con- formal predictions. As simple score function, distance to conditional mean ie o y)
Related Works
fa(x)) where ((x)) an estimate of Eyyy x)) was prominently used (Papadopoulos et al. 2002 Lei al, 2018 Guha et al. 2023). Instead, (Romano et al., 2019) suggests estimating a conditional quantile instead and a conformity score function based on the distance from a trained quantile regressor, i.c.
(ar y) = max ( - -y.--11-o/// /11-//(11)) where (4a, are the a-th quantile regressors. Within the
literature, our strategy is more closely related to the distribution-base
methods (Lei et al.) 2013).
Following similar line of work (Chernozhukov et al., [02T] argued that the conformal quantile regression score function might be less adaptive since the distance of the quantile behaves similarly to the distance of the mean estimate. Instead, they suggested estimating the cumulative (conditional) distribution function and directly outputting {v F(x, 3) E [a/2, 1-a/2)] However, their method cannot account for bimodality since it can only output single interval by design. An equally in- teresting approach for distribution-
prediction sets is based on learning Bayesian estimator, which, however, may require a well-specified prior and can be computationally expensive (Fong & Holmes 202Ip. Variants based on estimating the conditional density of the response using histogram regressors (Sesia & Romano 2021) could detect a possible asymmetry of the ground-truth distribu- tion. However, our densities estimated through classification techniques, whereas their densities are learned through a histogram of many regressors. Smoothness over the distribution is encoded in our loss function, whereas they use a post hoc subinterval finding algorithm through linear program- ming to prevent many disjoint intervals. Our techniques have some overlap with ordinal regression and ordinal classification Xu et al. (2023) discusses various risk categories (similar coverage) for ordinal classification, while our work considers different score functions where coverage serves as the loss function. Lu et al. (2022) discuss the adaptation of APS style score functions to accommo- date the ordinal structure of classes, which was subsequently utilised in (Sesia & Romano 2021).
3 CP VIA REGRESSION
CLASSIFICATION CEAA
3.1 CLASSIFICATION CONFORMAL PREDICTION
We aim to compute conformity function that accurately predicts the appropriateness of label for a specific data point. Given that the distribution function across labels can adopt diverse forms, such being bimodal, heavy-tailed, or heteroscedastic, our approach must effectively factor in such shapes while upholding its coverage precision. A frequently employed technique involves using the conditional label density as conformity function, leading to reliable results in the classification context. Typically, practitioners perform conformal prediction for classification with probability estimates from a Softmax neural network that covers K output logits using cross-entropy loss. Let us denote the parametrized density
exp(v;)
(- x) = softmax(fo(x)), where softmax(v), (v); Ek exp(vx)
as the outputted discrete probability distribution over the labels of the input x. Traditionally, we fit our neural network by minimizing the cross-entropy loss on the training set:
E arg min
(L ll q0 xx)).
3

---

Published as a conference paper at ICLR 2024
Here, 6y. is the Dirac Distribution with all of the probabilitistic mass y/- Let assume that have trained and acquired f that has minimized the traditional cross-entropy loss function on the training dataset. A natural conformity score is simply the probability of a label according to the learned conditional distribution, i.e., oxx v) = gg(y x).. This approach both straightforward and efficient. The neural network's flexibility allows it to learn numerous label distributions with adaptivity across examples with less explicit design of specific prior structure.
3.2 REGRESSION TO CLASSIFICATION APPROACH
Naturally, we strive to embody such practicality and effectiveness in regression settings. However, the distribution of labels in the regression scenario is continuous, and learning continuous distribu- tion directly using a neural network is challenging (Rothfuss et al. 2019).. Often, Bayesian or kernel density estimators are employed to estimate this distribution. Other techniques acquire knowledge of this distribution by training numerous regressors and categorizing the regressors, for example, Conformal Histogram Regression (Sesia & Romano 2021).
However, these methods look different from the conformal prediction approaches for classifica- tion. It would be desirable to be able to use similar methods for both classification and regression conformal prediction. One method of unifying classification and regression problems outside the conformal prediction literature is known as Regression-as -aaaaaassaaaaaaaiinn We simply turn a regres- sion problem into a classification problem by binning the range space. Specifically, we generate K bins with K equally spaced numbers covering the interval y = [ymmin: Ymma]], where ymmin (or ymax) is the minimum (or maximum) value of the labels observed in the training set. More explicitly, we define our discretization of the label space as
5 = -i .ū.... where ūK+1 Di + OK 91
K-1 with 51 Umin and ÿK- Vmaxx-
These values ý 5 form the midpoints for each bin of our discretization. Naturally, the kth bin all the labels in the range space y' closest to ýt- Intuitively, we can treat each bin as class. Thus, have turned regression problem into classification problem. This method is simple but has yielded surprising results. Some work has suggested that this form of binning results more stable training (Stewart et al., (023) and gives significantly better results for learning conditional expectations. To unify classification and regression conformal prediction, a simple solution is to employ the Classification Conformal Prediction model with discrete labels ū; arg minge [y: -0) This will aid in training the neural network with modified labels through cross-entropy loss, resulting in discrete distribution of go(- x), as outlined in the previous section.
To compute conformity scores for all labels, employ linear interpolation from the discrete proba- bility function ( x) to generate the continuous distribution {e æ).. Nevertheless, this approach encounters a critical issue when employed for regression.
3.3 DATA FITTING
A critical problem with employing CrossEntropy loss in the classification conformal prediction con- text is that any structural relationships between classes are disregarded. This is not surprising given that in the classification context, structure exists between classes, and each class is independent. Therefore, the CrossEntropy loss does not need to differentiate between whether go allocates proba- ble mass far away close the actual label. Instead, it only incentivizes the allocation of probable mass on the correct label. However, within the regression setting, despite a multitude of labels, the labels adhere an ordinal structure. Thus, to enhance the accuracy of labeling, is imperative to devise a loss function that incentivizes the allocation of probabilistic mass not only to the correct bin but also to the neighboring bins. Formally, given an input and output pair (x. y). our goal is to determine a density estimate qo x)) that assigns low (resp. high) probability to points that are far (resp. close) the true label y. i.e.,
qo ( x;)) high (resp. small) when the loss l() 1R) is small (resp. high).
4

---

Published as a conference paper at ICLR 2024
Hence, natural desideratum for learning the probability density function ge is that their product Ely. V) æ) is small in expectation. We propose to find a distribution de minimizing the loss
K
11 [(( - Σ Cly, (i x))
Minimizing this loss in the space of all possible distributions q(- æ)) is equivalent to minimizing the original loss -), where the minimizing distribution is a Dirac delta og. at the minimizer V. of E(g. -). Therefore. we expect an unregularized version of this loss to share similarities with the typical empirical risk minimization on E -). However, key difference is that when minimizing in restricted family of distributions (for example, those representable by neural network with fixed architecture), the distributional output can represent multi-modal or heavy-tailed label distributions. Minimizing the original loss Eyy -). one would always be confined to a point estimate.
3.4 ENTROPY REGULARIZATION
Although the proposed loss function better encodes the connection between bins, tends towards outputting Dirac distributions, outlined in the previous paragraph. Overconfidence in our neural network is a commonly reported problem in the literature (Wei et al., 2022). Nevertheless, smooth- ness has been traditional requirement in density learning. As such, we rely regularization technique for learning density estimators (Wainwright & Jordan 2008). Given the set of density estimators that match the training label distribution well and put high probability mass on the best bins, we prefer the probability distribution that maximizes the entropy since this intuitively takes fewer assumptions on the data distribution structure. Our choice is based on selecting density estimators that effectively match the distribution of the training labels and assign a higher probabil- ity to the best bins. Formally, we can calculate the entropy of our probability distribution by using the Shannon entropy H of the produced probability distribution q(- ()) as penalty term as follows:
classical entropy-
Hige( (-)e((11))) Σ qo (fx(xx) log qo (ū(())z))
Summary We learn a distribution by minimizing the following expected loss over a given training data Drr {111 Bi)...., ...., (Irer yn } of size ner:
K
((0)
E(u,, (ix) (r (2:)) - (-o xx))),
In particular, we choose E(y,, (&) as Ely,, Uz) 1yA - du l where p > is a hyperparameter. This selection functions natural distance metric that meets all required objectives. As a result, can employ a technique similar to the aforementioned Classification Conformal Prediction methods. Initially, we train Softmax neural network fo with K logits, grounded on the loss function Equa- tion (2) on the training dataset. To calculate conformity scores for the calibration set, we utilize the linearly interpolated (ar, y) = go (y 1:) for specific data point (ar, y). Namely, for any y between ÿN and Wk+1, (y x) is defined as
go x) = 71:00 (fK (x)) + (1 - (0 |x)
Øk+ - y
where 7 Dc+1 Úk
Complete details of this procedure are outlined in Algorithm
4 EXPERIMENTS
All code run our method can be installed via
pip install r2ccp.
We investigate the empirical behavior of our R2CCP diction) method, which have explained in detail in Algorithm [ We have three sets of ex- periments. The first one is described in Section 4.] and presents empirical evidence of the al- gorithm's 's ability produce narrower intervals by utilizing label density characteristics, including
Conformal Pre-
5

---

Published as a conference paper at ICLR 2024
Algorithm Regression to Classication Conformal Prediction (R2CCP).
1: Input:
Dataset D, { yı).. ...., (In, (n)) and new input In+1
Desired confidence level a E (0,1)
2: Hy perparameters:: temperature
0. p 0, number of bins K 3: Discretize the output space [ymin Mmax]] into K equidistant bins with midpoints (ūr..... ........ Dk}
4: Randomly split the dataset D,, in training Drr and calibration Deal
5: Find distribution qg( (-)ar) by (approximately) optimizing on the training set Der
K
i arg min BERU
10:
Ukl"ggo (ūx x:) TH(9o(-1
where go (8k x²) = softmax (fo(x)) for model (e.g., neural net) fo :R TR
6: S [ x) for (z. y) E [eal) # #gg(-12) is linear interpolation of softmax probabilities.
7: (Daal) + quantile(S,
8: refurn Conformal Set [(a) ((0) {= E R ( kat1) (Dea)))
heteroscedasticity, bimodality, or a combination. Section 4.2 demonstrates the effectiveness of our algorithm on synthetic and real data by comparing with various benchmarks from the Conformal Prediction literature in terms of length and coverage. Section 4.3 evaluates the effect of different loss functions on the final learned densities and their impact on the intervals produced. All exper- iments were run over 5 different seeds at coverage level of 90%, and the standard error over the experiments is reported in the subscript. We do not tune the hyperparameters and keep values of K = 50, p 0.5, and length, meaning the length of all the sets predicted, and coverage, the percent of instances where the true label is contained in the predicted intervals.
0.2 constant across all experiments. For all experiments, we report
4.1 SPECIFIC CHARACTERISTICS OF LABEL DISTRIBUTION
Heteroscedascity We generate a toy dataset where the input is one-dimensional. It contains sam- ples from the following distribution: y (O, [a]). The label distribution is heteroscedastic, mean- ing the variance of the labels changes as the input changes. In traditional Conformal Prediction literature, many existing algorithms fail capture heteroscedassicity, resulting wide intervals for inputs æ where the label distribution of has low variance. However, our learned algorithm can directly learn this relation and adjust the outputted probability distribution accordingly. Thus, we see that the lengths of the intervals will increase the variance of the label distribution increases, which is desirable. We see this relation in Figure Ta Moreover, we also use the dataset generated from (Lei & Wasserman 2014) as discussed in Appendix D This dataset exhibits heteroskedascity and bimodality as X passes 0.5. We see that our learned method can adjust intervals accordingly to maintain coverage and length for all X. We plot this in Figure [b] We plot how the intervals (grey) change as the data distribution (black) changes. As the variance of the labels increases as x increases, the produced intervals adaptively get wider, taking advantage of the heteroscedasticity.
Bimodality We showcase our algorithm's 's capability to address labels with a bimodal distribution. Our bimodal dataset is generated by repeatedly (1) sampling two sets of random features that are close geometrically and (2) giving one set of features a label of plus some Gaussian noise and giv- ing the other label of 1 plus some Gaussian noise. Therefore, our dataset comprised of many similar data points with bimodally distributed labels. This bimodal distribution is particularly hard for many existing CP algorithms to solve since it requires outputting two disjoint conformal sets to achieve low length. CQR, for instance, cannot deal with this circumstance and will generate a con- formal set that covers the entire range space. However, our method is flexible enough to assign a low probability to labels between -1 and 1. and our resulting conformal set will not include these inter- mediate labels. We see this in Figure [dd In Figure/od) our outputted probability distribution has two modes around labels and 1 and assigns low probability value the valley between the modes.

---

Published as a conference paper at ICLR 2024
4.2 COMPARISON TO OTHER CONFORMAL PREDICTION ALGORITHMS
The crucial criteria for assessing a Conformal Prediction algorithm consist of (1) coverage, rep- resenting the percentage of generated conformal sets that incorporate accurate labels, and (2) the length of the generated conformal sets. Our baseline techniques consist of the Kernel Density Esti- mator (KDE) as proposed by Lei & Wasserman (2014), alongside a conformity score shown by the estimated probability density. Furthermore, we take into consideration Fong & Holmes (2021) (CB), which employs the likelihood of posterior distribution
their conformity function, and [Sesia & Romano 2 (CHR). which uses quantile regressors on every bin of a histogram density estimator. We have included the Conformal Quantile Regression as described by Romano et al. (2019) (CQR), which employs conformity based on the labels' distance from quantile regressors. Moreover, we had the Distributional Conformal Prediction which uses a Cumulative Distribution Function to form its intervals from Chernozhukoy et al. (202T) (DCP), Additionally, we have used the Lasso Conformal Predictor with a distance to mean regressors, which is the most straightforward option (LASSO). We use synthetic data exhibiting bimodally distributed and log-normally distributed illustrate particu- lar weaknesses of existing methods. We use real datasets also in Romano et al. (2019). Specifically, these are several datasets from the UCI Machine Learning repository (Bio, Blog, Concrete, Com- munity, Energy, Forest, Stock, Cancer, Solar, Parkinsons, Pendulum) (Nottingham et al. 2023) and the medical expenditure panel survey number 19-21 (MEPS-19 (9-9--19-21) (Cohen al. [0099. These re- gression datasets are commonly used to benchmark regression models. Our approach yields tighter intervals on real datasets than some of the strongest baselines.
We report lengths and coverages results in Table I respectively. We added figures depict- ing example probability distributions on these datasets in Figure 6 in Appendix H The intervals produced by our method are the shortest over 10 of the 16 datasets. From Figure 66] our method can learn many different shaped distributions well, which accounts for this significant improvement in intervals. Overall, the Kernel Density Estimation and our method can accurately predict the best intervals on datasets where the connection between the data and feature is simple, such Bimodal or Log-Normal. While the Kernel Density Estimator can fail when the labels complexly related to inputs, no such connection exists in these datasets since the labels are indepen- dent of the features. Thus, the Kernel Density Method's simplicity allows it learn the label density quickly. Our method also seems to handle the case where there is no connection between the data and the labels, as seen in Figure 6d in Appendix H. Moreover, on datasets where the label density is smooth and close to the Laplace prior, such as on Figure 6c and Figure 6il Conformal Bayes and our method can accurately learn this distribution since both methods can output smooth distribu- tions. Moreover, very sharp datasets such as Concrete in Figure oj and Energy in Figure 6h both our method and CQR can capture the sharp and unnoisy distribution needed to achieve strong length. Moreover, on complex distributions such in Pendulum in Figure 6e and Bio in Figure 6f we see that both CHR and our method have the flexibility to portray complex distributions resulting in the most accurate intervals. Thus, the flexibility of our algorithm smoothly learn sharp, wide, complex, and simple conditional label densities results in our method achieving the best length most consistently over the entire dataset.
Results
that
4.3 ABLATION STUDIES
Our loss function consists of an eтror and entropy terms. The error term penalizes distributions that put weight far away from the true label, whereas the entropy term acts as a regularizing term in the probability distribution space. We will do ablations both the error and entropy terms to illustrate the importance of each part. For the error term, there are several notable alternatives that practitioner may use. An alternative is the log maximum likelihood cross-entropy formulations of the error term, which we denote as
CMLE Σ
log go (ū: x;))).
where we remind the reader that arg mingey |0 we. We note that both MLE and CE are equivalent formulations in this setting. These two are standard error terms often used in practice. We will train our models with this loss function over all datasets to see how the change of error term affects the intervals' length and the learned density. We will show that our chosen error term is better
7

---

Published as a conference paper at ICLR 2024
DNTASET
BIMODAL
LOGNORM
CONCRETE
MEPS-19
MEPS-20
MEPS-21
BIO
COMMUNITY
CQR
2.14
(.38
(.87 (.39
2.00
1.99ça.03)
1.34
1.44 os)
KDE
0.35
1.40 04)
1.54
2. 2.10;o 02)
2.51
2. 2.39ço oa)
2.27 00;
2. 2.23
LASSO
2.14(0.00)
3.30 09)
2.74(0.03)
4.64(0.06)
4.79 04)
4.92;o.04)
3.89
3.26ro.05;
CB
2.16(0.00)
1.45 o1)
0.96
4.47;o 02)
4.50;a o2)
4. 013
2.09;o o0;
1.80,0.00;
CHR
2.14(0.00)
1.52(0.05)
0.47
2.60ro 08)
2.53 oa)
2.75 0a)
1.59
1.49 06)
DCP
(..1)
1.74(0.0r) 07)
0.47 01)
68 18)
06.7 (0.71ca.m 10)
67. 57.500c an)
1.74
1.59
R2CCP (OURS)
0.46(0.01)
1.96(0.03)
0.38
1.60
1.72 1.70
oa)
1.47 (.1
DNTASET
DIABUTES
SOLAR
PARKINSONS STOCK
CANCER
PUNDULUM
ENERGY
FOREST
CQR
(..0)
(.
(o.2(o.01)
1.85ço 25)
(.09(
2.25 313
0.19(a.01)
3.18 19;
KDE
(.34
0.50 01)
3.79
4.72:o 29)
3.82
3.96ço.00)
2.72(0.07)
2.90 17)
LASSO
(0.04)
3.54().12)
3.40 os)
1.30ça 04)
3.56;a.14)
3.99ça.07
(.29
(o.7
CB
1.19(0.01)
3.78(0.02) 02)
3.42 01)
1.32(0 01)
3.71;o 3.14(0.04)
oa)
1.26 o1:
3.75
CHR
(.40)).02)
1.49
0.68 02)
1.50
1.69 3.4
0.23
3.0
DCP
(0.02)
0.83 (5.69(0.00)
04)
1.0 10)
3.07.c.07)
1.76ço 10;
(.
6.
R2CCP (OURS)
1.34 (0.4)
(.80
0.50 ou)
0.92
3.21 (0.08)
(.60
(.20
3.80
Table I: This is the length results over all datasets. We see that our method achieves the best length on 10 of the 16 datasets. Meanwhile, CQR is best at 5, CHR is best at 3, CB is best at 1, and KDE is the best at 3. Our method achieves the shortest intervals across these datasets.
Oly 1
Glv (+.11)
0.10
[(y) [1
0.06
Confidonna T.rvel
y
(a) Original
(b) No Entropy
Ground Truth
Ground Truth
ciyl
0.06
civ Fu.1))
0.05
0.8
[(y) 22
0.00
0.03
Confiddenc
0.82
Faroul
0.
(c) MLE
(d) MIE with Entropy
The resulting density estimates with different loss functions. We see that that removing entropy from our loss function using MLE as error terms causes sharp density estimates. Moreover, adding in the entropy regularization with MLE does not smooth the density estimate but instead raises the entire distribution uniformly; which does not provide valuable information for CP.
Figure 2:
for producing optimal intervals empirically. Moreover, we will also demonstrate the importance of our entropy term. We do this by retraining our models with the entropy part omitted. We also test by combining the MLE error term with entropy regularization. Therefore, we will perform 4 different
8

---

Published as a conference paper at ICLR 2024
DNIASET
BIMODAL
LoG-
CONCRETE
MEPS-19
MEPS-20
MEPS-21
B10
COMMUNITY
NORMAL
ONE
0. 0.43p.001
3. 3.250.058
1. 1899.037
3. 3.730.057
3. 3.79u.081
8. 8.788. 18s
2.01a.00e
3.6-l.005
CMLE
0.35o. 001
1.760.02 020
0. 0.740.016
1.58o.00 oos
1.590.000 000
1.710.084
2.71o.00 001
2.32a.047
EMEE+
0.366,001
3.520.008
1.910.017
1. 1.60g.um
1.630.027
1.700.000
2.32a.
3.77a.013
0.44p.002
1882p.051
0.37a ops
1.60g nos
1.599.011
1.69g.013
1. 1000.004
1.50p.o25
DNTASET
DIABETES
SOLAR
PARKINSONS
STOCK
CANCER
PENDULUM
ENERGY
FOREST
ENE
1.92g ona
2.20
4.760.o3x o3a
10.17o.103
3. 3.26o 207
12.82o 479
3.30o 093
3.33a 254
CMLE
1.56g.081
0.14
0.3 one
4. 4.480.214
3,260. 117
3,000, 203
0.250.011
3.0
E
1.96p.012
0.150.005
0. 001
9.730.004
3.950.022
13. 13.400. 221
0.25a.n09
5.24a.055
C
1.370.037
0.66p.ow2
0.s
1.95g.039 035
3.04
1.22.002
0 0.21p.025
2.920.127
Table 2: We present the length results over all of the variant loss functions. We find that our loss function delivers the best over 12 datasets, demonstrating that our chosen loss function often gen- erates the best intervals. For datasets where our method does not deliver the best results, is likely that tuning the weight on the entropy and the smoothing term p would likely have improved the results, but we do not do this for the sake evaluation.
ablation experiments on the loss function by retraining on different variations of the loss functions and reporting the final length and coverage generated. Overall, the four loss functions we will use are the original loss function L. the original loss function without entropy LNF. the MLE loss function LMLE. and the MIEE loss function with entropy added LMLE+ We will demonstrate that our chosen loss function delivers the best results across all the loss functions. We report our results in Table [
Results We see that our chosen loss function achieves the best length across most loss functions in T The only datasets where our chosen loss function does not achieve the best length are Energy, Solar, Bimodal, and Log-Normal. We now discuss the individual differences between our loss function and each variant loss function. When comparing our loss L with the variant without entropy LN,, we see the lengths constantly increase except for the bimodal distribution. Without an entropy regularizing term, the outputted probability distributions are not smooth. placing much mass on single data point. This overconfidence results in the oth quantile of the low probability of the true label as in Figure 2
When looking at the differences between our loss function when changing the error term to Maxi- mum Log Likelihood, we see similar overconfidence. While MLE loss works on several datasets, such as Energy. Solar, Bimodal, and Log-Normal, it also performs poorly on other datasets. MLE works well for datasets with less simple label distributions but fails otherwise due to similar over- confidence. Even when adding entropy regularizer to the MLE loss as in LMLE E. we that the addition of entropy does not improve the length. Since MLE loss does not treat nearby bins differently than far away bins, regularization decreases the overconfidence but does so uniformly across all bins. The result is a roughly uniform distribution with a single spike. This does not utilize the structure of regression where bins near the best bin are preferable. Thus, prioritizing nearby bins is crucial for achieving strong length. Moreover, when adding entropy regularization to our error term, we do not see uniform increase in probability across all bins but instead smoothening of direct. This connection between entropy regularization and our to be a powerful synergy per the intervals produced. Thus, the error term utilizing the regression structure by prioritizing nearby bins and the regularizing entropy term preventing overconfidence seem crucial for producing good intervals.
loss function appears
Another important insight is that the datasets where other loss functions excel are the same ones where other CP methods perform better than ours. In particular, on the Bimodal, Log-Normal, Parkinson's. ss and Solar datasets, our CP method is not superior, and other loss functions outperform it. This indicates that the MLE's sharp distributions result in greater accuracy on these specific datasets. Hence, adjusting the weighting terms for entropy and smoothing p could enhance the smoothness of the learned distribution and, thereby, the performance of our algorithm across these datasets, leading to the best outcomes. Nevertheless, to maintain faimesss, we avoid this approach.

---

Published as a conference paper at ICLR 2024
REFERENCES
Victor Chernozhukov, Kaspar Wüthrich, and Yinchu Zhu. Distributional conformal prediction. Proceedings of the National Academy of Sciences, 1 118(48)c21007794118 2021.
Joel W Cohen, Steven Cohen, and Jessica Banthin. The medical expenditure panel survey: national information resource to support healthcare cost research and inform policy and practice. Medical Care, pp. 44-50, 2009.
Isidro Cirrés-Crriano and Andreas Bender. Concepts and applications of conformal prediction in computational drug discovery. Artificial Intelligence in Drug Discovery, pp. 101, 2020.
Raul Diaz and Amit Marathe. Soft labels for ordinal regression. In IEEE Conference on Computer
Vision and Pattern Recognition (CVPR), 2019.
Edwin Fong and Chris C Holmes. Conformal Bayesian computation. In Advances in Neural Information Processing Systems (NeurIPS), 2021.
Huan Fu, Mingming Gong, Chaohui Wang, Kayhan Batmanghelich, and Dacheng Tao. Deep ordinal regression network for monocular depth estimation. In IEEE Conference Computer Vision and Pattern Recognition (CVPR), 2018.
Subhankar Ghosh, Taha Belkhouja, Yan Yan, and Janardhan Rao Doppa. Improving uncertainty quantification of deep classifiers via neighborhood conformal prediction: Novel algorithm and theoretical analysis. arXiv preprint arXiv:2303, 10694, 2023.
Etash Kumar Guha, Eugene Ndiaye, and Xiaoming Huo. Conformalization of sparse generalized
linear models. In International Conference on Machine Learning (ICML., 2023. URL. https:
//proceedings mlr wtwppppppww.....e////tt.// 2ttp///2222gguua2222 guha23b. html
Alex Krizhevsky, Ila Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convo- lutional neural networks. Advances in neural information processing systems, 25, 2012.
Antonis Lambrou, Harris Papadopoulos, and Alex Gammerman. Evolutionary conformal prediction for breast cancer diagnosis. In 2009 9th international conference on information technology and applications in biomedicine, pp. 1-4. IEEE, 2009.
Antonis Lambrou, Harris Papadopoulos, Efthyvoulos Kyriacou, Constantinos S Pattichis, Marios S Pattichis, Alexander Gammerman, and Andrew Nicolaides. Assessment of stroke risk based on morphological ultrasound image analysis with conformal prediction. In Artificial Intelligence Applications and Innovations: 6th IFIP WG 12.5 International Conference, AIAI 2010, Larnaca, Cyprus, October 6-7, 2010. Proceedings 6, pp. 146-153. Springer, 2010.
Jing Lei and Larry Wasserman. Distribution- prediction bands for non-parametric regression. Journal of the Royal Statistical Society Series B: Statistical Methodology, 76(1):71-96, 2014.
Jing Lei, James Robins, and Larry Wasserman. Distribution- prediction sets. Journal of the American Statistical Association, 108(501):278-22 287, 2013.
Jing Lei, Max Selll, Alessandro Rinaldo, Ryan J Tibshirani, and Larry Wasserman. Distribution- free predictive inference for regression. Journal of the American Statistical Association, 113 (523):199411111 2018
Zhen Lin, Shubhendu Trivedi, and Jimeng Sun. Locally valid and discriminative prediction intervals for deep learning models. Advances Neural Information Processing Systems, 34:8378-8391, 2021.
Charles Lu, Anastasios N Angelopoulos, and Stuart Pomerantz. Improving trustworthiness of ai disease severity rating in medical imaging with ordinal conformal prediction sets. In International Conference on Medical Image Computing and Computer- Assisted Intervention, pp. 545-554. Springer, 2022.
Eugene Ndiaye. Stable conformal prediction sets. International Conference on Machine Learning, pp. 16462-16479. PMLR, 2022.
10

---

Published as a conference paper at ICLR 2024
Eugene Ndiaye and Ichiro Takeuchi. Root-finding approaches for computing conformal prediction set. Machine Learning, 112(1):151-176. 2023.
Kolby Nottingham, Rachel Longjohn, and Markelle Kelly, Machine Learning Repository, 2023.
URL httpps://archiieeicss.s.. www.::iii....daaaaeette.ae...e.e Accessed: September, 2023.
Harris Papadopoulos, Kostas Proedrou, Volodya Vovk, and Alex Gammerman. Inductive confidence
machines for regression. In Machine Learning: ECML 2002: 13th European Conference on Machine Learning Helsinki, Finland, August 19 23, 2002 Proceedings 13, pp. 345-356- Springer,
2002.
Y. Romano, E. Patterson, and E. J. Candes. Conformalized quantile regression. In Advances in Neural Information Processing Systems (NeurIPS). 2019.
Rasmus Rothe, Radu Timofte, and Luc Van Gool. Dex: Deep expectation of apparent age from single image. In IEEE International Conference on Computer Vision Workshops (ICCVW), 2015.
Jonas Rothfuss, Fabio Ferreira, Simon Walther, and Maxim Ulrich. Conditional density estimation with neural networks: Best practices and benchmarks. arXiv preprint arXiv: 1etl::190..000 903.00954, 2019.
Matteo Sesia and Yaniv Romano. Conformal prediction using conditional histograms. In Advances in Neural Information Processing Systems (NeurIPS). 2021.
Lawrence Stewart, Francis Bach, Quentin Berthet, and Jean-Philippe Vert. Regression as classifi- cation: Influence of task formulation neural network features. In International Conference on Artificial Intelligence and Statistics (AISTATS), 2023.
Aäron Van Den Oord, Nal Kalchbrenner, and Koray Kavukcuoglu. Pixel recurrent neural networks. In International Conference on Machine Learning (ICML), 2016.
V. Vovk, A. Gammerman, and G. Shafer. Algorithmic learning in a random world. Springer, 2005.
Martin Wainwright and Michael Jordan. Graphical models, exponential families, and variational inference. Foundations and Trends@ in Machine Learning, 1(1-2):1-300 2008
Hongxin Wei, Renchunzi Xie, Hao Cheng, Lei Feng, An, and Yixuan Li. Mitigating neural net- work overconfidence with logit normalization. In International Conference on Machine Learning, pp. 23631-23644, PMLR, 2022.
Andreas S Weigend and Ashok N Srivastava. Predicting conditional probability distributions: A connectionist approach. International Journal of Neural Systems, 6(02):109-11 118, 1995.
Yunpeng Xu, Wenge Guo, and Zhi Wei. Conformal risk control for ordinal classification. In
Uncertainty in Artificial Intelligence, pp. 2346-2355. PMLR, 2023.
Richard Zhang, Phillip Isola. and Alexei A Efros. Colorful image colorization. In European
Conference on Computer Vision (ECCV), 2016.

---

Published as a conference paper at ICLR 2024
A AUTHOR CONTRIBUTION STATEMENT
Etash Guha, Shlok Natarajan, Thomas Möllenhoff, Eugene Ndiaye, and Mohammad Emtiyaz Khan all responsible for the development of the main idea and writing of the paper. Etash Guha and Shlok Natarjan were responsible for the experiments and implementation of the algorithm. Etash Guha, Thomas Möllenhoff, and Eugene Ndiaye were responsible for the design of the loss function, binning, and entropic regularization. Etash Guha and Eugene Ndiaye were responsible for writing the related works.
B LIMITATIONS
There several limitations to algorithm. The loss function we choose to optimize needs larger representational power due to increased complexity of the output. Therefore, the neural networks used in practice to minimize this loss function are larger than the ones used for CQR. Moreover, to achieve strong training, we had to slightly vary the hyperparameters for a dataset depending on its size. For example, larger datasets needed models be trained with smaller initial learning rate to avoid divergence. This was not needed for Quantie Regressors, If the network does not train well or achieve good training validation loss, the intervals produced will be suboptimal, training setup that minimizes the loss effectively is crucial for our algorithm to produce meaningful intervals. Moreover, we do not directly learn the label distribution and our produced probability distributions do not directly mirror that of the ground truth distribution. One reason for this is the use of entropy regularization. While using entropy regularization introduces bias, we found to be necessary to prevent the neural network from being overconfident on simple bin in practice.
finding
C MORE DETAILS ON CONFORMAL PREDICTION AND VARIANTS
Given an input observation without its label y, Conformal Prediction method Vovk et al. (2005) allows select set of values that its unobserved response y are likely take. For an arbitrary value 2, the rational is to score how the example (I is typical/conformal observation given
a sequence of iid data D, {(E,, yi) he(l) Intuitively, the scoring function, that we denote Ø, measures how different is the new example compared to examples that we have already observed. We compute the score function (x., y) for every example in the augmented dataset Dn+1(2) = D, U (Ir, z) and report the fraction of time previous example score better than the new one.
This define the so called conformity function T--) E [0. 1] as
Rank(2) where Rank(z)
. (z) = 1
Σ (#+)))
Ho(o, S 2))+11)
(3)
+1
If (22) is small (resp. large) then is non conformal (resp. conformal). Therefore, for threshold level E (0, 1), simple way to find our conformal set [(a) for the response Un+t1 of the input 2n+1 is to select all the values 2 whose conformity exceed the threshold i.e.
( (In+1) {2 R x(2) 2 a).
The validity of the method rely on the fact that if we assume that order in which the observations arrive is irrelevant and that the score function is invariant wIt permutation of the data, then the sequence of scores {a vi)....,o of yn a (Ix+1, bn+1)} are also equally likely and thus the Rank(ya+1) is uniformly distributed in f1,.... n
n+11. 1}. As such, for any confidence level (0, 1),
P(yn41 E (r())( 2 (((())+1) 2 a:) ((Rank().+1) s (n + 1)(1 a)) 2
Conditional validity The previous coverage guarantee may not always be ideal because it is not conditional on the particular input Tn+1- Unfortunately, it is not possible to have such conditional validity when the distribution of the inputs is continuous. Nevertheless, it can be approximated using some localisation strategies based on a weighted conformity function; see (Lin et al., 2021 Ghosh et al. 2023). This also allows further reduce the size of the conformal prediction sets.
Validity can be conditional on a discrete attribute of the input, In+1. through the use Mondrian Conformal Prediction. This method involves slight modification of the conformity function. Let
12

---

Published as a conference paper at ICLR 2024
taxonomy be function that partitions the data into finite number of categories eg k(x,) is a classification of the example z. This allows to modify the pi-value function in order to achieve validity conditional on the category of the test example:
1o(x,)) S (an+1,, 2), k(z,)) 3) x(zn+1,2)) (c.))E ((x))ED [x((, 9) = k((x++122)
T(2)=
E
Computational issues As mentioned in the introduction, the selection the score function affects the size and shape of the prediction sets. There are also computational issues that arise when the full data set is used to fit the function. For example, consider using the estimated distance to the
conditional mean (x. y) = ly E [z]] The estimator E[-[z] [-]] learned on the extended dataset (=) for all possible values z, which is computationally infeasible for most methods without stronger assumptions; see (Ndiaye & Takeuchi, (023), (Ndiaye) (022). As such, most of the popular
strategies are based on the fitting of the prediction model to an left-out independent set of data, as we did in this paper.
D DETAILS ON DATASET FROM LEI WASSERMAN (2014)
For the dataset referenced in Figure lb from the following distribution.
generate the dataset by sampling many (X, Y) pairs
X ~ Unif[-1.5, 1.5]
Y X / ((f(X) 1000gg(()) g(X), 1 +(xx) -
f(x) -x + +1), s(X) 4V(( + +1/2)((X>> 2 -1/2), o²(X) 1/4+
This dataset demonstrates bimodality after o passes the threshold of -0.5. This dataset was similarly used by the Lei & Wasserman (2044)
E COVERAGE DATA
We have presented the coverage data the Conformal Prediction comparison as well as the ablation experiments Table 3 and Table A respectively. Since all methods obey the classical Conformal Prediction framework, expect coverage at the guaranteed level of 1- This guarantee is indeed what we see across all experiments. This confirms that our coverage guarantee indeed holds in practice.
DATASET
RIMODAL.
LOGNORM
CONCRETE
MEPS- 19
MEPS-20
MEPS-21
BIo
COMMUNITY
(p.0)
0.91 (0)
0.91 o0)
0.90 oo)
0.90 00)
0.90ça.00
0.90
0.90
KDE
0.93(p.00)
0.90 oo)
0.90 oo)
0.91 0o)
0.92ço 00)
0.91ço o0;
0.90;o o0:
0090
LASSO
(.90
0.91(p.0 (0.00)
0.90
0.90ça 0o)
0.90
0.90
0.90
0.90(o.00;
CB
pp90
0.91 (o (1)01)
0.88 01)
0.90 oo)
0.90 00)
0.89 o03
0.90 o0:
0.90
CHR
0.90)).oo)
0.91 (0.00)
0.91
0. 0.90 no)
0.90
D. 0.90,o.00
0.90
0. 0.90ço.00:
pp.0)
(p (.91
0.91 (0)00)
1.00 oo)
1.00 oo)
1.00ça.00
1.00
0.90
R2CCP (OURS)
0.90
0.90 oo)
0.90 003
0.90 003
0.90 003
0.90 003
0.90 o0:
0.90;o.00:
PARKINSONS STOCK
CANCER
PENDULUM
ENERGY
FOREST
(p.00)
0.91 oo)
0.90 oo)
0.91ga oo)
0.92 00)
0.91ço.00:
0.90
(o.00)
KDE
0.91
0.71
0.89 00)
0.90 00)
0.92 01)
0.90 01)
0.91,o. o0:
0.81ço.08;
LASSO
0.91 (p.00
0.90
0.90
0.91gg.00)
0.92
0.90ga.00
0.90ço.00;
0.90(o.00:
(p89
0.92 o1)
0.90 oo)
0.88çq.03)
0.87 02)
0.88ço.01)
0.89(o.01)
0,90
CHR
0.91 (p.0)).00)
0.91
0.90
0.91.a no)
0.92
0.91.o.00;
0.91.o.
0.91 (.91
DCP
0.91 (0.00)
1.00rp.oo)
0.90 oo)
0.91 oo)
0.92 oo)
0. ao:
0.91(o.00:
1.00(o.00;
R2CCP (OURS)
0.90
0.92
0.96 000
0.92
0.90 000
0.90)oooo)
0.91ço.o1:
0.89
Table 3: This is the coverage data all datasets.
We that all methods achieve the roughly
- a coverage guaranteed by conformal prediction.
13

---

Published as a conference paper at ICLR 2024
DATASET
BIMODAL
LOGNORM
CONCRETE
MEPS-19
MEPS-20
MEPS-21
BIo
COMMUNITY
C
0.90 02)
(..0))
0.90ço.02)
(.00(..o1)
(o.0
0.90ço.013
0.90
0.90
LNE
0.90
0.81
0,84ço oa)
0.90
0.90
0.90ço.o0 o1)
0,68 01)
0.88
CMLE
0.90 02)
(0.00)
0.90ça.02)
(o 01)
(..03
0.90ça.03) 053
(o.0(0.00)
(o.0
CMER-R
0. 02)
(..0))
0.90ço 02)
0.90 01)
0.90
0.90ço 01)
0.92(a.00)
0.90
DIABETES
PARKINSONS
STOCK
CanceR
PENDULUM
ENERGY
FOREST
C
0.90 os)
(0.0))
0.95ço.as>
(.90().0s)
0.90
0, 0.90ça.om 03)
(o90().0s)
0.89
ENE
0.94 02)
(0.00)
0.73;o.013
0.89
0.90
0.90ço. 08)
0.90
0.77(0.04)
CMLE
0.90 03)
0. (o.02)
0, 0190
(.90
0.90to.05)
0. 0.90ço.om
(a90
0.89to.0m)
CML.-n
(..0().03)
(0.02)
0.90
(.90
(c90().00)
0.90ço.om 08)
(.90(a.03)
0.90(o.08)
Table 4: This is the coverage data for different loss functions from the ablation.
F MORE EXPERIMENTAL. DETAILS
In order to maintain fairess across all baselines, we use the same size neural network for our method across all experiments. Specifically, we discretize the range space into K 50 points, weight the entropy term by = 0.2, use a 1000 hidden dimension, use 4 layers, use weight decay of le 4, use p .5, and use AdamW learning rate le - 4 and batch size 32. However, for certain datasets, namely the MEPS datasets, we used a larger batch size of 256 to improve training time and used smaller learning rate to prevent training divergence. We did change any other parameter between all of our runs. For the baselines of CQR and CHR, we the neural network configurations mentioned in the paper. We found that the parameterizations mentioned by the authors in the papers achieved the best performance and changing the parameterizations weakened their results.
optimizer. For most of the experiments, we use
G ADDITIONAL EXPERIMENTS
We add several additional experiments help contextualize our results. First, we add experiments tracking how our method performs over different coverage levels relative to CHR and CQR across several datasets in Appendix G.2 Second, Appendix G.3 we analyze what percentage of the time the predicted intervals are singeltons, meaning there is only one continuous interval. Third, we analyze the correlation between the residual prediction error and the predicted interval's length with our method in Appendix G.4
G.1 PERFORMANCE WITH DIFFERENT NUMBER OF BINS
We the number of bins constant K 50. natural question is how the number of bins affects the performance of our conformal prediction methodology. To test this, we retrain several models with varying number of bins on the Concrete dataset. We report the size of the predicted intervals for each number of bins in Figure 3 We observe that the number of bins increases, the average length of the predicted intervals decreases and stabilizes. This suggests that after a sufficient number of bins, the length of the intervals becomes relatively constant. In practice, classification problems with 1000 bins or more have been routinely solved since 2012 (Krizhevsky et al. 2012)). This suggests that the increasing number of bins does not make the learning task more difficult, explaining how performance remains relatively unaffected as the number of bins is increased.
G.2 DIFFERENT COVERAGE LEVELS
While the coverage level of .1 is relatively standard across the Conformal Prediction literature, we want to see how our method compares to other methods across different coverage levels. We take the trained models from the original experiments and compute the intervals across different coverage levels for our method, CHR, and CQR. As expected, for all methods, the length of the predicted intervals increases as the required coverage level increases. We report our results in Figure 4
14

---

Published as a conference paper at ICLR 2024
0.42
Length
0.40
0.38
20
40
60
100
Number of Bins
Figure We present an ablation of how the number of bins affects the average length of the generated intervals.
G.3 NUMBER OF SINGLETONS
We also provide analysis of the percentage of times that predicted intervals singleton, meaning only one interval is predicted. We present our results in Table We see that for a majority of the datasets, our model tends predict singleton intervals. However, for both the Bimodal dataset and the Bio dataset, we see that our method predicts far less singletons. This observation is intuitive for the Bimodal dataset as two intervals are expected account for the different modes.
G.4 CORRELATION BETWEEN PREDICTION ERROR AND LENGTH OF PREDICTED INTERVALS
We want to how the prediction prediction error as the absolute error residual between predicted label and the true label. Given our model is lored generated intervals, there are many ways to predict single label for given of features. For example, we can select the bin with the largest probability predicted by our model
correlates with the predicted interval lengths. We define
j = dt where & = arg maxqe (Vv(
kE[K]
We can also use the expected label according distribution the predicted label
gon (z)).
We choose the second method for the sake of these experiments. We compute the absolute residual emor as - - y||- We plot our results Figure(3) We see that the prediction error not strongly correlated with the length of the predicted interval.
H OUTPUT DISTRIBUTION
Here, plot the distribution of lengths of all Conformal Prediction methods over all datasets. We also plot example density functions learned by out method, CHR, and KDE on all datasets. We note that KDE's learned densities from CHR are noisy and not smooth. Moreover, we plot several examples of only our learned probability distribution. We use this when referencing the experiments to demonstrate the different shapes of label distributions from the data.
be relatively less informative. Moreover, the learned densities
15

---

Published as a conference paper at ICLR 2024
1.5
COR
COR CHR
1.5
DOR
Longth
CHR
Length
1.5
Length
3,0
CHIR
0.5
0.0
0.0
0.0
0.2
0.4
0.6
0.8
0.2
0.4 Coverage Lovel
0.6
0.9
0.2
0.4
0.6
0.8
Coverage Level
Coverage Level
(a) MEPS-20 20
(b) MEPS-21
(c) DIABETES
1.5
CQR
Ours
1.5
CQR
Ours
Lengtt
Length
Ours CHR
0.5
Coverage Level
Coverage Level
Coverage Level
(d) BIMODAL
(e) PENDULUM
(f) BIO
2.0
255
1.5
COR
COR
Length
GHR
Length
Ouns
Length
GHK
L.
0.5
0.5
0.2
0.4
0.6
0.8
0.2
0.4 Coverage Love)
0.6
0.8
0.2
0.4 Coverage Level
0.6
0.B
Coverage Level
(g) LOG NORMAL
(h) ENERGY
BREAST CANCER
Ours
Ours
1.5
CUR
COR
Longth
CHR
Leagth
Leagth
1.0
2.0
1.0
0.5
Coverage Level
Coverage Level
Caverage Level
(i) CONCRETE
(k) STOCK
(1) COMMUNITY
Langth
CILR
Length
CIIR
Length
1.5
CIR
1.0
3.0
0.
0.0
0.2
0.4
0.6
0.8
0.2
0.4
0.6
0.8
0.2
0.4
0.6
0.8
Coverage Level
Coverage Level
Coverage
Level
(m) FOREST
(n) PARKINSON'S
(0) MEPS-21
Figure 4: We plot the width of the predicted intervals for our method, CHR, and CQR across differ- ent coverage levels.
16

---

Published as a conference paper at ICLR 2024
DATASET
BIMODAL
LOG- NORMAL
CONCRETE
MEPS-
MEPS-
MEPS-
B10
COMMUNITY
NUMBER OF EXAMPLES
400
200
206
3157
3509
3132
9146
399
DATASET
DIARFTES
SOLAR
PARKINSONSSTOOCK
CANCER
PENDULUM
ENERGY
FOREST
NUMBER SINGLETONS
1103
105
37
115
153
NUMBER OF EXAMPLES
89
214
1175
108
39
126
154
104
Table 5:
We present an analysis of the number of singletons predicted by our method over all datasets. For most datasets, see that our method tends to predict singletons.
17

---

Published as a conference paper at ICLR 2024
Lengths
Lengnnr
Lengthe
Emors
Errors
Errors
(a) MEPS-21 20
(b) MEPS-21
(c) DIABETES
Lengeens
Lengths
Lengthh
Enrrs
Errors
Errors
(d) BIMODAL
(e) PENDULUM
(f) BIO
Lenggts
Lengths
Lengtho
Errors
Errors
Errors
(g) LOG NORMAL
(h) ENERGY
BREAST CANCER
Lengthh
Lengths
Lengths
Errrr
Errors
Errors
ii CONCRETE
(k) STOCK
(I COMMUNITY
Lenguts
Lengthe
Errors
Errors
Errors
(m) FOREST
(n) PARKINSON'S
(0) SOLAR
Figure 5: We plot the corrleation between absolute residual error and the length of our predicted intervals over several datasets. We see that there is not a strong correlation.
18

---

Published as a conference paper at ICLR 2024
Ground Truth
Ground Truth
Ground Truth
(L+uu " a)
Conlidence Level a
Confidence Level
Contidence
1000000
y/
y
(a) MEPS-20
(b) MEPS-21
(c) DIABETES
Ground Truth
Ground Truth
Ground Truth
Olg Nn
Olp [1.
Oig (Rs.))
Confidence Level i
Conlidence Level a
Confidence
1
V
(d) BIMODAL
(e) PENDULUM
(f)
Ground Truth
Ground Truth
Ground Truth
(lv
Gis
Oly (R..))
Conlidence Level or
Confidence
Confidence Levra
k0
Level
100
V
(g) LOG NORMAL
(h) ENERGY
ii
BREAST CANCER
Ground Truth
Ground Truth
Ground Truth
Olv)
0tyl Peet)
(V xa+tl
Confidence Level oi
Confidence
Conndencs
V
V
(i) CONCRETE
(k) STOCK
(I COMMUNITY
Ground Truth
Ground Truth
Ground Truth
OlylF...)
Oly))
(1+ bee (t)
Confidence
Level
Confidence Level a
Confidencs
g
y
(m) FOREST
(n) PARKINSON'S
(o) SOLAR
Figure 6: Example Probability Distributions outputted by our learned methodology on different
datapoints from different datasets. We there variety of different shapes of distributions learned.
19

---

Published as a conference paper at ICLR 2024
KDB
CHR
CB
0.5
1.0
1.5
2.0
0.0
0.5
1.0
1.5
2.0
2.5
3.5
4.0
Sest
Length
Wonel
Seat
Longth
Wond
(a) Bimodal Length Distribution
(b) Bio Length Distribution
XDB
KDB
La00
Length
(a) Breast Cancer Length Distribution
(b) Community Length Distribution
Ours
CHR COR
CHR COR
0.5
3.5
2.0
0.5
1.0
3.5
2.0
2.5
3.0
Longth
Worel
Length
Word
(a) Conerete Length Distribution
(b) Diabetes Length Distribution
20

---

Published as a conference paper at ICLR 2024
CHR
Sest
Length
Worel
Sext
Length
Word
(a) Energy Length Distribution
(b) Forest Length Distribution
XDB
KDB
Laa0
COR
Longth
Length
(a) Log Normal Length Distribution
(b) MEPS 19 Length Distribution
DOR
COR
CHR
20
50
30
50
Worel
Lengthh
Wond
(a) MEPS 20 Length Distribution
(b) MEPS 21 Length Distribution
21

---

Published as a conference paper at ICLR 2024
KDB
KDE
0OR
Longth
Length
(a) Parkinsons Length Distribution
(b) Pendulum Length Distribution
KDB
KDB
COR
COR
Length
Length
(a) Solar Length Distribution
(b) Stock Length Distribution
CQR
Ground Truth CQR Upper
CQR Lower
Ground Upper
Ctyl It-1)
r241)
0.30
KDE
KDE
CHR
CHRR
0.20
(g) (2
0.15
0.10
8.5-
Goo Soot
Confideace
Leval
</5
3/3
(a) BIO
(b) BREAST CANCER
Figure 7: Plot of outputted density functions for ours, KDE, and CHR on for BIO and BREAST CANCER
22

---

Published as a conference paper at ICLR 2024
Ground Truth
Ground Truth
COR Lmweer
DOR Upger
DOR Lower
COR Upger
07-
Cty [2)
Gtgl Fn-1)
04-
KDE
CHR
0.4
CHR
0.8
22
0.2
Condidence
6-1
Level
Comfidenc Level
04
0.-
25
0.5
V
V
(a) BIMODAL
(b) COMMUNITY
Figure 8: Plot of outputted density functions for ours, KDE, and CHR on for BIMODAL and COM-
MUNITY
Ground Truth
Ground Truth
COR Lower COR Upper
COR
COR Upper
1
05-
KDE CHR
KDE
CHR
0.2
0.1
Contidien
Love
Confideacc
Lavyed
0.0
2.0
(a) CONCRETE
(b) DIABETES
Figure 9: Plot of outputted density functions for ours, KDE, and CHR on for CONCRETE and DIABETES
Ground Truth
Ground
CQR Lower COR Upper
COR Upper
0rg R4 KDE
0ly F4.
KDE
CHR
8.16
(1
10
ConE
Confidesics
68
-15
8.0
1,5
V/
y
(a) ENERGY
(b) FOREST
Figure 10: Plot of outputted density functions for ours, KDE, and CHR on for ENERGY and FOR- EST
23

---

Published as a conference paper at ICLR 2024
CQR Lower Ground Truth
CQR Upper
Ground Truth
Upper
#
x
CHR
KDE
0.0
CHR
8.25
0.00
(P
0.4
0.10
Confideace
02-
0.65
Confidence
Level
LOG NORMAL
(h) MEPS-21
Figure 11: Plot of outputted density functions for ours, KDE, and CHR on for LOG NORMAL and MEPS-19
Upper
Upper
0.-
Ky [1) KDE
4ix (x))))
KDE
CHR
CHR
6.8
0.4
Confiduacc
Level
811
0.1
Confidersco
y (a) PARKINSON'S
y
(b)
PENDULUM
Figure 12: Plot of outputted density functions for ours, KDE, and CHR on for PARKINSON'S S and PENDULUM
Lower
CQR Upper
10
Kg 1)
G(v Pr)
KDF
KDF
0.
8.8
0.6
22
0.4
Con
0.1
Confidesios
0.0
(a) SOLAR
(b) STOCK
Figure 13: Plot of outputted density functions for ours, KDE, and CHR on for SOLAR and STOCK
24


Reviews:
[{"review_id": "9goRzFpPsP", "reviewer": "Reviewer_RG4V", "summary": "The paper proposes a new way to do conformal prediction for regression based on conformalizing a regression-to-classification method. It is claimed that the new method offers higher flexibility compared to using a traditional regressor, making it more suitable for heteroscedastic or multimodal data.", "questions": "- Could you elaborate on what linear interpolation is used to go from q_\\theta to \\bar{q}_\\theta? It is not clear to me at all. \n- You have pointed out a number of papers which advocate for regression-as-classification. However, it is still not convincing as to why we should first destroy the ranking structure of reals, and then reinstate it using the entropic regularization. Why not just do usual regression?\n- Which aspects of the R2C method are novel and which ones directly derive from previous work? Have you made significant changes to previous R2C methods to adapt them to the conformal problem? The key relevant technique seems to be smoothing; has these been proposed before?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The paper tackles a traditional conformal problem, but still seems to offer improvements to the state-of-the-art. \n- The method proposed is interesting; I had not previously seen this regression-as-classification setup in literature and appreciate the authors providing a number of references to the broader framework. \n- The paper is easy to follow, although there are some issues which I will point out in weaknesses. I also noticed the choice to not state and/or highlight the conformal coverage guarantee, which is often oversold nowadays. (I still think you should state the guarantee at least in the Appendix so that readers unfamiliar with conformal prediction are aware of it.)", "weaknesses": "## Writing \nSome parts of the paper were difficult to follow: \n- The proposed method was not easy to understand before seeing Algorithm 1. I felt Section 3 had too many comments on \"motivation\" that do not focus on the actual method and are a bit counterproductive to a quick read. E.g., Sec 3.1, \"We  aim to compute... classification context\" does not add much. In the end, you have a traditional neural network with softmax output, which everyone understands. Similarly the comment \"This approach is both straightforward and efficient ... information or structure.\" does not add anything substantial. Sec 3.2 \"It would be desirable to be able to use similar methods for both classification and regression conformal prediction\". This is a somewhat subjective (and distracting) claim that I believe is best avoided. The paragraph starting with \"These values yˆ ∈ Yˆ f...\" is unnecessarily verbose. Sec 3.3 introduces a loss function without giving an example of what it could be. \n- Table 1 is very hard to understand since (i) the methods are not connected to the acronyms, (ii) it is not clear what is inside the brackets, (iii) length and coverage is not defined in the paper. \n\n## Technical concerns\n- Most results are not reported with error bars making them less reliable. For instance, in Table 1, CHR does worse than CQR but it should not? (both methods are proposed by similar authors and CHR comes later). \n- Have you considered a baseline where you learn a regressor directly using the same neural network architecture, treat that regressor as \\bar{q}, and apply the same conformalization? It is similar to the \"optimal prediction sets\" in Appendix F here: https://arxiv.org/pdf/1910.10562.pdf . \n\nI am not familiar with the regression-as-classification (R2C) literature. The main concern I have is that it is unclear which aspects of the R2C method are novel and which ones directly derive from previous work. Have you made significant changes to previous R2C methods to adapt them to the conformal problem? The key relevant technique seems to be smoothing; has these been proposed before?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "cyfEIT05RF", "reviewer": "Reviewer_HZzS", "summary": "The paper proposes to see a regression problem as a classification one, and then to apply ideas issued from conformal prediction in order to derive predictions regions that may not be compact sets (here, union of intervals).", "questions": "- Could you position the current paper with respect to papers dealing with ordinal conformal regression? Two I know of (there could be others, but not much more) are \"Xu, Y., Guo, W., & Wei, Z. (2023, July). Conformal Risk Control for Ordinal Classification. In Uncertainty in Artificial Intelligence (pp. 2346-2355). PMLR.\" and \"Lu, C., Angelopoulos, A. N., & Pomerantz, S. (2022, September). Improving trustworthiness of ai disease severity rating in medical imaging with ordinal conformal prediction sets. In International Conference on Medical Image Computing and Computer-Assisted Intervention (pp. 545-554). Cham: Springer Nature Switzerland.\". \n\n- It seems to me that the conformal scores used in Algorithm 1 mostly rely on modal values. In ordinal regression, it is much more common to use the average rank (under $L_2$ loss) or the median (under $L_1$ loss) as predictions. Could you comment on such options? \n\n- It would be nice to have an idea of how often the output predictions regions are not interval, to have an idea of how often we depart from classical methods. Would it be possible ot have an idea (in the appendix or main paper).\n\n- Could you display full coverage curve (say, from 90% to 99%) rather than table for a fixed value? (side remark: there is a double ?? in appendix B). \n\n- Bonus question for myself: do you have an idea of how the present method could be adapted to multi-variate output settings?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "+: an interesting view of the problem, especially as this allows the conformal predictions to be easily something else than intervals, something that may indeed be of important interest for regression. No other conformal regression methods that I know of achieve this kind of things, with maybe the exception of \"Conformal prediction in manifold learning\" (not cited by the authors, but the paper is only weakly related to the present work), since intervals on the manifold may well turn out to be non-compact in the original space. \n\n+: a quite well written paper, and a method simple enough to be applicable in a wide range of settings. \n\n+: experiments that are convincing enough to show the interest of the method.", "weaknesses": "-: the way authors frame the regression problem as a classification is very, very close to the standard ordinal regression problem, and I really missed some positioning with respect to this literature. There is a huge literature on ordinal regression (maybe look at a general paper, e.g., \"Tutz, G. (2022). Ordinal regression: A review and a taxonomy of models. Wiley Interdisciplinary Reviews: Computational Statistics, 14(2), e1545.\"), but also a couple papers using ordinal conformal classification (see questions).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4Hxjb5og7i", "reviewer": "Reviewer_iHe5", "summary": "The paper proposes to convert 1D regression problems into K-class classification problems by partitioning the label space into K bins.  This\nallows leveraging known techniques for classification CP. Like Lei et al. 2004, the authors use the conditional density produced by the classification model as a conformity score. On top of it, a regularization model is trained to enforce smoothness when the discrete distribution is converted back to a continuous one.", "questions": "- Is the smoothness-enforcing penalty new? \n- How were K and Tau chosen? Is part of the data used for training the conditional distribution?\n- Why should the entropy regularization be expected to be good at learning bi-modal distributions?\n- The size of the prediction intervals is a good measure of efficiency. Another one is the correlation between the test errors and the corresponding intervals. It would be interesting to see a scatter plot of the absolute residual versus the prediction intervals in a couple of data sets where R2CCP is or is not the best algorithm.\n- Is the proposed approach equivalent to training a parameterized conformity function (see for example in [2] or [3])?\n\n[1] Guan, Localized conformal prediction: A generalized inference framework for conformal prediction (2023)\n[2] Einbinder et al., Training Uncertainty-Aware Classifiers with\nConformalized Deep Learning (2022)\n[3] Colombo, On training locally adaptive CP (2023)", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "I like the idea of learning the conformity score is interesting.  Works like [1] also use density estimation to adjust the conformity score. But the authors' approach is intrinsically different.  The proposed objective function seems a good alternative to optimizing the efficiency of the prediction intervals.", "weaknesses": "The role of the discretization step may be explained better. The authors could emphasize the difference with other techniques for estimating conditional densities or explain why the proposed approach does not suffer from the usual instability of standard estimators.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "AYMj9GGJFt", "reviewer": "Reviewer_ZE11", "summary": "This paper presents a new method to model the conditional density in heteroscedastic regression problems. The idea is to convert metric regression to an ordinal regression problem, in which the conditional density is approximated by a number of fixed bins. This is in essence a histogram approach. Unlike most ordinal regression methods, the authors do not consider an underlying latent continuous variable, but they model every bin via a separate neuron propagating from the embedding layer. A softmax operation is used to guarantee that a valid probability density function is returned, but, unlike classification methods, a loss function that incorporates the order in the bins is used during training. \n\nIn a conformal prediction type of experiments, the authors show that this new model results in better prediction sets compared to some baselines for modelling the conditional density, such as quantile regression and kernel density estimation.", "questions": "See above.\nIn addition, what's novel in this paper compared to reference Stewart et al.?", "limitations": "", "rating": 3, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The paper is well written and easy to follow\n- The paper has some novelty, although I would argue that the core idea is just an application of reference Stewart et al. \n- I enjoyed reading this paper, and I am convinced that the method can work relatively well for regression problems with multimodal conditional density distributions\n- I liked that some limitations of the method are discussed at the end. However, I do see other limitations as well.", "weaknesses": "I do see several important weaknesses. \n\n1. Contributions:\nAfter reading the introduction it was not clear to me what the contributions are. This only became clear to me after reading the experiments. Contributions should be clear from the beginning. To my opinion, the main contribution is proposing a new nonparametric method for modelling the conditional density. However, I don't see any contribution w.r.t. conformal prediction. In the experiments, the authors only consider the standard split conformal prediction, which is arguably not the most useful type of evaluation for heteroscedastic regression problems. I would argue that conditional coverage guarantees need to be analyzed in such cases. This can be established via Mondrian conformal prediction or normalized nonconformity scores, such as normalized absolute residuals. Estimates of the variance could be obtained with any method that models conditional densities, or with methods that directly estimate the variance, such as mean-variance neural networks. So, in that regard, I find the scope and the experimental setup quite limited. \n\n2. Baselines:\nThe authors consider a few other methods for estimating the conditional density, such as kernel density estimation and quantile regression. However, many other methods exist for estimating the conditional density. Since this is the key contribution of the paper, I would like to see a more thorough experimental and theoretical comparison with such methods. Examples of such methods are mean-variance neural networks, conditional transformation models and methods based on normalizing flows. The proposed method estimates the conditional density via a specific histogram. Methods based on nearest neighbors and regression trees also produce histogram-style conditional densities. Another approach to model multimodal distributions is via Gaussian mixtures. So, there is a lot out there already... Where should we situate the new method is this diverse landscape? I would classify the new method as yet another way of modelling the conditional density. Not better or worse than others, perhaps useful in specific situations, such as multimodal distributions, but more discussion is needed.  \n\n3. Limitations of the method:\n- By modelling the conditional density as a histogram, one obtains a very non-smooth approximation of the density. So, that's why interpolation is needed to obtain good nonconformity scores. Some of the other methods that are described above don't have this limitation. \n\n- The number of bins looks like a very tricky hyperparameter to tune. The performance probably highly depends on that hyperparameter. What would be a good way to tune this hyperparameter? Another tricky hyperparameter is the level of entropy regularization. I would assume that the results are also very sensitive to that hyperparameter.\n\n- I believe that the proposed method might work reasonably well for multimodal distributions that are not too complex, but I would not recommend this method for unimodal regression problems. For such problems, there are too many degrees of freedom.\n\n4. Unclear aspects of the experiments\n- In Table 3 a lot of standard deviations are zero, or close to zero. How is that possible? What is randomized? With randomizations of the testset these numbers are unrealistically small I would say. \n- Table 1 versus Table 3: normally one would see that the length of the intervals increases when the coverage increases. So, the two criteria highly depend on each other. Any comparison of two methods should always make the trade-off between those two criteria.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)