Predictive Inference with Feature Conformal Prediction
Authors: Jiaye Teng
Chuan Wen
Dinghuai Zhang
Yoshua Bengio
Yang Gao
Yang Yuan
Venue: ICLR 2023 poster (2023)
Keywords: conformal prediction
uncertainty

Abstract: Conformal prediction is a distribution-free technique for establishing valid prediction intervals. Although conventionally people conduct conformal prediction in the output space, this is not the only possibility. In this paper, we propose feature conformal prediction, which extends the scope of conformal prediction to semantic feature spaces by leveraging the inductive bias of deep representation learning. From a theoretical perspective, we demonstrate that feature conformal prediction provably outperforms regular conformal prediction under mild assumptions. Our approach could be combined with not only vanilla conformal prediction, but also other adaptive conformal prediction methods. Apart from experiments on existing predictive inference benchmarks, we also demonstrate the state-of-the-art performance of the proposed methods on \textit{large-scale} tasks such as ImageNet classification and Cityscapes image segmentation.

Published as a conference paper at ICLR 2023
PREDICTIVE INFERENCE WITH FEATURE CONFORMAL PREDICTION
Jiaye Tengl.3.1,, Chuan Wen Dinghuai Zhang
Yoshua Bengio?, Yang Gao 13,44 Yang Yuan 1,3,4,1
Instiittee for Interdisciplinary Information Sciences, Tsinghua University
Mila Quebec AI Institute
Shanghai Artificial Intelligence Laboratory
"Shanghai Qi Zhi Institute
{t jtjy00, owen20 @mails t tsinhuua edu. dinghuai. bengghaiizhangggiiii zhanggmmila quebec
ABSTRACT
Conformal prediction distribution free technique for establishing valid pre- diction intervals. Although conventionally people conduct conformal prediction in the output space, this not the only possibility. In this paper, we propose feature conformal prediction, which extends the scope of conformal prediction to semantic feature spaces by leveraging the inductive bias of deep representation learning. From a theoretical perspective, we demonstrate that feature confor- mal prediction provably outperforms regular conformal prediction under mild assumptions. Our approach could be combined with not only vanilla conformal prediction, but also other adaptive conformal prediction methods. Apart from experiments existing predictive inference benchmarks, we also demonstrate the state-of-the-art performance of the proposed methods on large-scale tasks such as ImageNet classification and Cityscapes image segmentation. The code available in /tttps://githbb /
INTRODUCTION
Although machine learning models work well in numerous fields (Silver et al., 2017 Devlin et al.) [Brown (rronn
al., 2020), they usually suffer from over-confidence issues, yielding unsatisfactory uncertainty estimates (Guo et al.) 2017a Chen et al.) 2021 Gawlikowski et al.) 2021). To tackle
the uncertainty issues, people have developed multitude uncertainty quantification techniques, including calibration (Guo et al. 201 7b Minderer et al. (021). Bayesian neural networks (Smith)
2014 Blundell et al. 2015), and many others (Sullivan 2015).
Among different uncertainty quantification techniques, conformal prediction (CP) stands out due to its simplicity and low computational cost properties (Vovk et al., 2005) Shafer & Vovk, 2008) An gelopoulos & Bates 2021). Intuitively, conformal prediction first splits the dataset into training fold and calibration fold, then trains a machine learning model on the training fold, and finally constructs the confidence band via a non-conformity score the calibration fold. Notably, the confidence band obtained by conformal prediction is guaranteed due to the exchangcability assumption in the data, With such a guarantee, conformal prediction has been shown perform promisingly on numerous realistic applications (Lei & Candès 2021b Angelopoulos et al. 2022).
Despite its remarkable effectiveness, vanilla conformal prediction (vanilla CP) is only deployed the output space, which is not the only possibility. As an alternative, feature space in deep learning stands out due to powerful inductive bias of deep representation. Take the image segmentation problem as example. In such problems, we anticipate a predictive model to be certain in the informative regions (e.g., have clear objects), while uncertain elsewhere. Since different images would possess different object boundary regions, it is inappropriate return the same uncertainty for different positions, as standard conformal prediction does. Nonetheless, if we instead employ conformal
"Equal Contribution.
1 Corresponn to yuanyang@mai ttiinnuua.edu

---

Published as a conference paper at ICLR 2023
Semantic feature space!
Calibration Set
Calibration Set
Conformal Predictor
Feature Conformal Predictor
Test Point
Test Point
Figure I: Illustration of vanilla CP (left) vs Feature CP (right). Feature CP operates in the semantic feature space, opposed to the commonly adopted output space. These methods are described in further detail in Sections and
prediction on the more meaningful feature space, albeit all images have the same uncertainty on this intermediate space, the pixels would exhibit effectively different uncertainty in the output space after a non-trivial non-linear transformation (see Figure
In this work, we thus propose the Feature Conformal Prediction (Feature CP) framework, which deploys conformal prediction in the feature space rather than the output space (see Figure However, there are still two issues unsolved for performing Feature CP: (a) commonly used non-conformity scores require ground truth term, but here ground truth in feature space is not given; and (b) transferring the confidence band in the feature space to the output space is non-trivial. To solve problem (a), we propose a new non-coo conformity score based on the notation surrogate feature, which replaces the ground truth term in previous non-conformity scores. As for (b), we propose two methods: Band Estimation, which calculates the upper bound of the confidence hand, together with Band Detection, determine whether response locates in the confidence band. More interestingly, feature-level techniques are pretty general and can be deployed into other distribution-fii inference algorithms, e.g., conformalized quantile regression (CQR). This shows the great potential application impact of the proposed Feature CP methodology (see the discussion in Appendix B.4).
From a theoretical perspective. we demonstrate that Feature CP is provably more efficient, in the sense that it yields shorter confidence bands than vanilla CP, given that the feature space meets cubic conditions. Here the cubic conditions sketch the properties of feature space from three perspectives. including length preserving, expansion, and quantile stability (see Theorem 6 At colloquial level, the cubic conditions assume the feature space has smaller distance between individual non-confornity scores and their quantiles, which reduces the cost of the quantile operation. We empirically validate that the feature space deep learning satisfies the cubic conditions, thus resulting in better confidence band with shorter length (See Figure] according theoretical analysis.
Our contributions can be summarized as follows:
We propose Feature CP, together with a corresponding non-conformity score and uncer- tainty band estimation method. The proposed method no longer treats the trained model as black box but exploits the semantic feature space information. What's more, our approach could be directly deployed with any pretrained model as a plug-in in component, without the need of re-training under specially designed learning criteria.
Theoretical evidence guarantees that Feature CP is both (a) efficient, where it yields shorter contidence bands, and (b) effective, where the empirical coverage provably exceeds the given confidence level, under reasonable assumptions.
We conduct extensive experiments under both synthetic and realistic settings (e.g., pixel-level image segmentation) corroborate the effectiveness of the proposed algorithm. Besides, demonstrate the universal applicability of our method by deploying feature-level operations improve other adaptive conformal prediction methods such COR.
2

---

Published as a conference paper at ICLR 2023
2 RELATED WORK
Conformal prediction is statistical framework dealing with uncertainty quantification prob- lems (Vovk et al., 2005 Shafer & Vovk 2008 Nouretdinov et al., 2011 Barber et al., 2020
Angelopoulos & Bates 2021). The research conformal prediction
be roughly split into the following branches. The first line of work focuses on relaxing the assumptions about data
distribution in conformal prediction, e.g., exchangeability (Tibshirani al. 2019. Hu & Lei 2020
Podkopacv & Ramdas] 2021; Barber et al. 20022) The second line aims at improving the efficiency
of conformal prediction (Romano et al. 2020b Sesia & Candès 2020 Izbicki et al. 2020a Yang
& Kuchibhotla 2021; Stutz et al. 2021) D. The third line tries to generalize conformal prediction to different settings, e.g., quantile regression (Romano et al. 2019), k-Nearest Neighbors (Papadopoulos et al.
2011), density estimator (Izbicki et al., 2020b). survival analysis (Teng et al. [
ct al.
2021 or conditional histogram regression (Sesia & Romano 2021). There are also works combining conformal prediction with other machine learning topics, such as functional data (Lei)
2013). treatment effects (Lei & Candès 2021a). time series analysis (Xu & Xie (202T). online ctt al.
learning (Gibbs & Candès 2021). adversarial robustness (Gendler al. 2022). and many others.
Besides conformal prediction, there are many other uncertainty quantification techniques, including calibration (Guo et al. 2017a, Kuleshov et al., 2018, Nixon et al.
2019) and Bayesian- based techniques (Blundell et al. 2015 Hermández-Lobard & Adams 2015 Li & Gal 2017]. Different from the above techniques, conformal prediction is appealing due to its simplicity, computationally
free, and model-free properties.
Image segmentation is traditional task in computer vision, which focuses partitioning images into different semantic segments (Haralick & Shapiro 1985 (Senthilummrrnn & Rajesh 20099] Minaee al. 2020). A line of researches applies conformal prediction with some threshold output for all pix- cls Angelopoulos & Bates] 202T] Bates et al. [021], or focus on the risk control tasks (Angelopoulos al. [222aa). Different from previous approaches, our method first achieves meaningful pixel-level conformal prediction results to the best of our knowledge.
3 PRELIMINARIES
Predictive inference. Let (X, ~ P denotes random data pair, e.g., image and its segmentation map. Given a significance level a, we aim to construct a confidence band (X), such that
P(Y e (X)) 2 1-a.
There is tradeoff between efficiency and effectiveness, since one can always set (X) to be infinitely large to satisfy Equation (I). In practice, we wish the measure of the confidence band (e.g.. its length) can be small as possible, given that the coverage in Equation (I) holds.
Dataset. Let D = {(x,, YEJ denotes the dataset, where I denotes the set of data index and (X,, denotes sample pair following the distribution P. Typically, conformal prediction requires that data in D satisfies exchangeabilityy (see below) rather than the stronger i.i.d. (independent and identically distributed) condition. We use [Z] to represent the cardinality of set I. Conformal prediction needs to first randomly split the dataset into a training fold Drr = {(X, Yi), and a calibration fold Dea {(,,, YJ) HEz,, where In UIa = I and In n Iea = Φ. We denote the test point as (X', Y'), which is also sampled from the distribution P.
Training process. During the training process, we train machine learning model denoted by (a() (e.g., neural network) with the training fold P For the ease of the following discussion, we rewrite the model as I f. where f denotes the feature function (i.e., first several layers in neural networks) and 9 denotes the prediction head (i.e., last several layers neural networks).
Calibration process. Different from usual machine learning methods, conformal prediction has an additional calibration process. Specifically, we calculate a non-conformity score V;2 = s sX,,, Y,, a)) based on the calibration fold Dc., where s(.:"," ...) -) is a function informally measuring how the model fits the ground truth. The simplest form of non-conformity score is sX,,, Y,, p)) = ||Y; - ((X.))l One could adjust the form of the non-conformity score according different contexts (e.g., Romano jet al. (2019): Teng et al. (202I). Based on the selected non non-conformity score, a matching contidence band could be subsequently created.
3

---

Published as a conference paper at ICLR 2023
Algorithm 1 Conformal Prediction
Require: Desired confidence level a, dataset D {(X,, Yiz, test point X', non-conformity
score function s(-)
1:
Randomly split the dataset 'D into a training fold Dr _ (X,, Vivez, and a calibration fold
Dea (X,,, Ve
2: Train base machine learning model pi()) with Du to estimate the response Yú;
3: For each E Iem, calculate its non-conformity score V; = s(X;, Y,,
4: Calculate the (1 a-t quantile Q -a of the distribution [T.l-I Eiez,.. dv; 80c
Ensure: C1 Ci-a(X)) = (Y: 3(XX,Y, S Q1-a).
We present vanilla CI in Algorithm 1 Moreover, we demonstrate its theoretical guarantee in Proposition 2 based on the following notation of exchangeability in Assumption] !
Assumption (exchangeability), Assume that the calibration data (X,, Y:), i E Iea and the test point (X', Y') are exchangeable. Formally, define Z = 1..... - + 11, the above data pair, then Zi are exchangeable if arbitrary permutation leads to the same distribution, i.e.,
(Z,,.., 7 Iml+1) 4 (7x(1))---. Zm(lI.l+1)),
with arbitrary permutation a over (1,-.. ITeu + 1/}.
Note that Assumption T weaker than the i.i.d. assumption. Therefore, it is reasonable assume the exchangeability condition to hold in practice. Based on the exchangeability assumption, one can show the following theorem, indicating that conformal prediction indeed returns a valid confidence band, which satisfies Equation ①
Theorem (theoretical guarantee for conformal prediction (Law 2006; Lei et al. 2018 Tibshirani
al. 2019)). Under Assumption the confidence band Ci returned by Algorithm satisfies
P(Y" E CI-a(X')) 2 aa.
METHODOLOGY
In this section, we broaden the concept of conformal prediction using feature-level operations. This extends the scope of conformal prediction and makes it more flexible. We analyze the algorithm components and details in Section 4.1 and Section 4.2 The algorithm is finally summarized in Section 4.3 We remark that although in this work we discuss Feature CP under regression regimes for simplicity's 's sake, one can easily extend the idea classification problems.
4.1 NON-CONFORMITY SCORE
Conformal prediction necessitates non-conformity score to measure the conformity between pre- diction and ground truth. Traditional conformal prediction usually uses norm-based non-conformity score due to its simplicity, i.e., s(X, () ||Y II, where Y is the provided ground truth target label. Nonetheless, we have no access to the given target features we want to conduct confor- mal prediction at the feature level. To this end, introduce the surrogate feature (see Definition which could serve as the role of ground truth Y Feature CP.
Definition (Surrogate feature), Consider a trained neural network ft go where denotes the composition operator. For a sample (X,Y) Y), we define i = f(X) to be the trained feature. Besides, we define the surrogate feature to be any feature v such that g(v) Y.
In contrast to commonly adopted regression or classification scenarios where the label is unidi- mensional, the dimensionality features could be much larger. We thus define a corresponding non-conformity score based on the surrogate feature follows:
(X, Y, g =E(e:)()==))
inf || f(X)]I
(3)
We use 5, represent Dirac Delta function (distribution) at point u.
4

---

Published as a conference paper at ICLR 2023
is usually complicated to calculate the score in Equation G to the infimum operator. Therefore, design Algorithm to calculate upper bound of the non-conformity score. Although the exact infimum is hard to achieve in practice, we can apply gradient descent starting from the trained feature to find a surrogate feature v around it. In order demonstrate the reasonability of this algorithm. we analyze the non-conformity score distribution with realistic data in Appendix B.9
4.2 BAND ESTIMATION AND BAND DETECTION
Utilizing the non-conformity score derived in Section 4.1 one could derive a confidence band in the feature space. In this section, we mainly focus on how to transfer the confidence band in feature space to the output space, i.e., calculating the set
(0(+) 110 -ll S Qi-a);
where é is the trained feature, 9 is the prediction head, and Qi-a is derived based the calibration set (even though slightly different, we refer to step 4 in Algorithm for the notion of Qi-a: formal discussion of it is deferred to Algorithm 3).
Algorithm 2 Non-conformity Score
Require: Data point (X, (X,Y), trained predictor
9 o f(.). step size 17, number steps M; 1: u + f(X);
Since the prediction head Ò is usually highly non- linear, the exact confidence band is hard to rep- resent explicitly. Consequently, we provide two approaches: Band Estimation which aims at esti- mating the upper bound of the confidence band, and Band Delection which aims at identifying whether a response falls inside the confidence interval. We next crystallize the two methods.
2: m + 0;
3: while m < M do
-
4: u
Ou
5: m m m + 1;
6: end while
Ensure: Y 11 f(xII
Band Estimation. We model the Band Estimation
problem as a perturbation analysis one, where regard in Equation the trained feature é, and analyze the output bounds of prediction head ğ. In this work, we apply linear relaxation based perturbation analysis (LiPRA) (Xu et al., 2020) tackle this problem under deep neural network regimes. LiPRA transforms the certification problem as a linear programming problem, and solves it accordingly. The relaxation would result in relatively looser interval than the actual band, so this method would give an upper bound estimation of the exact band length.
perturbation of
Band Detection. Band Estimation could potentially end up with loose inference results. Typically, are only interested in determining whether a point Y is in the confidence band C(X') for test sample X'. To achieve this goal, we first apply Algorithm 2 Jsiin data point (X', Y), which returns a non- non-coonormityy score V. We then test whether the score V smaller than quantile Q 1- on the calibration set (see Equation
so, we deduce that Y E C(X') (or vice versa if not).
4.3 FEATURE CONFORMAL PREDICTION
Based on the above discussion, we summarize Feature CP in Algorithm Different from vanilla CP (see Algorithm|] Tp, Feature CP uses different non-conformity score based on surrogate features, and we need an additional Band Estimation or Band Detection (step 5) transfer the band from feature space to output space.
We then discuss two intriguing strengths of Feature CP. First, the proposed technique is universal and could improve other advanced adaptive conformal inference techniques utilizing the inductive bias of learned feature space. Specifically, we propose Feature CQR with insights from CQR (Romano) et al. 2019), prominent adaptive conformal prediction method with remarkable performance. to demonstrate the universality of our technique. We relegate related algorithmic details to Section [B.4 Second, although methods such CQR require specialized training criteria (e.g., quantile regression) for the predictive models, Feature CP could be directly applied to any given pretrained model and
We here show the Feature CP algorithm with Band Estimation. We defer the practice details of using Band Detection in step of Algorithm 3 to Appendix.
5

---

Published as a conference paper at ICLR 2023
Algorithm 3 Feature Conformal Prediction
Require: Level a, dataset D {(X,, Y.) test point X';
1: Randomly split the dataset 'D into training fold Drr (X,, ViJEz, together with calibration
fold cen - (X;, T
2: Train base machine learning model o f()) using Drr to estimate the response Y;;
3: For each E Iea. calculate the non-conformity score V, based Algorithm 2
4: Calculate the (1 -h quantile Qı-a of the distribution |T.1-T EiEZ,, dv: + dzz
5: Apply Band Estimation on test data feature f(X') with perturbation Q1-a and prediction head
f. which returns C (X);
Ensure: ChP Cx=a(X).
could still give meaningful adaptive interval estimates. This trait facilitates the usage of our method with large pretrained models, which common modern language and vision tasks.
4.4 THEORETICAL GUARANTEE
This section presents theoretical guarantees for Feature CP regarding coverage (effectiveness) and band length (efficiency). We provide an informal statement of the theorem below and defer the complete details Appendix A.I
Theorem 4 (Informal Theorem the Efficiency of Feature CP). Under mild assumptions, if the following cubic conditions hold:
1. Length Preservation. Feature CP does not cost much loss in feature space.
2. Expansion. The Band Estimation operator expands the differences between individual length and their quantiles.
3. Quantile Stability. The band length stable in both the feature space and the output space for given calibration set.
then Feature CP outperforms vanilla CP in terms of average band length.
The intuition of Theorem 4 iis as follows: Firstly, Feature CP and Vanilla CP take quantile operations in different spaces, and the Expansion condition guarantees that the quantile step costs less in Feature CP. However, there may be an efficiency loss when transferring the band from feature space to output space. Fortunately, is controllable under the Length Preservation condition. The final Quantile Stability condition ensures that the band is generalizable from the calibration fold to the test samples. We provide the detailed theorem in Appendix A. and empirically validate the cubic conditions in Appendix B.2
5
EXPERIMENTS
We conduct experiments synthetic and real-world datasets, mainly to show that Feature CP is (a) effective, i.e., could return valid confidence bands with empirical coverage larger than - az (b) efficient, i.e., it could return shorter confidence bands than vanilla CP.
5.1 SETUP
Datasets. We consider both synthetic datasets and real-world datasets, including (a) realistic uni-
dimensional target datasets: five datasets from UCI machine learning repository (Asuncion) 2007): physicochemical properties of protein tertiary structure (bio), bike sharing (bike), community and crimes (community) and Facebook comment volume variants one and two (facebook 1/2), five datasets from other sources: blog feedback (blog) (Buza 2014), Tennessee's 's student teacher achievement ratio (star) (Achilles et al., 2008). and medical expenditure panel survey (meps19-2) (Cohen et al. 2009); (b) synthetic multi- dimensiannni target dataset: Y WX e, where X [0, 10 160 " R

---

Published as a conference paper at ICLR 2023
92-
Feature CP
91
68
communtyy
Docebouk
facetook2
musia1
mapazu
b
coemedyy
baitbank1
Data
Date
93
CON
Feature COR
2.5
Covergg
90
1.0
0.5
community
facebrakd
facebook2
menss mopo26
mapr21
6io
bige
conmtedi
facebbokk
(borbooks
mepeb0
Dtar
Data
Date
Figure 2: Performance on tasks with one-dimensional target (a = 0.1). Left: Empirical coverage. Right: Confidence interval length, where smaller value is better. The proposed Feature CP and Feature CQR could consistently achieve shorter bands while maintaining good coverage performance.
Table 1: Performance of both methods munti-diimessionll regression benchmarks (a 0.1), where "w-length" denotes the weighted confidence band length.
DATASET
SYNTHETIC
CITYSCAPES
METHOD
COVERAGE
LENGTH
COVERAGE
LENGTH
W-LENGTH
BASELINE
89.91+1.00
00.01
91.41:0.51
40.15:20.02
10.10:20.02
FEATURE CP
90.13:0.59
0.37340.05
90.7740.91
1.032+0:01
0.90640.01
follows the standard Gaussian distribution, and W is fixed randomly generated matrix; and (c)
real-world semantic segmentation dataset: Cityscapes (Cordis et al. 2016). where we transform the original pixel-wise classification problem into high-dimensional pixel-wise regression problem. We also extend Feature CP to classification problems and test on the ImageNet (Deng et al. 2009) dataset. We defer more related details to Appendix B.I
We compare the proposed Feature CP against the vanilla conformal baselines without further specified, which directly deploy conformal inference on the output space. For both methods, we use -type non non-conformity score, namely, s(X, Y, I [(X)] lc.
Algorithms.
Evaluation. We adopt the following metrics evaluate algorithmic empirical performance.
Empirical coverage (effectiveness) is the empirical probability that a test point falls into the predicted confidence band. A good predictive inference method should achieve empirical coverage slightly larger than for a given significance level a. To calculate the coverage for Feature CP, we first apply Band Detection on the test point (XX', Y') to detect whether Y' is in CfT (X'). and then calculate its average value to obtain the empirical coverage.
Band length (efficiency). Given the empirical coverage being larger than 1-a, we hope the confidence band to be as short as possible. The band length should be compared under the regime of empirical coverage being larger than zero band length. Since the explicit expression for confidence bands is intractable for the proposed algorithm, could only derive estimated band length via Band Estimation. Concretely,
otherwise
always set the confidence band to empty get
7

---

Published as a conference paper at ICLR 2023
Image
Ground Truth Label
Length
04
-02
Figure 3: Visualization of Feature CP image segmentation. The brightness of the pixels in the third column measures the uncertainty of Feature CP, namely the length of confidence bands. The algorithm is more uncertain the brighter regions. For better visualization, we rescale the interval length to the range of [0, 1]. Feature CP is more uncertain in non-informative regions, which are object boundaries.
0.814
BA35
E/06
B:34
0.80
difle Rus
RAnILS IJI
0010
BJS R. 0L5S
-U08
(a) Facebook1
(b) Synthetic
(c) Cityscapes
Figure 4: Empirical coverage under different confidence levels. For good conformal prediction
method, the y-axis (i.e., empirical coverage minus (1 a)) should keep being above zero for different a. These three figures above show that Feature CP generally performs better than the baseline, sense that this difference is above zero most of the time.
first use Band Estimation estimate the confidence interval, which returns band with explicit formulation, and then calculate the average length across each dimension.
We formulate the metrics as follows. Let Y (Y)...... (()) ) E gd denotes the high dimensional response and C(X) C Rd denotes the obtained confidence interval, with length in each dimension forming vector [C(X] E Rd. With the test set index being Iςe and [d] (I...., the empirical coverage and band length respectively as
.....
......,d, d}, we calculate
1(Y C(X))),
((
T
]E[d]
5.2 RESULTS AND DISCUSSION
Effectiveness.. We summarize the empirical coverage in Figure (one-dimension response) and Table (multi-dimension response). Theorem illustrates, the empirical coverage of Feature CP all exceeds the confidence level demonstrates that that the effectiveness holds with different significance levels a. For simple benchmarks such facebook1 and synthetic data, both methods achieve similar coverage due the simplicity; while for the harder Cityscapes segmentation task, the proposed method outperforms the baseline under many confidence levels.
indicating that Feature CP is effective. Besides, Figure
Efficiency. We summarize the confidence band in Figure 2 (one-dimension response) and Table ! (multi-dimension response). The band length presented is estimated version via Band Estimation.
8

---

Published as a conference paper at ICLR 2023
Table 2: Feature CP outperform previous methods on large-scale classification tasks (ImageNet.
0.1). We compare our results with APS (Romano et al., 2020a) and RAPS (Angelopoulos et al. 2021b). The results of baselines are taken from Angelopoulos et (2021b)
METHOD
ACCURACY
COVERAGE
LENGTH
MODEL
ToP-1
TOP-5
APS
RAPS
FEATURE CP
APS
RAPS
FEATURE CP
RESNET 18
0.698
0.891
0.900
0.900
0.90210.0002.
16.2
4.43
3.82.1..18
RESNET 50
0.761
0.929
0.900
0.900
0.900110.004
12.3
2.57
2.14100.03
RESNET 101
0.774
0.936
0.900
0.900
0.90010.0025
10.7
2.25
2.1710..01
RESNEXT 101
0.783
0.945
0.900
0.900
0.900#0 0030
19.7
2.00
1.80
SHUFFLENET
0.694
0.883
0.900
0,900
0,901 0019
31.9
5.05
5.06+0001
VGG 16
0.716
0.904
0.901
0.900
0.901#0.00417
14.1
3.54
3.26:#0.03
Note that Feature CP outperforms the baseline in the sense that it achieves a shorter band length and thus a more efficient algorithm.
Comparison to CQR. prediction techniques. As an example, we propose Feature CQR which is a feature-level generalized version of CQR, whose details are deferred to Appendix B8.4| We display the comparison in Fig- ure 2 where our method consistently outperforms CQR baseline by leveraging good representation. Besides, we evaluate the group coverage performance of CQR and Feature CQR in Appendix demonstrating that Feature CQR generally outperforms CQR in terms of condition coverage. One can also generalize the other existing techniques to feature versions, e.g., Localized Conformal Prediction (Guan 2019; Han et al. 2022).
The techniques proposed in this paper can be generalized to other conformal
Extension to classification tasks. The techniques in Feature CP can be generalized classification tasks. We take the ImageNet classification dataset as an example. Table Feature CP classification tasks. We experiment on various different architectures and use the same pretrained weights as the baselines. We provide additional details in Appendix B.7
2 showws the effectiveness of
Ablation on the splitting point. We demonstrate that the coverage is robust the splitting point in both small neural networks (Table Appendix) and large neural networks (Table 10 in Appendix). One can also use standard volos-valiittion to choose the splitting point. Since the output space is one of the special feature layers, such techniques always generalize the scope of vanilla CP.
Truthfulness. We visualize the segmentation results in Figure which illustrates that Feature CP returns large bands (light region) the non-informative regions (object boundaries) and small bands (dark region) on the informative regions. We do not show baseline visualization results since they return the same band in each dimension for each sample, and therefore does not contain much information. We also evaluate the performance with weighted band length, defined in Appendix B.
How does Feature CP benefit from good deep representation? Here we provide on the success of Feature CP algorithm: we claim it the usage of good (deep) representation that enables Feature CP to achieve better predictive inference. To validate this hypothesis, we contrast Feature CP against the baseline with an unlearned neural network (whose feature is not semantic as desired). This "random" variant of Feature CP does outperform its vanilla counterpart with the same neural network, which confirms our hypothesis. We defer the results to Table 6 and related discussion to Appendix B.3
intuition
More discussion. We analyze the failure (i.e., inefficient) reasons of vanilla CP in image segmentation task from the following two perspectives. Firstly, this paper aims to provide provable coverage, namely, the confidence band should cover the ground truth for each pixel. Since vanilla CP returns the same band for different samples, the loss is pretty large such that the returned interval is large enough to cover the ground truth. Secondly, an intuitive explanation relates to our usage of lox to form the non-conformity score during the training. We choose the infinity norm because reporting the total band length requires the band length in each dimension. As result, the non-conformity score is large as long as there exists one pixel that does not fit well, contributing to an unsatisfying band for vanilla CP.

---

Published as a conference paper at ICLR 2023
ACKNOWLEDGMENTS
This work has been partly supported by the Ministry of Science and Technology of the People's Republic of China, the 2030 Innovation Megaprojects "Program on New Generation Artificial Intelligence" (Grant No. 2021AAA0150000,, This work is also supported by a grant from the Guoqiang Institute, Tsinghua University.
REFERENCES
CM Achilles, Helen Pate Bain, Fred Bellott, Jayne Boyd-Zaharias, Jeremy Finn, John Folger, John Johnston, and Elizabeth Word. Tennessee's student teacher achievement ratio (star) project. Harvard Dataverse, 1 1:2008, 2008.
Anastasios N. Angelopoulos and Stephen Bates. A gentle introduction to conformal prediction and distributiion-free uncertainty quantification. CoRR, abs/2107.07511, 2021. URL https
//arxiv..rr//bb//2117.
Anastasios N Angelopoulos, Stephen Bates, Emmanuel Candès, Michael Jordan, and Lihua Lei. Learn then test: Calibrating predictive algorithms to achieve risk control. arXiv preprint 1rXiv:2110.01052 2021a.
Anastasios N. Angelopoulos, Amit S. Kohli, Stephen Bates, Michael Jordan, Jitendra Malik, Thayer Alshaabi, Srigokul Upadhyayula, and Yaniv Romano. Image-to-image regression with distribution-fres uncertainty quantification and applications in imaging. CoRR, abs/2202.05265,
2022. httttt:/s///aaa org/ abs 2202. 05265
Anastasios Nikolas Angelopoulos, Stephen Bates, Michael I. Jordan, and Jitendra Malik. Uncertainty sets for image classifiers using conformal prediction. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021b. URL net forum? L_
Arthur U. Asuncion. Uci machine learning repository, university of california, irvine, school of information and computer sciences. 2007.
Rina Foygel Barber, Emmanuel Candès, Aaditya Ramdas, and Ryan J. Tibshirani. The limits of distribution conditional predictive inference. Information and Inference: A Journal of the IMA, 2020.
Rina Foygel Barber, Emmanuel Candes, Aaditya Ramdas, and Ryan J Tibshirani. Conformal prediction beyond exchangeability. arXiv preprint 4rX1v:2202.13415, 2022.
Stephen Bates, Anastasios Nikolas Angelopoulos. Lihua Lei, Jitendra Malik, and Michael I. Jordan. Distribution free,, risk controlling prediction sets, 2021. URL
2101 02703
Charles Blundell, Julien Cornebise, Koray Kavukcuoglu, and Daan Wierstra. Weight uncertainty in neural network. In Francis R. Bach and David M. Blei (eds.), Proceedings of the 32nd International Conference on Machine Learning, ICML 2015, Lille, France, 6-11 July 2015, volume 37 of JMLR Workshop and Conference Proceedings, pp. 1613-1622. JMLR.org, 2015. URL hhttp::/proceeeeee ngs m www.prrss/vv.. https//////bu/eele/// 11..html1l
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agar- wal, Ariel Herbert-Vooe Voss, Gretchen Krueger. Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCan- dlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In Hugo Larochelle, Marc' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan- Tien-Tienn Lin (eds.). Advances in Neural Information Processing Systems 33: Annual Conference Neural Information Processing Systems 2020, NeurlPS 2020, December 6-12, 2020, virtual, 2020. URL https:tppp.... http:/taa////occoaniiceeooooo/ee wtwp:rowwwwciiio.o....o.eeeeeeee new.ips.r.pp... http://eeeeeeccaoaooooooc//////a 7457c00d 6b 44 8b
Abst html
10

---

Published as a conference paper at ICLR 2023
Krisztian Buza. Feedback prediction for blogs. In Data analysis, machine learning and knowledge discovery. pp. 145-152. Springer, 2014.
Emmanuel J Candès, Lihua Lei, and Zhimei Ren. Conformalized survival analysis. arXiv preprini arXiv:2103.09763, 2021.
Yanzhi Chen, Dinghuai Zhang. Michael U. Gutmann, Aaron Courville, and Zhanxing Zhu. Neural approximate sufficient statistics for implicit models. In International Conference on Learning Representations, 2021.
Joel W. Cohen, Steven B. Cohen, and Jessica S. Banthin. The medical expenditure panel survey: A national information resource to support healthcare cost research and inform policy and practice. Medical Care, 47:S44-$555 2009.
Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urban scene understanding. In Proc. of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, K. Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. 2009 IEEE Conference on Computer Vision and Pattern Recognition, pp. 248-2588 2009.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: pre-training of deep bidirectional transformers for language understanding. In Jill Burstein, Christy Doran, and Thamar Solorio (eds.), Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume I (Long and Short Papers), pp. 4171- 4186. Association for Computational Linguistics, 2019. doi: 10.18653/v1/n19-444 URL https://doi.org/ wtt.:://1r.11113311111113133 1423
Jakob Gawlikowski, Cedrique Rovile Njieutcheu Tassi, Mohsin Ali, Jongseok Lee, Matthias Humt, Jianxiang Feng, Anna M. Kruspe, Rudolph Triebel, Peter Jung, Ribana Roscher, Muhammad Shahzad, Wen Yang, Richard Bamler, and Xiao Xiang Zhu. A survey of uncertainty in deep neural networks. CoRR, abs/2107.03342 2021. URL http:: /tattts://axivv.oo
210. 03342
Asaf Gendler, Tsui-Wei Weng, Luca Daniel, and Yaniv Romano. Adversarially robust conformal prediction. In International Conference on Learning Representations, 2022. URL https
//openreviews eww net. wow.mmiddd"" Bs 4wP1
Isaac Gibbs and Emmanuel Candès. Adaptive conformal inference under distribution shift. In NeurlPS, 2021.
Leying Guan. Conformal prediction with localization. arXiv preprint arXiv: 08558, 2019.
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger. On calibration of modern neural networks. In Doina Precup and Yee Whye Teh (eds.), Proceedings of the 34th International Conference on Machine Learning, ICML 2017, Sydney, NSW, Australia, 6- August 2017, vol- ume 70 of Proceedings of Machine Learning Research, pp. 1321-1330. PMLR, 2017a. URL http::/proceedinng www.mmrrrrsssse. prwss/ vv0 http:///o//uuooa.oo.....
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. On calibration of modern neural networks. ArXiv, abs//770.00459) 2017b.
Xing Han, Ziyang Tang, Joydeep Ghosh, and Qiang Liu. Split localized conformal predictialpeedcciinnn CoRR, abs/2206. 13092, 2022. doi: 10.48550/arxiv.2206. 13092 URL /tR/ 48550/arXiv. 2206. 13092
Robert M. Haralick and Linda G. Shapiro. Image segmentation techniques. Comput. Vis. Graph. Image Process., 29(1):100-132 1985, doi: 10.1016//0734- 10..006/507344180005555555 (890085).. URL https
(85) 90153-7

---

Published as a conference paper at ICLR 2023
José Miguel Hernández-Loben and Ryan P. Adams. Probabilistic backpropagation for scalable learning of bayesian neural networks. In Francis R. Bach and David M. Blei (eds.), Proceedings of the 32nd International Conference on Machine Learning. ICML 2015, Lille, France, 6-11 July 2015, volume 37 of JMLR Workshop and Conference Proceedings, pp. 1861-1869. JorR. 2015. URL
ngs m wtt.innn........ee..e..e www.ppess//v///eeneeaeee
lobatoc15 htmll
Xiaoyu Hu and Jing Lei. A distribution-fre free test of covariate shift using conformal prediction. arXiv: Methodology. 2020.
Rafael Izbicki, Gilson Shimizu, and Rafael B Stem. Cd-split and hpd-split: efficient conformal regions in high dimensions. arXiv preprint arXiv:2007.12778, 2020a.
Rafael Izbicki, Gilson T. Shimizu, and Rafael Bassi Stern. Distribution-fres conditional predictive bands using density estimators. ArXiv, abs/ 1ets/190000555, 2020b.
Volodymyr Kuleshov, Nathan Fenner, and Stefano Ermon. Accurate uncertainties for deep learning using calibrated regression. In Jennifer G. Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference on Machine Learning, ICML. 2018, Stockholmsmässsan Stockholm, Swe- den, July 10-15, 2018, volume 80 of Proceedings of Machine Learning Research, pp. 2801 2809. PMLR, 2018. URL
mir.
1a..mmlll........ html
James Law. Review of "algorithmic learning in random world by vovk, gammerman and shafer". springer, 2005, ISBN: 0-387-00152-2. SIGACT News, 37(4):38 40, 2006. doi: 10.1145/1189066 1189065. URL http::::ts/......... / org/ 10. 1145/155/1111 /11//11885566. 1189065
Jing Lei, Alessandro Rinaldo, and Larry A. Wasserman. A conformal prediction approach to explore functional data. Annals of Mathematics and Artificial Intelligence, 74:29-43 2013.
Jing Lei, Max G'Sell, Alessandro Rinaldo, Ryan J Tibshirani, and Larry Wasserman. Distribution-free predictive inference for regression. Journal of the American Statistical Association, 113(523): 1094-1111,220 2018.
Lihua Lei and Emmanuel J. Candès. Conformal inference of counterfactuals and individual treatment effects. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 83, 2021a.
Lihua Lei and Emmanuel J Candès. Conformal inference of counterfactuals and individual treatment effects. Journal of the Royal Statistical Society: Series B (Statistical Methodology). 2021b.
Yingzhen Li and Yarin Gal. Dropout inference in bayesian neural networks with alpha-divergencer In Doina Precup and Yee Whye Teh (eds.), Proceedings of the 34th International Conference Machine Learning. ICML 2017, Sydney, NSW. Australia, 6-11 August 2017, volume 70 of Proceedings of Machine Learning Research, pp. 2052-20611 PMLR, 2017. URL http://
proceedings www.ang.smm....mmm www..mllr......... wtt.:..pp.es////11111111
Shervin Minace, Yuri Boykov, Fatih Porikli, Antonio Plaza, Nasser Kehtarnavaz, and Demetri Terzopoulos. Image segmentation using deep learning: A survey. CoRR, abs/2001.05566, 2020.
hbtp://bs//00000 05566
Matthias Minderer, Josip Djolonga, Rob Romijnders, Frances Ann Hubis, Xiaohua Zhai Neil Houlsby, Dustin Tran, and Mario Lucic. Revisiting the calibration of modern neural networks. In NeurIPS, 2021.
Jeremy Nixon, Michael W. Dusenberry, Linchuan Zhang. Ghassen Jerfel, and Dustin Tran. Measuring calibration in deep learning. In IEEE Conference on Computer Vi- sion and Patlern Recognition Workshops, CVPR Workshops 2019, Long Beach, CA, USA, June 16-20, 2019, pp. 38-41. Computer Vision Foundation / IEEE, 2019. URL http: //openaccess.o theevf com wwwo//ooneet
CVPRN 2019/stml/nneeraaitt and
Robustness in Deep Visualliiiaalaeeeee Learning/Niieeieeeeein
in in_Deep_Leare Learni cetaaniing_cvi cvw_p www.w//www...... 2wt00111ppppe_pm_m_m html
Ilia Nouretdinov, Sergi G. Costafreda, Alexander Gammerman, Alexey Ya. Chervonenkis, Vladimir Vovk, Vladimir Vapnik, and Cynthia H. Y. Fu. Machine learning classification with confidence:
Application of transductive conformal predictors mri- mri-based diagnostic and prognostic markers in depression. Neurolmage. 86(2) 2011. doi: 10. 10.10166j wwuroimage. 201..00.......0002.... URL
https://do! .org 1016/ neuro mage. 2010. 050033
12

---

Published as a conference paper at ICLR 2023
Harris Papadopoulos, Vladimir Vovk, and Alexander Gammerman. Regression conformal prediction with nearest neighbours. J. Artif. Intell. Res., 40:815-8444 840, 2011. URL /
papeess/paerr318..ttmm
Adam Paszke, Abhishek Chaurasia, Sangpil Kim, and Eugenio Culurciello. Enet: A deep neural network architecture for real-time semantic segmentation. arXiv preprint VrXiv:1606.0214. 2016.
Aleksandr Podkopaev and Aaditya Ramdas. Distribution-fiet uncertainty quantification for classifi- cation under label shift. In UAI, 2021.
Yaniv Romano, Evan Patterson, and Emmanuel J. Candès. Conformalized quantile regression. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d' Alché- Buc, Emily B. Fox, and Roman Gamett (eds.), Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurlPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pp. 3548-3548, 2019. URL https: http:::://ppreeeeee rgs neurips wctccc/pppeee cc/paper/ Abbsttttttttt-baaaatat.t..... htmi
Yaniv Romano, Matteo Sesia, and Emmanuel Candès. Classification with valid and adaptive coverage. In Hugo Larochelle, Mare' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin (eds.), Advances in Neural Information Processing Systems 33: Annual Conference Neural Information Processing Systems 2020, NeurlPS 2020, December 6-12, 2020, virtual, 2020a. URL /RR/attpa://pooeeeeingg htwp://wwweeiiinnn..nii.......e. nwwrips wttpp//pppp//////aaaaaaaaaaae (0l.eaaeeed..10000666660606
.tml
Yaniv Romano, Matteo Sesia, and Emmanuel J. Candès. Classification with valid and adaptive coverage. arXiv: Methodology, 2020b.
N. Senthilkumaran and Reghunadhan Rajesh. Image segmentation A survey of soft computing approaches. In ARTCom 2009, International Conference on Advances in Recent Technologies in Communication and Computing, Kottayam, Kerala, India, 27-28 October 2009, pp. 844 846. IEEE Computer Society, 2009, doi: 10. 111109/AATTCmm.20000000 URL https::/doi...... org/
10. www11119AACoott http//ARCCooo...0..... 2009.219
Matteo Sesia and Emmanuel Candès. A comparison of some conformal quantile regression methods. Stuf, (11)=2261, 2020.
Mattco Sesia and Yaniv Romano. Conformal prediction using conditional histograms. In Mare' Aurelio Ranzato, Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan (eds.), Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 202 NeurlPS 2021, December 14, 2021, virtual, pp.
6304-6315, 2021. URL Itttps://proceeeiinge http:://ww....rrrooooor//rrp///e ct/ppapee//222//aaaa..
3313331aa22233337000661111222221 Abstaaaa. .tmll
Glenn Shafer and Vladimir Vovk. A tutorial on conformal prediction. Mach. Learn. Res., 371-421, 2008. URL htthhttps:/dd.
wtt.://///..aaat........... 33969999333
David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, Yutian Chen, Timothy P. Lillicrap, Fan Hui, Laurent Sifre, George van den Driessche, Thore Graepel, and Demis Hassabis. Mastering the game of go without human knowledge. Not., 550(7676):354 359, 2017. doi: 10. 00.103//ature222270..
Ralph C. Smith. Uncertainty Quantification - Theory, Implementation, and Applications. Com- putational science and engineering. SIAM, 2014. ISBN 978-1-661173-21-1. URL http:
//bookszore. siam. crg/
David Stutz, Krishnamurthy Dvijotham, Ali Taylan Cemgil, and A. Doucet. Learning optimal conformal classifiers. ArXiv,
2021.
Timothy John Sullivan. Introduction uncertainty quantification, volume 63. Springer, 2015.
13

---

Published as a conference paper at ICLR 2023
Jiaye Teng, Zeren Tan, and Yang Yuan. T-SCI: A two-stage conformal inference algorithm with guaranteed coverage for cox-mlp. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference Machine Learning. ICML 2021, 18-24 July 2021. Virtual Event, volume 139 of Proceedings of Machine Learning Research, pp. 10203-10213. PMLR, 2021. URL
http:://rroceedings ngs ml wwwps/sess///// /vtaas///v1tt/te//leeeee html
Ryan J. Tibshirani, Rina Foygel Barber, Emmanuel Candès, and Aaditya Ramdas. Con- formal prediction under covariate shift. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d' d'A, Emily B. Fox, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Pro- cessing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pp. 2526 2019. URL https: httptttttttttpeeeeeeeeeeeeeteeet http:///ccccececc.ceeeccscc wewreiinnnnn................ 8 3r22ueee222212225555522222222222 ottp:///ss.ooooooooo////////////
.htll
Vladimir Vovk, Alexander Gammerman, and Glenn Shafer. Algorithmic learning in random world. Springer Science Business Media, 2005.
Chen Xu and Yao Xie. Conformal prediction interval for dynamic time-series. In ICML, 2021.
Kaidi Xu, Zhouxing Shi, Huan Zhang, Yihan Wang, Kai- Rei Chang, Minlie Huang, Bhavya Kailkhura, Xue Lin, and Cho-Jui Hsieh. Automatic perturbation analysis for scalable certified robustness and beyond. In Hugo Larochelle, Marc' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan- Tien Lin (eds.), Advances in Neural Information Processing Systems 33: An- nual Conference on Neural Information Processing Systems 2020, NewrlPS 2020, December 6- 12. 2020, virtual, 2020. URL https: http://ppp/ppppceeedeeeeeoeeee neurips wwwppppppp/////////cccc (ac.ae26666111111111............ -ww.:leeeeeaaa.taaaaaaaaaaeeaeee html
Yachong Yang and Arun Kumar Kuchibhotla. Finite-sample efficient conformal prediction. arXiv preprint arXiv:2104.1383 2021.
14

---

Published as a conference paper at ICLR 2023
Appendix
Section A provides the complete proofs, and Section B.1 provides experiment details.
A THEORETICAL PROOFS
We first show the formal version of Theorem in Section A.1 Theorem 5 and Theorem showows the effectiveness (empirical coverage) and the efficiency (band length) in Feature CP.
and show its proof in Section A.2
We additionally provide Theorem 9
(see Section A.3 and Theorem 15 (see Section A.4 to better validate our theorem, in terms of the length variance and convergence rate.
A.1 THEORETICAL GUARANTEE
This section provides theoretical guarantees for Feature CP regarding coverage (effectiveness) and band length (efficiency), starting from additional notations.
Notations. Let P denote the population distribution. Let De. ~ Pn denote the calibration set with sample size n and sample index Tea, where we overload the notation pa to denote the distribution of a set with samples drawn from distribution P. Given the model ğ o f with feature extractor f and prediction head ÿ. we assume is continuous. We also overload the notation QV-a(V) to denote the (1 a) -uuuntle of the set V U {co). Besides, let M||] denote the mean of a set, and a set minus a real number denote the broadcast operation.
Vanilla CP. Let VB_. = {uf) hez. denote the individual length in the output space for vanilla CP, given the calibration set Dea- Concretely, of = 2/y; - ğil where y denotes the true response of sample and iJ denotes the corresponding prediction. Since vanilla CP returns band length with 1- a quantile of non-conformity score, the resulting average band length is derived by Q1-a (VB..).
Let V.d. = {uf JiET. be the individual length (or diameter in high dimensional cases) in the feature space for Feature CP, given the calibration set Dea- To characterize the band length in the output space, we define H(v, X) the individual length on sample X in the output space, given the length v in the feature space, i.e. H(v X) represents the length of the set (ō(u) R f|u- < v/2]. Due to the continuity assumption function ÿ. the above set is always simply- cinpel.connected. We here omit the dependency of prediction head ğ in H. for simplicity. The resulting band length in Feature CP is denoted by E
Feature CP.
((11-a
), X'). Without abuse of notations, operating H
on a dataset (e.g.. SVV. Daa)) means operating H on each data point (o .X,) in the set.
Coverage guarantee. We next provide theoretical guarantees for Feature CP in Theorem 5 which informally shows that under Assumption [] the confidence band returned by Algorithm 3 is valid, meaning that the coverage is provably larger than 1 a. We defer the whole proof to Appendix
Theorem 5 (theoretical guarantee for Feature CP). Under Assumption for any 0, the confidence band returned by Algorithm 3 satisfies:
P(Y E (()
where the probability is taken over the calibration fold and the testing point (X', Y").
Length (efficiency) guarantee. We next show in Theorem 6 that Feature CP is provably more efficient than the vanilla CP.
Theorem 6 (Feature CP is provably more efficient). Assume that the non-conformity score is in norm-type. For the operator H. we assume a Holder assumption that there exist a > 0, L > 0 such that [H(v,, X)- -(u,, X)) L|u - ul for all X. Besides, we assume that there exists 0, such that the feature space satisfies the following cubic conditions:
1. Length Preservation. Feature CP does not cost much loss in feature space in quantile manner, namely, E ED (1o (HV(f D)) < EDDppa Q Q1-a (VB) + c.
15

---

Published as a conference paper at ICLR 2023
2. Expansion. The operator H(,, X) expands the differences between individual length and their quantiles, namely, 1B5D-p-MMQQQ--- (V-) V-210 <Ep--p.NilC)--a(((((
H(VD, D)] -c -- max [c/ min.
3. Quantile Stability. Given calibration set Dea, the quantile of the band length is stable in both feature space and output space, namely, EDupp (01 (Q--((/)) - Q (V S and Ep-pp (1-aa= QV 5 vn
Then Feature CP provably outperforms vanilla CP terms of average band length, namely.
(V)))))((( < QV
where the expectation is taken over the calibration fold and the testing point (X', YY).
The cubic conditions used in Theorem 6 sketch the properties of feature space from different aspects. The first condition implies that the feature space is efficient for each individual, which holds when the band is generally not too large. The second condition is the core the proof, which informally assumes that the difference between quantile and each individual is smaller in feature space. Therefore, conducting quantile operation would not harm the effectiveness (namely, step 4 in Algorithm| and step 4 in Algorithm 3) leading the efficiency of Feature CP. The last condition helps generalize the results from the calibration set to the test set.
Proof of Theorem
We start the proof with Assumption 2, which claims that
LEDM(QI-a(V)) ViS1" <EpM Q1- (HLV,, H(VV, D)
-c - 2 max L.
We rewrite it as EpMH(V D) <EDQ1-a (H(V)) D)) -
-2 2maxx (///c/////mmm//a/. LEAMi]Qlaa - Vfl".
Due to Holder condition, have that MH((1-a(VV))))) M(H(V] D)) LM Vifia, therefore
EpM [(Q1 D)] EoQo-a D)) - e max L// min([l.)
Therefore, due assumption 1, we have that
EpMH((1-a(VV).D)) < ExQ1-a(V))
/ii 1.a
Besides, according to the quantile stability assumption, we have that Ep (M MVH( D)I s Llc/ vn]a, and Eo(Qi-a - Q13)(V3) s c/ Vn. Therefore,
EH( (V/ X')
= EDMPH (V. D.N D)
<Q1aa (VB.) max(1, Le min L/c/v/ tc//n
< <Q1-a
A.1.1 EXAMPLE FOR THEOREM 6
This section provides an example for Theorem 6 The key information is that Feature CP loses less efficiency when conducting the quantile step.
16

---

Published as a conference paper at ICLR 2023
Table A concrete example for the comparison between Feature CP and CP. Let Lo. IL, denote the individual length in the feature and output space. Let Q(-) denote the quantile operator, and H-(-) denote the operator that calculates output space length given the feature space length. We remark that the average band length returned by Feature CP (3.1) outperforms that of vanilla CP (4.0).
METHOD
VANILLA CP
FEATURE CP
SAMPLE
1Lo
Q(ILo)
1L,
H(ILs)
Q(IL))
H(Q(IL;))
1.0
4.0
1.1
1.2
1.3
1.4
B
2.0
4.0
1.2
2.1
1.3
2.3
3.0
4.0
1.1
2.8
1.3
3.1
D
1.3
3.8
1.3
3.8
5.0
4.0
1.6
5.2
1.3
4.9
QUANTILE AVERAGE
1.3
4.0
1
/
3.1
Assume the dataset has five samples labeled A, B, C, D, and When directly applying vanilla CP leads to individual length in the output space IL, 3,2,3,4,3 5,5, respectively. By taking 80% quantile (namely, a = 0.2), the final confidence band returned by vanilla CP (Q(LLo)) would be Qo.s 2, 3, 4, 5}) 4. Note that for any sample, the returned band length would be and the final average band length 4.
We next consider Feature CP. We assume that the individual length in the feature space (IL,) is 1.1, 1.2, 1.1, 1.3, 1.6, respectively. Due to the expansion condition (cubic condition #2). the difference between JL, and ((IL)) is smaller than that between Ih, and Q(IL.). Therefore, the quantile step costs less in Feature CP. Since I[, is close to Q(ILy). their corresponding output length H(LL,). ((L()))) are also close. Besides, to link conformal prediction and vanilla CP, the length preservation condition (cubic condition #1) ensures that L,, is close to H(LLy). Therefore, the final average length M.{(((L,)) is close the average length MIL,, which is better than Q(II.)) Finally, the quantile stability condition (cubic condition #3) generalizes the results from the calibration set the test set.
A.2 PROOF OF THEOREM 5
Theorem 5 (theoretical guarantee for Feature CP). Under Assumption for any 0, the confidence band returned by Algorithm satisfies:
P(Y' E (X'))
2
where the probability is taken over the calibration fold and the testing point (X', Y").
Proof of T'heormm 5 The key to the proof to derive the exchangeability the non-conformity score, given that the data in the calibration fold and test fold exchangeable (see Assumption
For ease of notations, we denote the data points in the calibration fold and the test fold as D' = {(X,, bE) where m denotes the number of data points in both calibration fold and test fold. By Assumption
[ the data points in D' exchangeable.
The proof can be split into three parts. The first step is to show that for any function independent of D'. h(X,, Y,) exchangeable. The second step is to show that the proposed score function satisfies the above requirements. And the third step is to show the theoretical guarantee based on the exchangeability of the non-conformity score.
We next prove the first step: for any given function h X' x y - R that is independent data points in D', we have that by (X,, Y,) exchangeable. Specifically, its CDF F, and its perturbation CDF
17

---

Published as a conference paper at ICLR 2023
FJ is the same, given the training fold Du-
Fo Un (2))
-(( Yi) ut, u,,..., -,..,,,,(( (X,: Yn) S un Dt.)
=P((X,, Yi) E (u)
(Xn, Yn) E C₁ 1 (up-)) ((r))
-P((X=(1)) (<(1))))))())) E C (u1) (Xr(n)) X((n) (un P(r)
=P()(Xx(1)> Vx(1)) S :.:..... ((xx(()) Vz(n)) un D(r)
=F (F1, Un Du),
where π denotes random perturbation, and ( ((X,) u..
The second step is to show that the proposed non-confornity score function (See Equation (3) and Algorithm 3) is independent of the dataset D To show that, we note that the proposed score function s in Equation 3 (we rewrite in Equation (5) is totally independent of dataset D', in that we only use the information of f and 9 which is dependent on the training fold Drr instead of D'
8 Y
inf ||0 - f(X)]l-
(5)
-E(e:(()=Y))
Besides, note that when calculating the non-conformity score in Algorithm 3 for each testing data/calibration data, do not access any information on the calibration folds for any other points, Therefore, the score function does not depend on the calibration fold or test fold. We finally remark that here we always state that the score function does not depend on the calibration fold test fold, but its realization s sXX, å f' can depend on the folds, if (X, Y) D', This does contrast with the requirement in the first step.
Therefore, combining the two steps leads to a conclusion that the non-conformity scores on D' are exchangeable. Finally, following Lemma 1 in Tibshirani et al. (2019), the theoretical guarantee holds under the exchangeability of non-conformity scores.
A.3 LENGTH VARIANCE GUARANTRE
The next Theorem 9 demonstrates that the length returned by Feature CP would be individually different. Specifically, the variance for the length is lower bounded by a constant. The essential intuition is that, for non-linear function g. the feature bands with the same length return different bands in output space. Before expressing the theorem, we first introduce a formal notation of length and other necessary assumptions. For of discussion, define in Definition a type of band length slightly different from the previous analysis. We assume Y E R below, albeit our analysis can be directly extended high-dimensional cases.
Definition 7 (band length). For given feature u and any perturbation e E C)(r) tē - wl S Q} in the feature band, we define the band length in the output space Lo(v) as the maximum distance between predictor g(v) and g(t), namely
Lo(v) A max TEC
(g(ū) - (dv))
Besides, we require Assumption 8 which is about the smoothness of the prediction head g.
Assumption 8. Assume that the prediction head is second order derivative and M M-smooth, namely, [[V*g(u)]| < .M for all feasible u.
The following Theorem 9 indicates that the variance of the band length is lower bounded, meaning that the bands given by Feature CP are individually different.
Theorem Under Assumption 8 if the band on the feature space is with radius Q, then the variance of band length on the output space satisfies:
E LL ELo]² /0² 2 [[lV Vo(v)l| MQEI[
From Theorem the variance of the band length has a non-vacuout lower bound if
B
V MQ TE [[[])())]]
(6)
18

---

Published as a conference paper at ICLR 2023
We next discuss the condition for Equation ( For linear function g. note that [
E|| Vg((]]] = and M = 0. thus does not meet Equation (6) But for any other non-linear function 9, least have E[] Vg(v)]l - [+))] > 0 and M >> O, and therefore there exists term Q
such that Equation (6) holds. Hence, the band length in feature space must be individually different for non-linear function 9 and small band length Q.
Proof of Theorem 9 We revisit the notation in the main text, where v = f(X) denotes the feature, and (v() = (ū=
i w S Q} denotes the confidence band returned in feature space. By Taylor Expansion, for any given 7 E (w)) there exists v' such that
g(ū) - g - (i(v)(ū v) + 1/2(6 -u) V'''((''
Due to Assumption 8
g>''(\ - M. Therefore, for any 6 (v)
1 1/2((i - v) """ -
1 MQ².
On the one hand, by Cauchy Schwarz inequality, we have
Lo max (g( g(0) = [V + 2 MQ².
On the other hand, by setting i - = (V((v) we have that
Lo = max (g(0) - g(v) 2 1g(v QVg(v)/\V(())) - - 1/2MQ²
Therefore, we have that
IL. - Q(Va(u)II - 1/2MQ².
We finally show the variance of the length, where the randomness is taken over the data v,
[[L,- EL,1
= -1
+ 2E +=
2 -
- -2081117440---V1066001000-0-000-
B(L, (V)
2 - 3 [Vy
Besides, note that BV(())
Therefore, we have that E The - R.I, 2 //22 E (V)( - V MMQ
A.4 THEORETICAL CONVERGENCE RATE
In this section, we prove the theoretical convergence rate for the width. Specifically, we derive that when the number of samples in the calibration fold goes to infinity, the width for the testing point converges to a fixed value. Before we introduce the main theorem, we introduce some necessary definitions. Without further clarification, follow the notations in the main text. Definition 10 (Precise Band). We define the precise band as
Cpr {g(v) || b|| S Qi-a).
Definition 11 (Precise Exact Band). We define the exact precise band
cTT {g(v) ||v i|1 S Qi-a),
(8)
where Q -a denotes the exact value such that
P(Zc ||u
il| < Qi-a:(()) = y) =1-
19

---

Published as a conference paper at ICLR 2023
Our goal is to prove that the band length (volume) of Cr (denoted by v(C)-a) converges to V(CI=a) We assume that the prediction head and the quantile function are both Lipschitz in Assumption and Assumption 13
Assumption 12 (Lipschitz for Prediction Head), Assume that for any v, o', we have (19(e) - g(0')ll S -1llu- r'Il.
Assumption 13 (Lipschitz for Inverse Quantile Function). Denote the quantile function as Quantile (Q.) IP(3c: ||0 - ö|| I Qu. = y) = u.
We assume that its inverse function is Lp-lipschitz that is to say,
||Quattle [u) Quantile (u''ll s L2Ilu u'l
Besides, we assume that the region of Ci has benign blow-up.
Assumption 14 (Benign Blow-up). Assume that C has benign blow-up, that is say, for the blow-up set (c) {e 3u CI I| 21| S E}. we have
[I((T=a((e)) V(CT- S.e,
where с denotes constant independent n.
In the one-dimensional case Y R. Assumption 14 easily holds. For the high-dimensional cases, such a bound usually requires that e depends on the dimension d.
Theorem 15 (Convergence Rate). Assume that the non-conformity scores the calibration fold have no lies. Under Assumption 72 Assumption 13 and Assumption [4 we have that
[IV(CT= -V(Cr=a)l S 2L1
Proof. Firstly, as derived in Romano et al. (2019), when the non-conformity score in the calibration fold has no ties (the probability is zero), have
P(5e: ||v i|| < Q1-a ((v) y) E a, a + 1/n),
(10)
where v, Q1-a denotes the surrogate feature, the trained feature, and the quantile value in Algo- rithm 3 respectively.
By Assumption 13 that the inverse quantile function I -iiischht around a, we have
ll - QI-all S L2 n
Therefore, for any E CTT there exists u' E CiTH such that
1
||u n' - ([g(v) -(lo)-gg'')l Tallu v'll 12
(11)
We note that bounding |/ o'|l requires that the region of v, v' are both balls, and therefore one can select o' the point with the smallest distance Since the region of C.T has benign blow-up, we have that
1
V(CTT.) S V(CT=a) + eL, L2
n
Besides, the following equation naturally holds due to Equation
10
(C 2 V(C
Therefore, conclude with the following inequality,
|V(CT==))) V(C S 221
Therefore, the sample size in the calibration fold goes to infinity, the length of the trained band converges to (C
20

---

Published as a conference paper at ICLR 2023
g
Linear Linear Linear Linear
X
Y
64
64
64
64
Figure 5: The model architecture of the uni-dimensional and synthetic multi-diimensionll target regression experiments. The dropout layers are omitted.
Conv
Conv
Image X
Bilinear
ResNet50
Interpolate Y
Figure 6: The model architecture of the semantic segmentation experiment.
B EXPERIMENTAL DETAILS
Section B.I introduces the omitted experimental details. Section B.2 provide experimental evidence to validate cubic conditions. Section B.3 shows that Feature CP performs similarly vanilla CP for untrained neural networks, validating that Feature CP works due to semantic information trained in feature space. Section B.4 introduces Feature CQR which applies feature-level techniques CQR and Section (B.5)reorrrrrrr the corresponding group coverage. Finally, Section B.9 provides other additional experiments omitted in the main text.
B. EXPERIMENTAL DETAILS
Model Architecture. The model architecture of the uni-dimensional and synthetic multi-dimennional target regression task is shown in Figure The feature function and prediction head g includes two linear layers, respectively. Moreover, the model architecture of the FCN used in the semantic segmentation experiment is shown in Figure which follows the official implementation Py Torch. The batch normalization and dropout layers are omitted in the figure. We use the ResNet50 backbone as f and take two convolution layers as g. We select the Layer4 output of ResNet50 as our surrogate feature v.
Training protocols. In the unidimensional and synthetic dimensional target regression experiments, we randomly divide the dataset into training, calibration, and test sets with the proportion 2 2 As for the semantic segmentation experiment, because the labels of the pre-divided test set are not accessible, we re-split the training, calibration, and test sets randomly Cityscapes. We remove the class (unlabeled) from the labels during calibration and testing, and use the weighted mean square error the training objective where the class weights are adopted from Paszke et al. (016))
the original training set of
Band Estimation. In estimating the band length, deploy Band Estimation based on the score In our experiment, we choose the number of steps M in Algorithm 2 via calculated by Algorithm(2 2 cross-validation.
21

---

Published as a conference paper at ICLR 2023
Table 4: Comparison between CP and Feature CP.
METHOD
CP
FEATURE CP
DATASET
COVERAGE
LENGTH
COVERAGE
LENGTH
COMMUNITY
89.82 +0.95
1.99=0.09
89.62
1.99
FACEROOK
90.11 40.33
3.17 40.59
90.07 40.32
2.08 10.17
FACEROOK2
89.99 10.18
2.72 #0.37
89.98 10.17
1.97
10.17
MEPS 19
90.51 +0.21
4.25 #0.21
90.55 10.17
3.58
#0.35
MEPS20
89.80 10.61
4.17 10.36
89.76 10.61
3.37 20.38
MEPS21
89.92 10.54
4.67 10.30
89.94 10.54
3.93
10.54
STAR
90.30 1.117
1.002
90.35 10.99
0.41 10.03
BIO
90.27 20.26
1.97 #0.01
90.20 #0.29
1.87 =0.04
BLOG
90.08 +0.27
3.32 +0.38
90.06 +0.33
2.81 #0.40
BIKE
89.65 +0.87
1.91 #0.04
89.61 +0.69
1.79 =0.08
Randomness. We train each model five times with different random seeds and report the mean and standard deviation value across all the runs the experimental results (as shown in Figure 2 and Table
Details of transforming segmentation classification problem into regression task. The original semantic segmentation problem to fit the one-hot label whose size is (C. W, H) via logistic regression, where C is the number of the classes, W and H are the width and height of the image. We use Gaussian Blur to smooth the values in each channel of y. At this time, the smoothed label ý ranges from 0 to Then, we the double log trick convert the label space from [0, to (-o0, oo), i.e., ý = log(- log(ū)). Finally, we use mean square error loss to fit ÿ.
Definition of weighted length. We formulate the weighted length
weighted length-
Σ Σ JE[d]
w (((:))
[Te
SEI,
where (0()) the corresponding weight in each dimension. We remark that although the formulation of w!!) is usually sample-dependent, omit the dependency of the sample and denote it by wG) when the context is clear. We next show how define w(1) in practice.
Generally speaking, we hope that w is large when being informative (i.e., in non-boundary regions). Therefore, for the j-th pixel after Gaussian Blur whose value is y() E [0, 1], its corresponding weight is defined
12 W
wi
E 10,
where W = E, 2)
is scaling factor.
At colloquial level, (i) is close if yu) is close to 0 or 1. In this case, y being close 0 or 1 means that the pixel is far from the boundary region. Therefore, the weight indicates the degree to which pixel is being informative (not in object boundary regions).
Calibration details. During calibration, to get the best value for the number steps M, we take subset (one-fifth) of the calibration set the additional validation conformity score on the rest of the calibration set with various values of step M and then evaluate on the validation set to get the best M whose coverage is just over 1 a. The final trained surrogate feature is close to the true feature because ((v) is sufficiently close to the ground truth Y. In practice, the surrogate feature after optimization satisfies [[ā(w) |
We caleulate the non-
Y
1%.
Comparison between Feature CP and Vanilla CP. We next present the specific statistics figure in Table and 7
22

---

Published as a conference paper at ICLR 2023
Table 5: Validate cubic conditions.
SPACE
FEATURE SPACE
OUTPUT SPACE
METRIC
MQQ V.L.. V.lll MIQ H(VV H(V
COMMUNITY
0.1150 10.0290
0.807310.
FACEBOOK]
0.2491 10.0391
1.8950 1..20588
FACEBOOK2
0.2387 1.009660
1.7918 10.5220
MEPSI9
0.2403 +0 0161
1,7511 +0.1150
MEPS20
0.2485 +0.0571
1.7936 +0.3532
MEPS21
0.2528 30,0488
1,8686 +0.3350
STAR
0.0230 10.0025
0.1605 #0.0123
0.1051 40.0056
0.8509 +0.0368
BLOG
0.3537 #0. 1335
2.3769 +0.5421
BIKE
0.0921 +0.0058
0.7759 10.0469
bio
bike
blog
1.00
6.75
f:1 9:3
1:1 9:3
9:3
2.65
g:2
6.30
g:2
g:2
cubic metfic
2.50-
g:1
cubic metric
0.72
g:1
cubic metric
2,14
g:1
6.72
3.88
0.72
3.00-
0.70
20-
0.72
1.80
1.20
1.44
1.55
1750
2.775
1.002
Ldbs
L850 length
1805
2.00
1.223
2.5
2.8
25
3.8 length
21
length
Figure 7: Length v.s. cubic metric. Larger cubic metric implies better efficiency (shorter band).
B.2 CERTIFYING CUBIC CONDITIONS
In this section, we validate the cubic conditions. The most important component for the cubic condition is Condition 2, which claims that conducting the quantile step would not hurt much efficiency. We next provide experiment results in Table 5 on comparing the average distance between each sample to their quantile in feature space MIQi-vV/ - V/_.1 and output space MI[Q H(VV Dea) H(V Dea)]. We here take a 1 for simplicity. The significant gap in Table validates that the distance in feature space is significantly smaller than that in output space, although we did not consider the Lipschitz factor for computational simplicity.
Besides, we plot the relationship between the efficiency (band length) v.s. cubic metric in Figure 7 Specifically, eubic metric here represents the core statement in cubic condition (statement 2), which implies metric form like M|Q1-a VV. - -V-ll-
The results are shown in Figure
B.3 FEATURE CP WORKS DUE TO SEMANTIC INFORMATION IN FEATURE SPACE
Experiment results illustrate that feature-level techniques improve the efficiency of conformal predic- tion methods (e.g., Feature CP vs. CP, Feature CQR vs. CQR). We claim that exploiting the semantic information in feature space is the key to our algorithm. Different from most existing conformal prediction algorithms, which regard the base model as a black-box mode, feature-level operations allow seeing the training process via the trained feature. This novel and greatly broadens the scope of conformal prediction algorithms. For well-trained base model, feature-level techniques improve efficiency by utilizing the powerful feature embedding abilities of well-trained neural networks.
In contrast, if the base model is untrained with random initialization (whose representation space does not have semantic meaning), Feature CP returns a similar band length as the baseline (see Table 6)) This validates the hypothesis that Feature CP's success lies in leveraging the inductive bias of deep representation learning. Fortunately, realistic machine learning models usually contain meaningful information in the feature space, enabling Feature CP perform well.
23

---

Published as a conference paper at ICLR 2023
Table 6: Untrained base model comparison between conformal prediction and Feature CP. The base model is randomly initialized but not trained with the training fold. Experiment results show that Feature CP cannot outperform vanilla CP if the base model is not well-trained.
METHOD
VANILLA CP
FEATURE CP
DATASET
COVERAGE
LENGTH
COVERAGE
LENGTH
COMMUNITY
90.28 11.70
4.85
90.6811.33
1.92.0.777
FACEBOOK|
90.15 10.15
+0.25
90.16 10.12
3.20
+0.50
FACEBOOK2
90.17 #0.11
3.51
#0.26
90.12 #0.14
3.34
+0.39
MEPS19
90.81 +0.46
4.02
#0.16
90.86 +0.30
4.22
=0.18
MEPS20
90.10 +0.60
4.10 10.28
90.28 40.46
4.02
10.41
MEPS21
89.78
4.08 10.16
89.85 10.58
3.81
STAR
90.07 #0.77
2.23 10.18
89.47 1.84
2.24
=0.40
BIO
90.06 10.19
4.25
10.11
90.11 #0.07
4.44
#0.74
BLOG
90.13 10.34
2.41 10.15
90.16 10.26
2.58
20.49
BIKE
89.53
4.65
89.61
4.13
10.38
Algorithm 4 Feature Conformalized Quantile Regression (Feature CQR)
Require: Level a, dataset D {(X,, Y.)JEET test point X'; Randomly split the dataset D into a training fold Dur ê (X,, Yi)Ezz, together with a calibration fold Dea (X;, Yi)iez.:
2: Train a base machine learning model gا o (.)) and أhو (.) using Drr to estimate the quantile
of response Y,, which returns [ y:h1];
3: For each Iea, calculate the index cf = I( I Y) and c = I({;"i 2 Y);
4: For each E Zea calculate the non-conformity score V/ V.uc where V7 is derived on the lower bound function with Algorithm
5: Calculate the (1 a -th quantile h of the distribution | E.ez., 8Yz- 8
6: Apply Band Estimation on test data feature f'"(X') with perturbation Q and prediction head
gho which returns [C], Cla);
7: Apply STEP 4-6 similarly with higher quantile, which returns |Ca Chil;
8:
Derive Cf-.r (X) based on Equation [2];
Ensure: cleqr c())
B.4 FEATURE CONFORMALIZED QUANTILE REGRESSION
In this section, we show that feature-level techniques are pretty general in that they can be applied to most of the existing conformal prediction algorithms. Specifically, We take Conformalized Quantile Regression (CQR, [Romano et al. (2019) as an example and propose Feature-level Conformalized Quantile Regression (Feature CQR). The core idea similar to Feature CP (See Algorithm 3 where we conduct calibration steps in the feature space. We summarize the Feature CQR algorithm in Algorithm
Similar to CQR, Algorithm also considers the one- nne-dimension case where Y E R. We next discuss the steps in Algorithm 4 Firstly, different from Feature CP, Feature CQR follows the idea of CQR that the non-conformity score can be negative (see Step 4). Such negative scores help reduce the band
length, which improves efficiency. This is achieved by the index calculated in Step + Generally, the predicted value is larger than the true value y > Y,, we need to adjust 8. to be smaller, and vice visa. Step 8 follows the adjustment, where summarize the criterion in Equation (12), given the two band (Clo Clu] and [Chi Chi
Here, +1 for simplicity
24

---

Published as a conference paper at ICLR 2023
Table Comparison between CQR and Feature CQR. Feature CQR achieves better efficiency while maintaining effectiveness.
METHOD
CQR
FEATURE CQR
DATASET
COVERAGE
LENGTH
COVERAGE
LENGTH
COMMUNITY
90.33 +1.36
1,60=0.00
90.23+1.68
1.23
FACEBOOK]
89.94 +0.23
1.15 +0.04
92.00 +0.20
1.00 =0.01
FACEROOK2
89.99 +0.03
1.25 10.08
92.19 10.17
1.08 =0.06
MEPS19
90.26 +0.38
2.41 20.26
91.25 +0.43
1.48
MEPS20
89.78 +0.60
2.47 10.10
90.9 +0.39
1.34 20.35
MEPS21
89.52 +0.32
2.26 +0.20
90.2 10.53
1.69 #0.20
STAR
90.99 11.08
0.20 10.00
89.88 10.33
0.13 20.01
BIO
90.09 10.36
1.39 10.01
89.88 10.27
1.22
BLOG
90.15 10.15
1.47 10.05
91.49 10.26
0.89 20.03
BIKE
89.38 + 25
0.58 #0.01
89.95 #0.98
0.38 =0.02
if dh < 0, c < 0, return cfeer (X) = cch chi]:
if c < 0, c
0, return Ch (X) CC c
(12)
if c > 0, c,
< O, return Clear (X) = ,
if d 0, 0, return Cheer (X) Ch
Similar Feature CP, need Band Estimation step to approximate the band length used in Step 6. One can change it into Band Detection if necessary. Different from Feature CP where Band Estimation always returns the upper bound of the band, Feature CQR only approximate it. We conduct experiments to show that this approximation does not lose effectiveness since the coverage is always approximate to lower with the same value, we adjust them separately, which is more flexible in practice (see Step 7).
Besides, different from CQR, which considers adjusting the upper and
We summarize the experiments result in Table 7 Feature CQR achieves better efficiency while maintaining effectiveness. Here we provide 90% confidence band using five repeated experiments with different random seeds.
B.5 GROUP COVERAGE FOR FEATURE CONFORMALIZED QUANTILE REGRESSION
This section introduces the group coverage returned by feature-level techniques, which implies the performance conditional coverage, namely P(Y E X). Specifically, we split the test set into three groups according to their response values, and report the minimum coverage over each group.
We remark that the group coverage of feature-level conformal prediction stems from its vanilla version. That is say, when the vanilla version has satisfying group coverage, its feature-level version may also return relatively satisfying group coverage. Therefore, we did not provide Feature CP here because vanilla CP cannot return good group coverage.
We summarize the experiment results in Table Although we did not provide theoretical guarantee for group coverage, Feature CQR still outperforms vanilla CQR in various datasets in terms of group coverage. Among ten datasets, Feature CQR outperforms vanilla CQR in four datasets, and is comparable with vanilla CQR five datasets. Although the advantage not universal, improving group coverage via feature-level techniques is still possible.
We note that there still one dataset where vanilla CQR outperforms Feature CQR. We attribute the possible failure reason of Feature CQR on the dataset FACEBOOK2 to the failure of base models. As stated Section B.3 Feature CQR only works when the base model is well-trained. However, when grouping according the returned values, it is possible that there exists one group that is not well-trained during the training process. This may cause the failure of Feature CQR on the dataset FACEBOOK2.
25

---

Published as a conference paper at ICLR 2023
Table 8: Comparison between Feature CQR and CQR in terms group coverage.
GROUP COVERAGE
FEATURE CQR
VANILLA CQR
COMMUNITY
76.05 #5.13
78.77=1117
FACEBOOKI
65.52 20.95
66.68 1.66
FACEBOOK2
65.66 11.41
70.78 1.39
MEPS 19
76.67 12.17
71.26 1.23
MEPS20
77.70 +0.90
71.26 43.20
MEPS21
74.71 22.36
70.74 +1.83
STAR
84.62 22.77
82.20 +5.72
BIO
84.80 +1.0
80.03 11.88
BLOG
59.43 +0.60
49.10 +0.54
BIKE
81.07 1166
78.22 +2.44
Table 9: Ablation study of the number of layers in f and g (a 0.1) in unidimensional tasks, where the default setting is 2 2.
DATASET
B10
BIKE
BLOG
METHOD
COVERAGE
LENGTH
COVERAGE
LENGTH
COVERAGE
LENGTH
2:2 9:2
90.20 + 0.39
1.873 + 0.06
89.61 + 0.94
1.794 0.11
90.06 + 0.44
2.811 + 0.34 3.319
f f:3
9:1
90.24 1 0.32
1.961 0.02
89.72 1 1.10
1.917
0.14
90.16
1 0.34
0.22
3
90.00 0.46
1.860 0.12
89.72 + 0.81
1.748 0.10
90.11 + 0.43
2.595 0.38
B.6 ROBUSTNESS OF SPLITTING POINT
As discussed in the main text, the empirical coverage of Feature CP is pretty robust to the splitting point. We show both experiments of small neural networks (Table 9) and large neural networks (Table [OP. We remark that one can also apply standard cross- validation to find the best splitting point, which may return a shorter band.
B.7 FCP IN CLASSIFICATION
In this section, we show how to deploy FCP classification problems. The basic ideas follow Algorithm 3 except for two points:
1. We use cross-entropy loss instead of MSE loss when calculating the non-conformity score.
2. We do not deploy LiPRA, but use sampling method to return the final confidence band, which induces a looser bound.
For each sample in the calibration fold, sample 1000 samples in the confidence band of feature space, and return the corresponding predictions in output space to get the final confidence band. We summarize the experiment results in Table 2 where follow the statistics of APS and RAPS in Angelopoulos et al. (2021b).
B.8 THE VALIDATION OF NON-CONFORMITE SCORE
Distribution for non-conformity score in calibration fold. We plot the distribution of calibration score Figure g We plot each non-conformity score in the calibration fold. The distribution of non-conformity scores is smooth and single-peak in the real-world dataset, meaning that the proposed score is reasonable.
Besides, one may wonder what happens if the predicted function is close to the ground truth function, which may lead to a zero quantile of the non-con non-conformity score. We conduct experiments to show that zero quantiles would not underestimate the final non-conformity score. Specifically, create dataset that the prediction target is given by y = f (x) + e, where f* denotes a three-layer neural network, 0 with probability 0.901 and follows standard Gaussian with probability 0.009.
26

---

Published as a conference paper at ICLR 2023
Table 10: Ablation study of the number of layers in and g (a 0.1) in large neural networks on ImageNet.
SPLITTING POINT (PREDICTION HEAD
COVERAGE
LENGTH
THE LAST LAYER
0.900 0.0018
3.32 0.10
LAST TWO LAYERS
0.901 + 0.0047
3.26 0.03
0.900 0.0029
0.001
160, 0,000744552
0,00074
calibration SEt
0.0006
0.0006
calibration set samples
600
Figure 8:
The distribution of the calibration score for the segmentation task. This indicates that the definition of our non-conformity score is proper one.
Here the 0.901 is to let the 90% quantile of the non-conformity score a calibration set to be 0. Besides, make the predicted model satisfy f f". We summarize the results (with 0.1) as follows (five repeated experiments with different f*). Here x is generated with uniform distribution between 0 and 1. Note that the zero- lergthength is implied by zero quantiles.
However, in practice it is impossible for the predicted model to be f", therefore exact zero-quantile is impossible. We then summarize the results when the model is obtained via training on a training fold. The non-zero length implies that the quantile is non-zero.
B.9 ADDITIONAL EXPERIMENT RESULTS
This section provides more experiment results omitted in the main text.
Visualization for the segmentation problem. We also provide more visualization results for the segmentation problem in Figure 9
27

---

Published as a conference paper at ICLR 2023
Table 11: Zero quantile does not effects the empirical coverage (f fr).
TRIAL
COVERAGE
LENGTH
90.63
2
90.08
3
89.72
0
4
89.66
0
5
90.21
Table 12: Zero quantile does not effects the empirical coverage (f+f*)
TRIAL
COVERAGE
LENGTH
1
90.55
0.1274
2
89.72
0.1105
3
89.75
0.1120
4
89.31
0.1202
5
90.10
0.1245
Image
Ground Truth Label
Length
0.8
0.0
Figure 9: More visualization results for the Cityscapes segmentation task.
28


Reviews:
[{"review_id": "kPdSRL_bvMx", "reviewer": "Reviewer_MB3r", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors propose feature conformal prediction for semantic feature spaces by leveraging the inductive bias of deep representation learning.\n", "summary_of_the_review": "Overall, the paper is well written with both theoretical and numerical guarantee. ", "strength_and_weaknesses": "Strength:\nThe authors aim to improve the original conformal prediction (CP) method by leveraging the idea of semantic feature spaces. The idea is novel and useful. They provided a solid theory on both the effective aspect (coverage) and efficiency (confidence band length). In addition, they compared their method with vanilla CP in various scenarios, validating their claims. They also provided many examples to demonstrate their theoretical statements.\n\nWeaknesses:\n\n1. The idea of conducting CP in semantic feature spaces is really interesting. However, the authors had little discussion on how we split f and g. The two functions f and g should be indistinguishable. I have this concern because it seems very important to choose an appropriate feature space. \n\n2. There are several \"surrogate\" steps throughout the method. They used $\\hat{u}$ to replace the ground truth label, utilized the gradient descent method to calculate the non-conformity score, and employed \"Band Estimation\" and \"Band Detection\" to approximate the confidence band in output space. As for the theoretical guarantees, do the authors need the assumption of the surrogate estimators? \n\n3. Numerically, the authors could consider comparing their method with more baselines. There are a lot of improvements to vanilla CP in the literature. Beating only the original one may not be convincing enough.  \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "Yi22geNf_zE", "reviewer": "Reviewer_9LYb", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper propose to construct predictive interval through confidence set on features. It is assumed that a predictor $\\hat\\mu$ can be written as $\\hat g \\circ \\hat f$, where  $\\hat f$ is estimated feature. This paper proposes to use conformal inference with a new conformity score to obtain a confidence set for feature $f$. Theoretical guarantees show that the new confidence interval is provably more efficient than the baseline conformal predictive interval under “cubic conditions”, which is validated empirically. Empirical experiments corroborate the merits of the proposed method.", "summary_of_the_review": "The idea of performing conformal inference in feature space considered by this paper is a promising idea, although I do not think this paper has contributed much to the core concepts of conformal inference. For the application in deep learning, I believe there are some major questions that need to be addressed before this paper can be accepted.", "strength_and_weaknesses": "- Overall, it is a well written paper and the proposed method looks promising.\n- The methodology of this paper hinges on the choice of maps $\\hat g$ and $\\hat f$ such that the predictor $\\hat \\mu$ can be represented as $\\hat\\mu = \\hat g \\circ \\hat f$. While the choice of $\\hat g$ and $\\hat f$ may be clear for some architectures, it may be that unclear or even arbitrary for others. Take a multilayer perceptron or CNN for example, it is unclear how many last layers should be treated as $\\hat g$. Different choice of $\\hat g$ could lead to a different set of surrogate features, and the structure of the set of surrogate features could change the outcome of Algorithm 2. More specifically,\n(a) How stable is the set of surrogate features with respect to the choice of $\\hat g$? I wonder if for some choice of $\\hat g$ we can’t find any surrogate features other than $\\hat f$, so the only conformal score possible is zero.\n(b) If the set of surrogate features can vary by different choice of $\\hat g$, would this lead to different conformal scores after running Algorithm 2? I think it may be good to consider MLP of various widths, and set different number of last layers of the MLP as $\\hat g$ and see if the conformal scores estimated by Algorithm 2 vary a lot.\n(c) If the there is a general guideline for selecting $\\hat g$?\n- It looks that Algorithm 2 finds a feature $\\hat u$ that minimize the difference between $Y$ and $\\hat g(u)$, but I am unsure whether the $\\hat u$ returned by Algorihtm 2 would be closest to $\\hat f$. For example, a large learning rate may cause $u$ to be far from the initializer $\\hat f$. However, a small learning rate may lead to slower convergence. Hence, in practice, how would one select the learning rate that achieve a good tradeoff between the distances of $\\hat g(u)$ to $Y$ and $u$ to $\\hat f$?\n- I am a little bit concerned about the computational cost, because Algorithm 2 has to be done for every test output Y. The computational cost would be high if there is a large number of test points to be considered. Particularly, I guess for the Band Detection method, one would first specify a find grid for Y and run Algorithm 2 for every grid point. The computational cost should be high.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "NEJALnXbCB", "reviewer": "Reviewer_2teK", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes feature conformal prediction, which considers a prediction set over feature space rather than output space. In particular, the proposed novel conformal prediction is proved to satisfy the coverage guarantee under some assumptions. Moreover, the paper provides that the feature conformal prediction is provably more efficient than the vanilla CP. The efficacy (in terms of coverage guarantees and efficiency) of the feature conformal prediction is evaluated over five UCI datasets and one semantic segmentation dataset (along with one synthetic data for semantic segmentation). ", "summary_of_the_review": "I think applying CP in feature space is very interesting. But, currently I’m not fully convinced on the claim that Feature CP is more efficient than Vanilla CP, as discussed. Thus, I lean towards rejection, but hope to hear the author's opinions and additional results (if any). \n\n\n=== After rebuttal\n\nThanks for the response. Most of my concerns are addressed, and the benefit outweighs the remaining concerns (e.g., on the justification of the cubic conditions). So, I raised my score — I like the idea of forming sets in feature space (but we can use only differentiable score functions, like DNNs, which is worth highlighting).", "strength_and_weaknesses": "**Strengths**:\n* Suggests an interesting extension for conformal prediction. \n\n**Weaknesses**:\n* Convincing arguments on the necessity of Feature CP compared to Vanilla CP is unclear.\n* Incorrect statements appear; assumptions are implicit in theorem statements. \n\n\n**Weakness1**: The paper claims that Feature CP is more efficient than the vanilla CP (e.g., “if we instead employ conformal prediction on the more meaningful feature space, albeit all images have the same uncertainty on this intermediate space, the pixels would exhibit effectively different uncertainty in the output space after a non-trivial non-linear transformation“). But, this justification is unclear. \n\n1. Before achieving the efficiency of prediction sets, Feature CP needs to satisfy the coverage guarantee as in the vanilla CP. This is proven in Theorem 4. However, Theorem 4 seems to have Equation (3) solved exactly, which is not clear. If that’s true, please clearly specify in Theorem 4. If not, it would be useful to highlight this. Otherwise, this hidden assumption undermines Theorem 4. \n\n2. I think Theorem 5 is the crux of this paper, as it explains why Feature CP can be more efficient than Vanilla CP. First, the notation $\\alpha$ is reused in the theorem statement, which is very confusing as $\\alpha$ is reserved for a desired coverage in CP. Please use another term if it is not intended. Also, I believe this theorem implicitly assumes a special score function, i.e., a norm of a vector. If so, please explicitly state in the theorem statement. Apart from this, I’m not sure if the cubic condition can be even satisfied in real datasets. Based on the proof, it looks like the paper made this assumption to prove the statement. If not, why is the cubic condition natural? In short, to my current understanding, Theorem 5 only holds under limited setups, thus I’m not sure if this can strongly support that Feature CP is more efficient than Vanilla CP. \n\n3. Figure 2 and Table 1 empirically show that Feature CP is more efficient than Vanilla CP. But, the score function by Vanilla CP is too naive; for any test sample, it always returns the same length (as in Algorithm 1 “Ensure” line). One easy way to address this issue is considering the standard deviation of the prediction (e.g., Equation (1) in [R1] — this is a widely used trick to convert a point estimator to an uncertainty estimator by adding one more header for standard deviation prediction in a neural network). Can Feature CP outperform this simple baseline?\n\n[R1] http://proceedings.mlr.press/v128/vovk20a/vovk20a.pdf\n\n**Weakness 2**: The coverage guarantee in Equation (1) is not valid. The probability is taken over also a calibration set (e.g, Equation (1) in Tibshirani et al., 2019). The same issue appears in Theorem 4 (but fortunately, it does not affect the proof).  \n\nAs mentioned in Weakness 1, each theorem statement would be better if it explicitly includes required assumptions if there is any (e.g., a requirement of a norm-style score function in Theorem 5). \n\nIn related work, the statement \"Different from the above techniques, conformal prediction is appealing due to its simplicity, computationally free, and model-free properties.” is not true for calibration work. Most calibration approaches (e.g., temperature scaling) is simple, computationally free, and model-free; they simply solve a different problem. \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "CE1Zb-Lp5OZ", "reviewer": "Reviewer_5bhg", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper looks at a different way to perform conformal prediction by proposing a new score function as well as guarantees that their proposed method achieves smaller interval lengths. They prove that under some conditions (not very well explained) their proposed method can achieve smaller interval lengths compared to standard CP.\n\nThey validate their method on various datasets \n", "summary_of_the_review": "I have clearly outlined my thoughts as well as concerns in the above review. \nI am more than happy to increase my score in case the authors are able to clarify the misunderstanding as well questions I have outlined above.\n", "strength_and_weaknesses": "I will split my review into strengths as well as weaknesses in this review.\n\nI will start off with the strengths:\n- The paper proposed a new way to thinking about conformal prediction by looking at perturbation in the feature space to construct a score function.\n- This is a new way to construct a score function to the best of my knowledge and interesting given their experimental results.\n- Given that they work in the feature space the problem hence lies in how to convert these to the output space and they propose two methods to do so.\n- Experimental results look really good especially Table 1 ~40 -> ~1 seems like a huge improvement while keeping the coverage.\n\n\nThe weakness of the papers are the following:\n- Firstly the explanations of the cubic conditions for theorem 5 are not very intuitively explained and I am left to just guess whether these conditions ever hold in practice. Could you please clarify how you determine if any of these conditions ever hold in practice? The reason I am asking this is that proofs can become trivial if one gives themselves too strong conditions and hence I will rely on other reviewers' expertise on this theoretical part of the paper.\n- Next, I am confused to how to use your algorithm in the classification setting. Looking at eq 3 wouldn't there be many $v$ for which we have the discrete label Y? In that case how do we choose which $v$ and therefore which score? I might have misunderstood sth, so please clarify this part please. My concern is that given that there is an optimization step, in alg 2 how do you determine how many steps to go and the step size? These seem to be user choices which I hope the authors can point me towards in the appendix or in this rebuttal.\n- The authors never discuss or even give intuition into how \" (LiPRA) (Xu et al., 2020)\" works and hence should be added. Please let me know if I missed it in the appendix or main script\n-EXPERIMENTS: Could the authors elaborate on Table 1 and how they achieved such amazing results on length 40 -> 1 . Especially I am curious why the l_inf norm was used throughout the paper and not standard L2. The authors mention this \"Secondly, an intuitive explanation relates to our usage of l∞ to form the non-conformity score during the training.\" I don't understand this part, and hence a clarification on the loss functions or pointing me to the appendix would be helpful in understanding the results.\n- EXPERIMENTS: Fig 4 seems pointless without the interval lengths? Could you please point me to the interval lengths with varying alpha in case I missed it?\n- EXPERIMENTS: Table 1 literally contradicts your assumption that your lengths should be smaller. The std are overlapping in the table and hence I wonder if the method doesn't even work on synthetic data how can it possibly work on real data?\n- EXPERIMENTS: Lastly, my concern also lie in the way the authors chose the layers on which the compute the scores. The authors have some experiments in the appendix showing the picking different layers does impact the performance of their method, however, I don't see a comparison to standard CP. Does that mean if you choose the wrong layer standard CP is better? and if yes can you tell me which part of theorem 5 would be violated in that case? Could the authors please add experiments where the layer choice has been well justified and a more thorough ablation study has been done on how to choose the layer in the first place?\n\n\nOverall, i quite like the idea and would happily increase me score if the above have been answered.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster