Are foundation models for computer vision good conformal predictors?
Authors: Leo Fillioux
Julio Silva-Rodríguez
Ismail Ben Ayed
Paul-Henry Cournède
Maria Vakalopoulou
Stergios Christodoulidis
Jose Dolz
Venue: Accepted by TMLR ()

Abstract: Recent advances in self-supervision and contrastive learning have brought the performance of foundation models to unprecedented levels in a variety of tasks. Fueled by this progress, these models are becoming the prevailing approach for a wide array of real-world vision problems, including risk-sensitive and high-stakes applications. However, ensuring safe deployment in these scenarios requires a more comprehensive understanding of their uncertainty modeling capabilities, which has received little attention. In this work, we delve into the behaviour of vision and vision-language foundation models under Conformal Prediction (CP), a statistical framework that provides theoretical guarantees of marginal coverage of the true class. Across extensive experiments including popular vision classification benchmarks, well-known foundation vision models, and three CP methods, our findings reveal that foundation models are well-suited for conformalization procedures, particularly those integrating Vision Transformers. We also show that calibrating the confidence predictions of these models, a popular strategy to improve their uncertainty quantification, actually leads to efficiency degradation of the conformal set on adaptive CP methods. Furthermore, few-shot adaptation of Vision-Language Models (VLMs) to downstream tasks, whose popularity is surging, enhances conformal scores compared to zero-shot predictions. Last, our empirical study exposes APS as particularly promising in the context of vision foundation models, as it does not violate the marginal coverage guarantees across multiple challenging, yet realistic scenarios.

Julio Silva-Rodríguez Published in Transactions on Machine Learning Research (01/2026)
Are foundation models for computer vision good conformal predictors?
Leo Fillioux
so.filliourSeeetrleesnpe Université Paris-Saclay, CentroleSupélee, Gustave Roussy, INSERM, Cancer Data Science Unit, IHU PRISM Na- Nionl Precision Medicine Center Oncology, Gif-sur- Yuelte
LIVIA, ILLS, ETS Montréal
Ismail Ben Ayed
ismail. ismail bemneel.benaeedd@@essst..om
LIVIA, ILLS, ETS Montréal
Paul-Henry Cournède
peau-hherry.coury cournede @centralesupelee. Université Paris-Suclay, CentraleSupétee, Gustave Roussy, INSERM, Cancer Data Science Unst, IHU PRISM Na- tional Precision Medicine Center in Oncology, Gif-sur- Yuette
Maria Vakalopoulou
maria, bererroeooeeetttreeeeeeeeee
Université Paris-Saclay, CentraleSupélec, Gustave Roussy, INSERM, Concer Data Science Unit, IHU PRISM Na- tional Precision Medicine Center in Oncology, Gif-sur- Yvette
Stergios Christodoulidis
stergios. www.rrssttooooidisseeeeeeeeeeeie
Université Paris-Saclay, CentraleSupélec, Gustave Roussy, INSERM, Cancer Data Science Unit, IHU PRISM Na- Kioal Precision Medicine Center Oncology, Gif-sur- Yuelle
Jose Dolz
jose. dose.dolz@eetmmtl.co
LIVIA, ILLS, ETS Montréal
Reviewed on OpenReview: https: opennevvee.. net/ forum? Kadg9892p4
Abstract
Recent advances in self-supervision and contrastive learning have brought the performance of foundation models unprecedented levels in variely of lasks. Fueled by this progress, these models becoming the prevailing approach for wide array of real-world vision problems, including risk-scnsitive and high-stakes applications. However, ensuring safe de- ployment in these scemarios requires a more comprehensive understanding of their uncer- tainty modeling capabilities, which has received little attention. In this work, we delve into the behaviour of vision and vision-language foundation models under Conformal Prediction (CP), statistical framework that provides theoretical guarantees of marginal coverage of the true class. Across extensive experiments including popular vision classification bench- marks, well-known foundation vision models, and three CP methods, our findings reveal that foundation models are well-suited for conformalization procedures, particularly those integrating Vision Transformers. We also show that calibrating the confidence predictions of these models, a popular strategy to improve their uncertainty quantification, actually leads to efficiency degradation of the conformal set on adaptive CP methods. Furthermore, few-shot adaptation of Vision Language Models (VLMs) downstream tasks, whose pop- ularity is surging, enhances conformal scores compared to zero-shot predictions. Tast, our empirical study exposes APS particularly promising in the context of vision foundation models, it does not violate the marginal coverage guarantees across multiple challenging, yet realistic scenarios.

---

Published in Transactions on Machine Learning Research (01/2026)
1 Introduction
Large-scale pre-trained vision foundation models, such as DINOv2 (Qqub et al. 2024,, as well as those integrating lext, such as CLIP RRdford et al. 2021 are driving a new learning paradigm in machine learning, achieving umprecedentdd results on a broad spectrum of tasks. Despite their desirable zero-shot and generalization capabilities, recent evidence has pointed out the existence of bias and factual errors in these models (Tu et al. 20230, which transcend the field of computer vision (Shuster et al., 2022 Barocas et al. 2023). For example, the original CLIIP paper (Radlord et. al., 2021 demons(rated gender and race biases in certain zero-shot tasks, whereas Tu et al. (2023) identified that CLIP-based models are not always better calibrated than other arguably simpler ImageNet--rained models. Furthermore, Murugesan et al. (024a) recently showed that adapted models magnify the miscalibration issue compared to the zero-shot setting, yielding overconfident, predictions. These problems underscore widespread societal concerns surrounding the reliable deployment and use of foundation models in sensitive contexts, such as decision-making processes in critical scenarios, e.g.. healthcare or security applications.
A popular solution to quantify the uncertainty present in the predictions of deep models is calibration. In this setting, the proposed strategies aim at reducing the discrepancies between model predictions and the actual correctness probability. Temperature Scaling (TS) (Guo el al.) 2017 simple variant of Platt Scaling Platt et al. 1999). provides a simple post-processing approach to adjust the softmax probability scores of the trained models. Other lines of methods have proposed training objectives to enforce the model to produce
less confident scores, either in the predictions space (Popordanoska el al.] 2022 Pereyra el al. 2017 logits
Liu et al.
2023a
2022 or modifying the ground truth labels (Mukhoti et al. 2020 Müller et al. 2019
Conformal Prodiction (CP) Vovk et al. 1999), an alternative strategy to quantify the uncertainty (Sadinle
et al. 2018 Romano et al., 2020 Angelopoulos et al., 2020). is a statistical framework which offers several
advantages over calibration methods. First, unlike most methods in calibration, CP works directly on the
model predictions, offering an appealing solution for black-box models. Second, instead of simply modeling the correctness of the predicted probability, CP produces a set of predictions, including the most likely classes, which can be of much interest certain problems. Last, CP methods have theoretical guarantees for the marginal coverage of the true class within the predicted set, under several assumptions, contrary calibration approaches.
Due to these properties, CP is gaining attention to conformalize the predictions of deep models (Barber ct al. 2023 Angelopoulos et al. 2020 Ding et al. 2024 Sesia & Romano 2021p. Nevertheless, albeit the efforts to study CP in large language foundation models Guui et al. 2024), , its impact on vision foundation models has been unexplored, besides the significant implications it may have on a variety of vision problems. The aim of this study is to shed light and provide important insights into this dircction. To achieve this, we conducted an extensive empirical analysis of the performance of three common CP methods on 17 popular vision foundation models across multiple vision datasets. Our extensive experiments further explore common situations encountered in practice, assessing the impact on CP: under distributional drifts, after confidence calibration and in few-shot adaptation to novel downstream tasks. Our key observations are:
Vision, and vision-language loundation models seem to lead to lower set sizes and higher class- conditional coverage compared to their more traditionally (fully-supervised) trained counterparts.
(ii) Across all the experiments, Adaptive Prediction Sets (APS) is the best CP approach in terms of empirical coverage, while Regularized Adaptive Prediction Sets (RAPS) presents the best alternative from a conformal set size standpoint.
(iii) Under distributional shifts, APS exhibits the highest robustness among CP methods in terms of coverage guarantees, albeit decreasing its set elficiency.
(iv) Confidence calibration decreases the efficiency of conformal sets, but typically improves coverage gap
(v) Few-shot adaptation of vision-language models (VLMs) yields lower set sizes and coverage gaps than zero-shot predictions in ID data, with marginal gains on OOD.
2

---

Published in Transactions on Machine Learning Research (01/2026)
(vi) Under domain shift, across different foundation models, those including visual transformers, such DINO and CLIP, lead to smaller decreases in the conformal metries compared to models integrating convolutional neural networks.
2 Related Work
Foundation models for computer vision. The landscape of foundation models has rapidly evolved in recent years. Traditionally, pre-trained convolutional networks based on ResNet architectures (He et al. 2016) were the main models used by the community. However, driven by the umprecedented advances in language models, e-g. GPT (Brown et al. 20200 or LLaMA (Touvron et al. 2023 as well as the vast availability image data online, there are groundbreaing advances in unimodal (Caron et al. 2021 Oquab et al. 2024 Kirillov et al. 2023 and multimodal (Radford et al. 2021 foundation models for vision tasks, commonly based on vision transformers. These large pre-trained models aim to generalize across a broad span of visual tasks by pre-training on massive, diverse image datasets, exhibiting strong zero-shot and generalization capabilities to new tasks. For example, vision foundation models such as DINO (Caron et al. 2021 Oquab et al., 2024) rely on self-supervised learning strategies on large datasets, leading to excellent semantic understanding of visual content. On the other hand, CLIP (Radford et al. 2021 bridges the gap between language and vision modalities through contrastive lcarning, effectively allowing the model to understand images the context of natural language prompls and enabling zero-shol capabilities.
Quantifying the uncertainty of the predictions of deep networks has recently garnered considerable interest. From calibration standpoint, popular strategies inchude post-hoc approaches (Guo et al. 2017
Ding et al.
2021 Joy el al. 2023). 0, which map the logits or softmax predictions to smoother distributions, and
explicit learning objectives Liu et al. 2022 2023a Müller et al. 2019 Mukhoti et al. 2020 Hebbalaguppe
Murugesan et al.] 2024b), which are integrated into the loss function. Nevertheless, a main limitation of calibration methods is that they lack theoretical guarantees of model performance. In contrast, CP has recently emerged as a promising alternative, which provides marginal coverage guarantees over unseen test samples (Vovk al.] 2005; Shaffer & Vovk 2008). Specifically, CP resorts to a non-conformity score function (i.e., measure of how "different" a particular data point is compared to a CP calibration dataset) to produce finite prediction set, which is guaranteed to contain the true label with a user-specified confidence level. A central objective of the CP literature has been to improve cither the set efficiency (i.c., smaller set sizes) or the class-conditional coverage. For this purpose, several non-conformity scores have been presented Romano et al.) 2020 Angelopoulos et al., 2020 Sadinle et al. 2018 Einbinder et al. 2022;
et al. 2022
(Stutz et al.
2022
Ding et al.
2024 Straitouri et al. 2023), with Romano et al. (2020) ; Angelopoulos et al. (000);; Sadinle
ct al. (2018)
being popular methods widely studied. straightforward solution directly uses the raw class softmax predictions to generate the prediction sets (Sadinle et al., 2018). Adaptive Prediction Sets (APS)
[Romano et al. 2020) provides an adaptive version, computing non-conformity scores by accumulating sorted softmax probabilities in descending order. To further improve the efficiency, RAPS (Angllooollos et al. 2020 introduces an explicit regularization term, which penalizes non-conformity scores for unlikely classes.
However, a main limitation of existing evaluations is the focus on more traditional models, usually trained on data collection that falls within the calibration and test data points distribution. Despite this transfer learning framework not necessarily affecting the marginal guarantees provided by CP, how it affects its efficiency and conditional coverage remains to be explored. Thus, quantifying the uncertainty of their predictions is paramount given the rising popularity of foundation models in strategic domains. However, whereas uncertainty quantification from a calibration perspective has been scarcely studied (Murugesan) et al.
2024a Yoon et al. 2024 Tu et al. 2024b, its exploration under CP is, to our knowledge, overlooked.
3
Background
3.1
Conformal Prediction Framework
Let X and y denote the input and output space, respectively. We assume access to a calibration set Deat = [(x; P of n independent and identically distributed (i..d.) samples, where each x; = (Pa)I represents the black-box probabilities and yı E y {1,2, K} is the associated label. The goal of CP is to
3

---

Published in Transactions on Machine Learning Research (01/2026)
construct prediction set ((x,+1) S y for new test input Xn+1 such that it contains the true label yn+1 with a user pre-specified coverage probability a, where (0,1) denotes the error level.
The core idea of CP is to assess the degree to which a new sample conforms to the underlying distribution of the calibration data by computing non-conformity scores. Let S(x, y) be a non-conformity measure (or scoring function) that assigns a score 8; to each (x,.8) pair, quantifying how unusual the pair relative the rest of the data. Given the non-conformity scores for all calibration examples and new input Xn+1 (unseen in the calibration set), the conformal prediction set C(x,+1) is defined as:
((x++1) = {y E y: S(x=+1, v) S 9a
(1)
where is the 1-a quantile of the non-conformity scores on the calibration set, obtained with the observed labels:
qa = QUANTILE (5) [(n + 1)(1 )
(2)
n
Coverage Guarantees. A key property of conformal prediction is its finite-sample coverage guarantee. This property ensures that the prediction sets achieve the desired coverage probability marginally over X' and y, irrespective the underlying data distribution, as long as the calibration and test data are exchangeable (Vok al. 2005 Formally, for any a, conformal prodictors satisfy:
P(ya+1 E C(xa+1)) 2 1 -
(3)
This property is crucial for applications requiring reliable uncertainty quantification, particularly where distributional assumptions (e.g., Gaussianity) may not hold.
Tightness of Prediction Sets. Conformal prediction guarantees valid marginal coverage. Nevertheless, the efficiency of the prediction sets, i.e., their size, depends on the choice of the non-conformity measure. Choosing an appropriate Sxx, y) is key to balancing the tradc-off between coverage and tightness of the prediction sets. In practice, we aim to minimize the size of the prediction sets while maintaining the desired coverage probability.
3.2 Non-conformity scores
Least Ambiguous Classifier (LAC) (Sadinle et al. 2018 under the assumption that the output is correct. Intuitively, it be interpreted as thresholding of the output probabilities for each category. Thus, the non-conformity score can be constructed as:
aims to construct the smallest possible set
SLAC (x, v) = (y))
(4)
Where (v) = P(Y = y|X = x) is the softmax of the true class. LAC also provides notable efficiency in scenarios using an imperfect classifier. However, it lacks adaptability, e.g, in under-represented categories or uncertain predictions.
that leverages the accumulated confidence in the ordered probability predictions. Thus, APS is known to be adaptive score, whose main objective is enhancing the coverage of uncertain predictions by sacrificing efficiency. Formally, APS is expressed as:
Adaptive Prediction Sets (APS) Romano et al. 2020 provides non-conformity
SxPs(X, y) p(x, y) + (z( u,
(5)
where p(x,y) y) is the accumulated confidence of the categories more likely than the evaluated label y, i.e., p(x, y) -xv Tk=k', with V' v) {M|z* > #k=y). Adaptive methods usually include u E {0, 1}. random variable to break ties to achieve exact marginal coverage.
Regularized Adaptive Prediction Sets (RAPS) (Angelopoulos et al. 2020 builds upon APS by adding regularization term to enforce smaller predicted sets. Thus, APS score is modified to penalize the confidence of introducing additional, unlikely categorics, after a certain set size is met:
SRAPS y) = p(x, 2) + TT( Mo(x 2) kree)
(6)
4

---

Published in Transactions on Machine Learning Research (01/2026)
where X, Kreg 2 0 are prper-parameters controlling the penalty strength, 0z(y) is the rank of the sorted label, o(x. v) = 3 (x. v)1 + 1, and (.) denotes the positive part.
We refer the reader to the different works (Aneloopollos et al. 2020 Sadinle et al. 2018 Romano et al. 2020 for the respective conformal calibration coverage guarantees.
4 Experiments
4.1
Experimental Setup
We employ a total of 17 foundation models: two DINO (Caron et al. 2021 (DINO-S and DINO-B), four DINOv2 (Qquab et all 2024 (DINOv2-S, DINOv2-B, DINOv2-L, and DINOv2-G), three VICReg Bardes et al. 2022 (with ResNett-5,, ResNet-50x2, and ResNet-200x2) and eight VLMs (five
Models:
CLIP Radford et al., p models, MetaCLIP (Xu et al. 2024 L.a. Va (Liu et al. 2023b and Phi (Abdin)
et al.
2024)) Our main analysis is conducted on three popular vision datasets: CIFAR- 10 Krizhevsky &
Hinton
2009 CIFAR-100 (Kizzeessy & Hinton 2009 and Imagenet (Deng et al. 2009 including its
versions integrating domain shifts [Hendrycks et al. 2021a b Wang et al., 2019 Recht et al. 2019 Each
dataset is split into two sets: one for training, and one for the conformal experiments. The latter is then split
into one calibration set to tune the CP method, and one test set for evaluation. For few-shot, we adhere to
the emerging CLIP few-shot literature (Zhang et al., 2022 Silva-oorriguez al.)
2024 Zhou et al.] 2022a
and evaluate models 10 additional fine-grained and general concepts classification benchmarks:
SUN397
(iio ct al. 2010 FGVCAircraft (Maji et al. 2013). EuroSAT (Helber
et al.
2018 StanfordCars Krauss
et al. 2012
Food101 (Bossard el al. 2014). OxfordPets (Parkhi el al.
2012
Flowers102
Nilsback & Zis-
serman
2008
Caltech101 (Fei-Fei et al. 2004 DTD Cimpoi et al., 2014 and UCF101
Soomro et al.
2012
Metrics. Given a pre-specified coverage probability a, a test set Drest and method for predicting conformal sets Ö(-), we resort to the following metrics.
Set size( =
Σ
(((x)
(7)
|Deest] xeDien
Cov(Drest)
= Drestt x
LyE
(8)
CovGap(Dteal
(Cov(Dres,,)-
(9)
MCCC(D (estt) = min (Cov(Drest.a) kEY
(10)
Where Equation 7 is the average predicted set size across the test set (sometimes referred to efficiency). Equation is the empirical marginal coverage, Equation g. represents the average gap between the empirical coverage and the class-conditioned coverage, and Equation 10 is the min cass-conditioned coverage (MCCC).
Adaptation to target tasks. The foundation models used in this study have been pre-trained using different strategies. Nevertheless, they need to be adapted for novel tasks, as their pre-trained versions do not accommodate classification tasks, i.e. there is no classification head. To do this, foundation models are frozen, and a linear probing (LP) head (one linear layer followed by softmax activation function) is trained each datasct by optimizing a cross-cntropy loss (more details in Appendix Section A
4.2 Results
To gain insights into the factors influencing the efficacy of CP in vision foundation models, we design four experiments. First, explore the impact of CP in standard scenarios, where large calibration set is
5

---

Published in Transactions on Machine Learning Research (01/2026)
available to conform to the predictions of different models. Then, challenge the status quo of CP and alter the conformal sets to accommodate real-world scenarios by including domain shifts. Furthermore, since confidence calibration is significantly linked to CP, we explore the impact of model calibration on CP performance. Last, we examine CP when adapting a very popular VLM, i.., CLIP, to novel tasks.
4.2.1 Performance in the General Setting
First, we study the performance of 17 vision foundation models paired with CP methods under the standard setting, and on the three datasets, which present ideal conditions: a sufficiently large calibration set, and absence of distributional drifts between calibration and test sets. We aim to determine whether we can prescribe a winner solution in this scenario and which factors can help identify it.
LIC on DOB THESS
ow EEE the RAti pivdli
LAC Te 81 The RAMi io 2. MiNLTT
1.8
Conforal set
8-
1.4
2-
1.0
0.06
0.08
0.90
0.92
0.94
0.96
0.90
1.00
0.65
0.70
0.75
0.80
0.85
0.90
0
725
0.750
0.775
0,800
0.825
0.850
LIC P- Ri-dds
0.080
UGG m-aas 27-676
0.030
LAC mi030 216338
0,065
0.060
PAPS rn tewo
0.075
PAPS ts
0.025
Baily Mer
Covergee g0p
0.070
0.020
0.050
0.065
0.045
0,040
0.060
0.010
0.035
0.055
0.96
0.88 0,30 0,92
0.34 0.36
0.38
1.00
0.55
0.70
0.75
0.50
0.85
0.90
0.725 0.750 0,775 0,800 0,825 0.850 Linear probing accuracy
Linear probing accuracy
Linear probing acturacy
Figure/ 1: Relationship between the linear probing model accuracy and conformal set size (top) and the coverage gap (bottom) across different tasks of increasing complexity. From left to right: CIFAR-10, CIFAR- 100 and ImageNet.
Figure depicts the relationship between the lincar probing performance of each model and the conformal set size (lop), and coverage gap (bollom). The initial observation highlights a clear trend: higher-performance models tend to produce smaller prediction sets, regardless of the conformal method used. Nevertheless, while set efficiency (i.c. size) is typically considered as a sufficient condition in most prior literaturo in conformal prediction (Angelopoulos el al. 2020 our analysis revealed that the relation with the accuracy does not consistently hold when examining other metrics. In particular, Figure I (bottom) exposes that different CP methods yield mixed results for the coverage gap, which do not correlate directly with accuracy across all CP approaches. While APS appears to be almost unaffected by the model performance, RAPS clearly benefits from a st rong performance, and LAC is negatively affected by more accurate models as the dataset becomes more complex.
Regarding comparison between the CP methods, if we consider the set size, APS is clearly outperformed by the other approaches, whereas LAAC provides the smallest prediction sizes, closely followed by RAPS. Indeed, RAPS specifically designed to reduce the conformal set size of APS. However, the coverage gap results indicate that this comes the cost of increasing the range of the class-conditional coverage. Below, analyze the underlying causes that may explain this behaviour.
RAPS class-conditional coverage, and therefore coverage gap, are more sensitive to the model's accuracy. Let us assune we have two models, Mi and M2, in a multi-class classification problem, whose accuracies are Ace; and Acc2, respectively. For each class y E D, we refer to C,, = P(Y e SM) = y) as the class-conditonaal coverage for y under model M,, which measures the probability that predictions include the true label when the true label is y. Furthermore, let das mingey CM, (3) and 8n, mingey CM. (v) denote the minimum class-conditional coverage achieved by each model. Under this scenario, we argue that
6

---

Published in Transactions on Machine Learning Research (01/2026)
due to the penalty in RAPS, if Ace1 < Aceg, then δM₁ θMg when using RAPS a conformal prediction method. In particular, Mi (with lower accuracy) needs to expand its prediction sets for certain classes to meet the marginal coverage target 1-a. However, the penalty term encourages small prediction sets, limiting an excessive number of classes. Thus, for some difficult classes, model M1 may still potentially fail to meet the largel coverage, as the enforced penalty discourages overly large sets. This ultimately results in lower coverage rates for those specific classes than M2- In contrast, since APS does not include any regularization term that encourages small set sizes, it will compensate for lower performing models by increasing its set sizes, ultimately attaining higher class-conditions coverages.
0.90
14
RAPS APS
Class-condine Diss-conditioned coverage
12
0.85
10
0.80
Set Set size
8-
0.75
6
0.70
RAPS
4
APS
0.65
APS MCCC
2
VICReg (RN200X2)
VICRE CiP (ViT-B) (RN50)
CLIP (ConvNee)))
DINO.S
ViCReg (RN50X2
Phi
MetaCLin
LLava
CLIP (ConyNexta
DINO-B
CLIP (VIT-H)
DINOVZ-B CLIP (VIT-L)
DINOVZ-L
DINOV2-G
VICReg (RN200X2) VICReg (RN50)
QUI (ViT-8)
CLIP (ConvNext)
DINO-S
ViCReg (RN50x2
Phi 3.5
MetaCl.
LIP
CLIP (ConvNeXts Xt-L)
LLaVa
DINO .B
DINOV2-5
CLIP (VIT-H)
DINOVZ
CLIP (VIT-L)
DINOVZ4
DINOVZ-6
Figure 2: Comparison (APS us RAPS) of the cassiconditional coverage and set size for the class for which RAPS has the worst class-conditional coverage. Experiments performed on CIFAR-100. Models sorted (in asconding order) by their LP performance (min 0.64 and max- = 0.92), indicated by the size of the circles.
To confirm this hypothesis, we perform the following experiment. First, we identify the class with the lowest minimum class-conditional coverage obtained by RAPS for each model m, which we refer to as y and find its corresponding set size, both represented blue circles in Figure g Then, we identify the cass-condiiional coverage and set size of the class v provided by APS across all models, whose values are shown as pink circles in both plots of Figure g Last, we also include the minimum class-conditional coverage obtained by APS across each method, depicted with a cross. Note that the minimum class-conditional coverage from APS does not necessarily correspond to the coverage of class y Upon close examination of these results, we observe that, indeed, for models presenting lower accuracy (those with smaller circles), the gap between the class-conditional coverage for the worst class in RAPS and the same class in APS is consistently larger than for more accurate models (larger circles). This empirical evidence shows that the minimum cass-conditional coverage is substantially reduced because RAPS is constrained from expanding its predictions set size. This effect is more pronounced in less accurate models, where the true class may rank far away from the maximum allowable set size in the softmax predictions. In contrast, APS tolerates worse models by increasing the set size, which ultimately degrades the set efficiency but yields better class-conditional coverages.
Last, LAC presents structural differences with RAPS and APS, as it lacks an adaptive mechanism, relying on a uniform fixed threshold. Thus, LAC may yield inconsistent coverage rates across classes, resulting in high variability in the class-conditional coverage and thus in the coverage gap.
Following this analysis, we are also interested in determining whether network pre-trained following a more traditional approach (iee,, standard supervised fine-tuning) offers similar conformal capabilities to self-supervised and contrastive ones. In particular, we select a ViT-B pre-trained on ImageNet, which is the same architocture as the visual encoder of the different foundation models. The results from this analysis (Table I reveal that, despite obtaining lower classification accuracy when using LP on the different foundation models, CP methods typically yield better performance than in ViT IeageNet- This analysis should
7

---

Published in Transactions on Machine Learning Research (01/2026)
Table 1: SSL vs supervised learning. Results on ImageNet obtained by CLIP (ViT-B), MetaCLIP
(ViT-B), and DINO-S (ViT-S) and a ViT-B trained in supervised manner on ImageNet.
Set size (4)
MCCC (t)
Coverage
Acc (t)
LAC
APS
RAPS
LAC
APS
RAPS
LAC
APS
RAPS
ViTcLiP
72.01
3.03
9.50
3.73
0.434
0.556
0.418
0.900
0.900
0.900
VITDINO-S
74.92
3.27
10.02
4.23
0.433
0.477
0.412
0.900
0.900
0.900
VIT MetaCLIP
75.80
2.39
9.43
2.84
0.479
0.535
0.467
0.900
0.900
0.900
ViT ImageNet 76.08
2.36
38.75
4.46
0.416
0.495
0.405
0.900
0.900
0.900
be put in perspective with the analysis in Figure 0 which found that higher accuracy leads to lower set size, indicating that the training scheme plays a very important role. These differences are significant under the APS approach, where the set size significantly degraded on ViT ImageNett Moreover, the class-conditional coverage is also substantially affected, with nearly 6% decrease compared to the best model. Note that our criterion for model selection in Table [ was to ensure comparable predictive capabilitics, essential to differences in accuracy could otherwise confound
enable fair assessment of their conformal performance, the analysis. As DINO-B classification performance is not comparable, conclusions from a conformalization standpoint cannot be drawn from it (see Table 7
To further delve into these differences, we compute, for each test sample, the difference between the conformal set size for APS when applied to ViT Imagee and ViTCLIP models, whose distribution is depicted in Figure g (additional results in Ap- pendix Section D.. These values confirm that set size differ- ences are not derived from small set of isolated outliers but from a considerably large group of samples that see their con- formal set increase when using the VIT trained in a supervised manner. These results suggest that the strategies used to train foundation models yield better CP metrics, resulting in con- formalized models that be deployed more safely on critical scenarios. It is important stress that this study is limited due to the different dataset scales used for training (i.e., Ima- geNet alone is insufficient to train foundation model). Our goal, however, is to understand the conformalization properties of readily available pre-trained models, regardless of how they were pre-trained.
0.2
Larger set
0.16
VIT mignee
ViTaie
0.08
0.04
200
150
100
50
50
100
150
200
Differance
Figure 3: ViT IiTimaeeNet VITCLIP. An-
alyzing the difference in set size between a ViTCLIP and ViT ImageNet- Equal set sizes not shown.
4.2.2 Impact under Distribution Shifts
The theoretical guarantees of the coverage for conformal prediction hold under the hypothesis that the calibration set and the test set are drawn from the same distribution, i.e., data exchangeabilily assumption. In this section, we analyze the impact of having calibration sets that present distributional shifts with respect to the testing set.
We resort to ImageNet and its different versions: ImageNet-R (Hendrycks et al. 2021a ImageNet-A [Hendrycks et al. 2021b), ImageNet-Sketch (Wang et al., 2019 and ImageNet-Vi V2 Recht et al.) 2019). To introduce the distributional drift. between the calibration and testing data, we adapt the pro-trained model to ImageNet. Then, ImageNet is used as the calibration set to conform the model, which is later tested on the ImageNet version used for adaptation. This is repeated for each ImageNet variant.
APS Romano et al. 2020 exhibits strong robustness against large distributional shifts,, at the cost of substan- tially degrading efficiency. One would expect that adaptive CP methods, such as APS and RAPS, somehow mitigate domain shifts due to their adaptive nature. Nevertheless, Figure 5 reveals several interesting obser-
8

---

Published in Transactions on Machine Learning Research (01/2026)
vations, which contradict this intuition. First, observe that, when resorting to APS as CP method the coverage gap is consistently satisfied (or nearly satisfied) across all domain shifts and models (Figure B middle). In contrast, RAPS generally shows very similar performances compared to LAC, obtaining lower marginal coverage under several models and domains, and substantially lower than APS. To understand this phenomenon, we now study how set sizes evolve across domains for the different methods (Figure 5 left). We can casily observe that (t) APS yield the largest conformal sets across ImageNet domains, regardless of the model, and (ii) APS experiences the largest set increases when the complexity of the domain grows. Thus, as exposed in the previous Sec- tion, APS satisfied marginal coverage by substantially including more predicted classes, therefore increasing conformal set.
0.06
0.05
0.05
0.00
0.00
sizes. The regularization term in RAPS, which pushes towards lower set sizes, lim- its the adaptation to the domain shift, coming at the cost of decrease in cov- erage.
0.03
0.03
0.02
0.02
0.00
20
40
60
100
0.00
20
40
60
80
100
Coverage Percentage
Coverage Percentage 1%)
Figure 4: Domain shift. analysis. Distribution of class- conditional coverages for CLIP (ViT-B) on ImageNet-A: APS (left) and RAPS (right).
APS and RAPS are adaptive methods that produce similar minimum class- conditional coverage, as exhibited in ImageNet-A (Figure right). However, they showcase significant. differences in the average coverage gap. In the following, we explore this phe- nomenon in detail. More concretely, values obtained by APS and RAPS on ImageNet-A for CLIP (ViT-B), more datasets and models in Appendix Section
depict in Figure ! the distribution of the conditional class coverage
D Interestingly, while both approaches see their minimum class-conditional coverage decrease, their distributions are completely different. Indeed, APS distributions exhibit a Gaussian shape, with a decreas- ing number categories presenting lower conditional coverage as they separate from 1-a. contrast, the disbtibttoon in RAPS exposes significantly worse scenario, where class-conditional coverages are almost uniformly spread, with non-negligible amount of classes below the expected coverage, i.e, 1-a.
Another interesting and valuable observation is that the low performance of all CP methods is magnified when coupled with VICReg (i.e., larger set sizes and lower coverage), suggesting that this family of foundation models may suffer more under distribution shifts.
4.2.3 Does calibration affect Conformal Prediction?
Confidence calibration is a popular strategy improve the uncertainty estimates of deep models. These tochniques, which be either added as post-processing step (Platt et al. 1999 Guo et al. 2017 Kull
ct al.
2019 integrated as training regularizer [Hebbalaguppe et al.) 2022 Bohdal et al. 2023 Müller
et al.
2020 adapt the model softmax predictions to reflect. their act performance accurately. The
exponentially growing adoption of vision and vision-language foundation models in critical areas makes
APS
RAPS
1.g
8.0
RAPS
DINDv2
DINO2
40
(RosNet 50x2)
VICRea (Reshit 50x2) (VIT)
30
LAC
20
RAPS
DINOV2-8 B
VRgg (ResNer 50x2)
CUP
Imagehot
-42
-Shotch
ImageNet
.22
Sketch
magaNe:
SSkttch
Figure Evaluation under domain-shift. Set size (L), coverage (t), and MCCC (t) across three CP methods and three foundation models. Net versions sorted based OOD performance in Silva- Rodríguez et al. (2024h.
9

---

Published in Transactions on Machine Learning Research (01/2026)
integrating confidence calibration natural progression, evidenced by recent works Murugesan et al, 2024a; Yoon et al. 2024 Tu et al. 20240. Thus, in this section, we investigate this important issue, the relationship between calibration and conformal prediction in vision foundation models remains largely unexplored. Specifically, we examine whether calibrating these models with the popular Temperature Scaling (TS) (Guo el al., 2017) affects the conformal performance of fixed and adaptive CP methods. Particularly, we apply TS the ImageNet results in the general case (Section 4.2.1] As proper validation set is not available, we evaluate the CP performance over set of T values (14 values from 0.85 to 2), and found that
T = 1.1 typically yielded well-calibrated model In Appendix Section C explore histogram binning another method for model calibration.
Our observations suggest that calibration decreases the efficiency of CP sets, typically increasing the mini- mum
coverage, particularly on adaptive conformal prediction methods.
Effect on set efficiency. Confidence calibration typically smooths the distribution of the class soft- max scores, which results in less confident predictions. Consequently, the dominant value in the pre- dicted softmax vector is typically lower in calibrated models. Nevertheless, since CP methods pro- vide theoretical guarantees (under the data exchangeability assumption (Vovk et al.) 2005 to sat- isfy the target marginal coverage of 1 a, these changes in the softmax distributions affect the con- formalization obtained pre-TS. We present in Table 2 the results for the average set size and min- imuum class-conditional coverage before and after scaling the logits with TS. Furthermore, we in- clude the Expected Calibration Error (ECE) values to verify that model calibration has improved. These results show that if the model is calibrated, its prediction set size tends to be larger, particularly for adaptive CP ap- proaches (RAPS, and more specifically, APS). Figure e further delves into these results, where we plot the distribution of diffcrences between the set size samples before and af- ter calibration (i.e., a point in the distribution is C(x,)) )(x We can identify that the overall larger size in APS is caused by consis- tent efficiency degradation across samples and not few atypical cases with large conformal sets.
0.2
Larger set size Uncalibrated Calibrated
0.16
Density
0.12
0.08
0.04
0.0
100
-75
-50
-25
0
25
50
75
100
Difference in set size
Effect on class-condittonal coverage. We also observe that class-conditional coverage typically improved on calibrated models, par- ticularly when they conformalized by adap- tive CP methods. Thus, these results suggest that calibrating vision foundation models de-
Figure 6: Difference in conformal set size (i.e., effi- ciency) when applying temperature scaling APS and DINOv2-B (T [T=11 1.1).
creases efficiency while marginally enhancing class-conditional coverage, particularly on adaptive CP.
Effect on coverage gap. The best coverage gop obtained by APS (Romano et al. 2020 approzimately aligns with the coverage gop achieved at the oplámal colibration poinl, whereas RAPS AAgeelopollss et al. 2020p coverage gap strongly differs. Figure 7 shows the impact of the temperature T on the different CP metrics on CLIP conformalized predictions. An interesting observation is that at the optimal temperature value T- = 1.1, the CovGap of APS is very close to its optimal point (0.0567 us 0.0561). In contrast, while the RAPS coverage gap at optimal calibration is 0.0701, it decreases to 0.0642 as T increases. This suggests that the coverage gap for APS tends to be associated with model calibration performance, with smaller gaps occurring near optimal calibration. On the downside, APS conformal sets efficiency degraded. As T increases, the softmax distributions get smoother, explaining the degradation in efficiency, which is even more drastic for APS, whose set size monotonically increases with T. We concede that, whereas APS
Note that goal is not to obtain the best-calibrated model but evaluate the impact of calibration CP. Furthermore, 2023 using TS for similar datasets, e.g., TinyImageNet.
this value aligns with existing works (Mukhoti al. 2020 Joy Results on Appendix Section show similar behaviour other models.
10

---

Published in Transactions on Machine Learning Research (01/2026)
minimizes the coverage gap, it does at the expense of increasing the conformal set size. However, we believe that increasing the size, up to some extent, while improving coverage gap is preferable in critical decision systems.
0.03
0.20
LAC
033
LAC
APS
08
3.7
1.2
3.4
1.6
3.8
2.0
5.8
1.0
3.2
3.8
18
2.5
0.8
3.0
1.2
1.4
1.5
2.0
3.0
1.2
14
1.8
18
20
Figure 7: Impact of the temperature T on ECE (4), set size (4). MCCC (1), and CovGap (J) (CLIP (ViT-B) on ImageNet). Green region indicates the area with ECE smaller than for 1. Green dotted line indicates value of optimal calibration (T 1.1). More plots in Appendix Section C
4.2.4 Effect on few-shot adapted models
Adapting zero-shot CLIP models for downstream tasks in a few-shot labeled regime is becoming increasingly These strategics can be mainly categorized into Prompt Learning (Zhou al. 2022b
popular in VLMs.
Hantao Yao
2023
Zhou el al., 2022al, which optimize the set of text prompts given to the lext encoder,
and Adapters Yu et al. 2023 Lin al. 2023 Huang et al. 2024 Silva- Miva--Roriiguez et al., 2024 Zhou et al,
2023 where only limited set of learnable parameters atop embedding representations is updated. Thus, questioning whether adapting these models hinges the performance of CP methods is of paramount importance, as it addresses a foundational aspect of effectively integrating uncertainty quantifi- cation with modern architectures. To assess the impact CLIP adaptation in conformal prediction, we resort representative methods of the few-shot adaptation familics presented above, and follow standard adaptation and evaluation protocols in the literature Laila-Rodriguez et al., 2024]: (t) for out-of distribution (OOD), we adapt CLIP fow-shot samples (M 16) from ImageNet, and evaluate ImageNet and its variants, and (ii) for in distribution (ID) we adapt on few-shot samples and evaluate the validation set from the same dataset.
2022b Hantao Yao
Few-shot VLM adaptation of pre-trained models lead to lower set sizes and coverage gaps in the ID scenario (Table B First, we observe that, across both ID and OOD scenarios, both Adapters (ZSLP and CLAP) and Prompt Learning (CoOp and KgCoOp) yield smaller conformal set sizes and higher coverage gap than zero-shot CLIP, regardless of the CP method, with Prompt Learning yielding slightly better performances. In contrast, on the OOD scenario, only APS consistently enhances the set efficiency of the ZS model, with all the few-shot adaptation methods obtaining scarce coverage gap improvements across CP methods. These observations are related to the findings from the previous section, which suggested that better-calibrated models lead to larger conformal sets. Indeed, recent evidence (Murugesan et al. 2024a demonstrated that
Table 2: Quantitative impact of calibrating vision foundation models with TS. Average size and minimum class-conditional coverage are reported between cealiiiratteccaaibraattt models (T = and T 1.1 respectively) for several models, with ImageNet as benchmark dataset. Arrows indicate a decrease (+) or an (ncrease()) in metric after calibration, color indicates better (green) and worse (red) performance.
ECE
Set size (J)
MCCC
(x1002)
LAC
APS
RAPS
LAC
APS
RAPS
DINOv2-S
2.71/1.37+
1.87/1.87
6.66/8..44
2. 2.12/2.19.
0.370/0.351+
0.520/0.535.
1.384/0.11 109.
DINOv2-B
2.85/1.81+
1.96/1.36
4.50/10.46.
1.67/1.77.
0.414/0.406+
0.476/0.482.
0.489/0.498.
DINOv2-L
1.99/1.814
1.21/1.21
6.77/9.674
1.50/1.5%
0.326/0.336.
0.502/0.5224
0.153/0. 1874
DINOv2-G
3.66/2.10v
1.18/1.18
4.08/3.69.
1.44/1.30.
0.244/0.245v
0.458/0.4814
0.350/0.375.
VICReg (RN-50x2)
2.34/2.16v
3.08/3.09
13.49/16.06.
3.89/3.80v
0.442/00.330
0.549/0.557.
VICReg (RN-100<2)
2.21/1.90+
2.50/2.49+
12.05/ 14.61
2.98/3.01.
0.440/0.4430
0.567/0.568 50.66
0.462/0.47 40.2/
CLIP (ViT-B)
2.70/1.63+
3.03/3.01
9.50/11.114
3.73/3.53+
0.434/0.1411
0.556/0.5644
0.418/0.A14
MetaCLIP
2.63 1.99/
2.39/2.40.
9.43/11.314
2.84/2.84
0.479/0.477
0.535/0.541.
0.467/0.169.
11

---

Published in Transactions on Machine Learning Research (01/2026)
Table 3: Few-shot adaptation (16-shots). Set size and CovGap for CLIP (VTT-B backbone) on in- distribution (average over 11 datasets) and results are presented in Appendix Section
(average over ImageNet versions). Additional
Set size (4)
CovGap (1)
LAC
APS
RAPS
LAC
APS
RAPS
ZS
5.29
6.98
6.49
0.114
0.094
0.102
7SLP
2.09
3.43
2.53
0.087
0.060
0.064
CLAP
Love-Rodrguees
2024
2.13
3.52
2.57
0.088
0.060
0.065
CnOp
Aoou ald A02An
2.07
2.87
2.47
0.083
0.059
0.067
KgCoOp Hantao Yao 2023
2.10
3.11
2.53
0.085
0.059
0.005
71
7.68
19.22
9.93
0.095
0.085
0.094
OOD
ZSLP
8.34
18.51
10.61
0.092
0.083
0.092
CLAP
Silva-Roo Rodriguez et al
2024
7.55
17.41
9.89
0.093
0.083
0.091
CoOp
Zhou at. 2024b
7.86
KgCoOp
Hantto Yan 2023
7.54
16.68
9.79
0.094
0.085
0.095
few-shot CLIP adaptation methods deteriorate the confidence estimates compared to ZS predictions, which,
following our observations, should result in smaller conformal sets, as validated Table D Further details are provided in Appendix Scction
5 Conclusion
In this study, we aimed answering the question: Which CP method and model should use, and what can I expecl, in the era of vision foundalion models? Our findings revealed that vision foundation models yield better conformal metrics than their traditional pre-trained counterparts, with models integrating vi- sual transformers outperforming those that use convolutional neural networks, particularly under domain shifts. Furthermore, we observed that several common situations encountered in practice (i.e., presence of distributional drifts and models undergoing confidence calibration) are indeed detrimental for some CP ap- proaches. Interestingly, an adaptive CP method, i.e., APS, exhibited stronger robustness to these scenarios, particularly terms of conditional coverage, but at the expense of degrading the set efficiency.
The final decision ultimately hinges the requirements of each task. In certain fields, e-g. medical diagnosis, it may be preferable to maximize the conditional coverage across classes, even it increases the conformal set size, as this minimizes the risk of critical errors. In contrast, in domains where errors are less consequential, maintaining smaller set sizes might be preferable to streamline decision decision-making and reduce computational demands. Thus, the choice depends on balancing the tradc-offs between accuracy, usability, and the specific demands of the application.
Acknowledgement
This work has benefited from state financial aid, managed by the Agence Nationale Recherche under the investment program integrated into France 2030, project reference ANR-21-RHUS--000) This work was partially supported by the ANR Hagnodice ANR-21-CE45-0007. This work was partially supported by the ANR DAFNI ANR-23- 22--23-EE4--002, This work was granted access to the HPC resources of IDRIS under the allocation 2023-AD01 4014802 made by GFNCT. We gratefully acknowledge the DATATA program for supporting JD as visiting professor at Université Paris-Saclay. This work was partially supported by IA CLUSTER program, reference ANR-23-IACL-0003 DATAIA CLUSTER.
References
Marah Abdin, Jyoti Aneja, Hany Awadalla, Ahmed Awadallah, Ammar Ahmad Awan, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Jianmin Bao, Harkirat Behl, Alon Benhaim, Misha Bilenko, Johan Bjorck, Sébastien Bubeck, Martin Cai, Qin Cai, Vishrav Chaudhary, Dong Chen, Dongdong Chen, Weizhu Chen, Yen-Chun Chen, Yi-Ling Chen, Hao Cheng, Parul Chopra, Xiyang Dai, Matthew Dixon, Ronen Eldan, Victor Fragoso, Jianfeng Gao, Mei Gao, Min Gao, Amit Garg, Allie Del Giorno, Abhishek Goswami, Suriya
12

---

Published in Transactions on Machine Learning Research (01/2026)
Gunasekar, Emman Haider, Junheng Hao, Russell J. Hewett, Wenxiang Hu, Jamie Huynh, Dan Iter, Sam Ade Jacobs, Mojan Javaheripi, Xin Jin, Nikos Karampatziakis, Piero Kauffmann, Mahoud Khademi, Dongwoo Kim, Young Jin Kim, Lev Kurilenko, James R. Lee, Yin Tat Lee, Yuanzhi Li, Yunsheng Li, Chen Liang, Lars Liden, Xihui Lin, Zeqi Lin, Ce Liu, Liyuan Liu, Mengchen Liu, Weishung Liu, Xiaodong Liu, Chong Luo, Piyush Madan, Ali Mahmoudzadeh, David Majercak, Mat Mazzola, Caio César Teodoro Mendes, Arindam Mitra, Hardik Modi, Anh Nguyen, Brandon Norick, Barun Patra, Danicl Perez-Becker, Thomas Portet, Reid Pryzant, Heyang Qin, Marko Radmilac, Liliang Ren, Gustavo de Rosa, Corby Rosset, Sambudha Roy, Olatunji Ruwase, Olli Saarikivi, Amin Saied, Adil Salim, Michael Santacroce, Shital Shah, Ning Shang, Hiteshi Sharma, Yelong Shen, Swadheen Shukla, Xia Song, Masahiro Tanaka, Andrea Tupini, Praneetha Vaddamanu, Chumyu Wang, Guanhua Wang, Lijuan Wang, Shuohang Wang, Xin Wang, Yu Wang, Rachel Ward, Wen Wen, Philipp Witte, Haiping Wu, Xiaoxia Wu, Michael Wyatt, Bin Xiao, Can Xu, Jiahang Xu, Weijian Xu, Jilong Xue, Sonali Yadav, Fan Yang, Jianwei Yang, Yifan Yang, Ziyi Yang, Donghan Yu, Lu Yuan, Chenruidong Zhang, Cyril Zhang, Jianwen Zhang, Li Lyna Zhang, Yi Zhang, Yue Zhang, Yunan Zhang, and Xiren Zhou. Phi-3 technical report: A highly capable language model locally on your phone, 2024. URL httts:///rxiiv http:://///xvv.../a/babab/a a2////4..////....///// 14219
Anastasios Nikolas Angelopoulos, Stephen Bates, Michael Jordan, and Jitendra Malik. Uncertainty sets for image classifiers using conformal prediction. In International Conference on Learning Represenlations, 2020.
Rina Foygel Barber, Emmanuel Candes, Aaditya Ramdas, and Ryan J Tibshirani. Conformal prediction beyond exchangeability. The Annals of Statistics, 51()):116-85,, 2023.
Adrien Bardes, Jean Ponce, and Yann LeCun. Vicreg: Variance-ivvriacccccoaaaaaa regularization for self-supervised learning, 2022. URL https://arxiz: http://rraa...rrrraaa22200...000
Solon Barocas, Moritz Hardt, and Arvind Narayanan. Fairness and machine learning: Limitations and opportunities. MIT press, 2023.
Ondrej Bohdal, Yongxin Yang, and Timothy Hospedales. Meta-calibration: Learning of model calibration using differentiable expected calibration error, 2023. URL
Lukas Bossard, Matthieu Guillaumin, and Lue Van Gool. Food-101 - mining discriminative components with random forests. In European Conference on Computer Vision (ECCV), 2014.
Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. In Proceedings of the 34th International Conference on Newral Information Processing Systems, pp. 1877- 1901, 2020.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In Proceedings of the LEEE/CVP international conference on computer vision, pp. 9650-9660, 2021.
Mircea Cimpoi, Subhransu Maji, lasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. Describing textures in the wild. In Proceedings of the IEEE/CVF Conference on Computer Vision and Paltern Recognition (CVPR), pp. 3606 3613, 2014.
Jia Deng, Wei Doug, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. 2009 IEFE conference on computer vision and poltern recognition, pp. 248 255. Teee, 2009.
Tiffany Ding, Anastasios Angelopoulos, Stephen Bates, Michael Jordan, and Ryan J Tibshirani. Class- conditional conformal prediction with many classes. Advances in Neural Information Processing Systems, 36, 2024.
Zhipeng Ding, Xu Han, Peirong Liu, and Mare Niethammer. Local temperature scaling for probability calibration. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 6889- 6899, 2021.
13

---

Published in Transactions on Machine Learning Research (01/2026)
Bat-Sheva Einbinder, Yaniv Romano, Matteo Sesia, and Yanfei Zhou. Training uncertainty-aware classifiers with conformalized deep learning. Tn Advances in Newral Information Processing Systems (NeurIPS), 2022.
Li Fei-Fei, R. Fergus, and P. Perona. Learning generative visual models from few training examples: An in- cremental bayesian approach tested on 101 object calegories. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Worskshops (CVPRW), pp. 178 188-178,-0 2004.
Yu Gui, Ying Jin, and Zhimci Ren. Conformal alignment: Knowing when to trust foundation modols with guarantees. In Advances in Neural Information Processing Systems (NeurIPS), 2024.
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. On calibration of modern neural networks. In International conference on machine learning, pp. 1321-1330. PMLR, 2017.
Changsheng Xu Hantao Yao, Rui Zhang. Visual-language prompt tuning with knowledge-guided context optimization. In Proceedings of the IEEE/CVF Conference on Compuler Vision and Pallern Recogrition (CVPR), 2023.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2016.
Ramya Hebbalaguppe, Jatin Prakash, Neelabh Madan, and Chetan Arora. A stitch in time saves nine: A train-time regularizing loss for improved neural network calibration. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 166881-60080 2022.
Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth. Introducing eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. In IEEE International Geoscience and Remote Sensing Symposium (IGARSS), pp. 3606 3613, 2018.
Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, Dawn Song, Jacob Steinhardt, and Justin Gilmer. The many faces of robustness: A critical analysis of aut-of-distribution generalization. ICCV, 2021a.
Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardt, and Dawn Song. Natural adversarial exam- ples, 2021b. URL htttps://aaxiv.org//bbbbt//
Yumshi Huang, Fereshteh Shakeri, Jose Dolz, Malik Boudiaf, Houda Bahig, and Ismail Ben Ayed. Lp++: A surprisingly strong linear probe for few-shot clip. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recogrition, pp. 23773 23782, 2024.
Tom Joy, Francesco Pinto, Ser-Nam Lim, Philip HS Torr, and Puncet K Dokania. Sample-dependent adaptive temperature scaling for improved calibration. In Proceedings of the AATT Conference on Artificial Intelligence, volume 37, pp. 14919 14926, 2023.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization, 2017.
Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, et al. Segment anything. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 4015-4026, 2023.
Jonathan Krause, Michacl Stark, Jia Dong, and Li Fei-Fei. 3d object representations for fine-grained cat- egorization. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition (CVPR), pp. (998-3505, 2012.
Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. Technical Report 0, University of Toronto, Toronto, Ontario, 2009. URL https::/www. cs. toronto. edu/-kriz/
Tearning-feature=22000
14

---

Published in Transactions on Machine Learning Research (01/2026)
Meelis Kull, Miquel Perello-Nicto, Markus Kängsepp, Telmo Silva Filho, Hao Song, and Peter Flach. Beyond temperature scaling: Obtaining well-calibrated multiclass probabilities with dirichlet calibration, 2019. URL
2rgg//bb/9910.186666 12656
Zhiqiu Lin, Samuel Yu, Zhiyi Kuang, Deepak Pathak, and Deva Ramanan. Multimodality helps unimodality: Cross Cross-modal modal few-shot learning with multimodal models. In Proceedings of the /EE//CVF Conference Computer Vision and Pattern Recognition (CVPR), 2023.
Bingyuan Liu, Ismail Ben Ayed, Adrian Galdran, and Jose Dolz. The devil is in the margin: Margin-based label smoothing for network calibration. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 80 88, 2022.
Bingyuan Liu, Jérôme Rony, Adrian Galdran, Jose Dolz, and Ismail Ben Ayed. Class adaptive network calibration. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 16070-16077 2023a.
Haotian Liu, Chunyuan Li, Qingvang Wu, and Yong Jae Lee. Visual instruction tuning, 2023b.
S. Maji, J. Kannala, E. Rahtu, M. Blaschko, and A. Vedaldi. Fine-grained visual classification of aircraft. In ArXie Preprint, 2013.
Jishnu Mukhoti, Viveka Kulharia, Amartya Sanyal, Stuart Golodetz, Philip Torr, and Puneet Dokania. Calibrating deep neural networks using focal loss. Advances in Neural Information Processing Systems. 33:15288 15299, 2020.
Rafael Müller, Simon Kornblith, and Geoffrey E Hinton. When does label smoothing help? Advances in neural information processing systems, 32, 2019.
Balamurali Murugesan, Julio Silva-Rodriguez, Ismail Ben Ayed, and Jose Dolz. Robust calibration of large lasion-language adapters. In European Conference on Computer Vision (ECCV), 2024a.
Balamurali Murugesan, Julio Silva- Rodr--ozr, Ismail Ben Ayed, and Jose Dolz. Class and region-adaptive constraints for network calibration. In International Conference on Medical Image Computing and Cospuder-Asisted Intervention, pp. 57 67, 2024b.
Rafael Müller, Simon Korblith, and Geoffrey Hinton. When does label smoothing help?, 2020. URL
Dtttpa:/arrxiv.rggee wwtp://996.00000
Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. In Indian Conference on Computer Vision, Graphics and Image Processing, 2008
Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Mare Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, Mahmoud Assran, Nicolas Ballas, Wojeiech Galuba, Russell Howes, Po-Yao Huang, Shang- Wen Li, Ishan Misra, Michael Rabbat, Vasu Sharma, Gabriel Synnaeve, Hu Xu, Hervé Jegou, Julien Mairal, Patrick Labatut, Armand Joulin, and Piotr Bojanowski. Dinov2: Learning robust visual features without supervision, 2024. URL
https://arxie: http:///xxvvv..r/aaba www./bs////eeeoeeeo..oooeeeeeeee 2204. 07193
Omkar M Parkhi, Andrea Vedaldi, Andrew Zissorman, and CV Jawahar. Cats and dogs. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3498-36 3008,,55,,40 2012.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Edward Yang, Zach DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library, 2019.
Gabriel Percyra, George Tucker, Jan Chorowski, Lukasz Kaiser, and Geoffrey Hinton. Regularizing neural networks by penalizing confident output distributions. International Conference on Learning Representa- lions, 2017.
15

---

Published in Transactions on Machine Learning Research (01/2026)
John Platt et al. Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods. Advances in large margin classifiers, 10(3):61-3 1999.
Teodora Popordanoska, Raphael Sayer, and Matthew Blaschko. A consistent and differentiable Ip canonical calibration error estimator. Advances in Newral Information Processing Systems, 35:7933 7946, 2022.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning (ICML), pp. 8748-8763, 2021.
Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do imagenet classifiers generalize to imagenet?, 2019. URL https://arxinn httg/abs/9902.... 10811
Yaniv Romano, Matteo Sesia, and Emmanuel Candes. Classification with valid and adap- tive coverage. In Н. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Aduances in Neural Information Processing Systems, volume 33, pp. 3581-3591. Curran As- sociates, Inc., 2020. URL https: https::/prooeeedngs.neem www.ddiinnnn.n.i.....m.m www.:neuuurcccceoo sheurips.cc/paperr cc/paper https/cccpppp.eeeeeeeeeeaaeeaeee filee///apperr/22///iel 24cddresscs50022755c776666-----3
pdf.
Mauricio Sadinle, Jing Lei, and Larry Wasserman. Least ambignous set-valued classifiers with bounded error levels. Journal of the American Statistical Association, (14(522))2223 234, June 2018. ISSN 1537-274X. doi: 111100/0112214992177.....
URL http://áx.dof/ doi http://xx..oo..r//111/// 1081/0022///119..... 00010010000011101000000111111111 2017 1395341
Matteo Sesia and Yaniv Romano. Conformal prediction using conditional histograms. Advances in Neural Information Processing Systems, 34:6304-6315, 2021.
Glenn Shafer and Vladimir Vovk. A tutorial on conformal prediction. Journal of Machine Learning Research, 9(3), 2008.
Kurt Shuster, Mojtaba Komeili, Leonard Adolphs, Stephen Roller, Arthur Szlam, and Jason Weston. Lan- guage models that seek for knowledge: Modular search & generation for dialogue and prompt completion. Tn Findings of the Association for Compulational Linguistios: EMNLP 2022, pp. 373 393, 2022.
Julio Rodra-Ro, Sina Hajimiri, Tsmail Ben Ayed, and Jose Dolz. A closer look at the few-shot adaptation of large vision-languag, models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pullern Recognition (CVPR), pp. 23681 23681-23690, 2024.
Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah. Ucf101: A dataset of 101 humnn actions classes from videos in the wild, In ArXiv Preprint, 2012.
Eleni Straitouri, Lequn Wang, Nastaran Okati, and Manuel Gomez Rodriguez. Improving expert predictions with conformal prediction. In International Conference on Machine Learning, pp. 32633 32653. PMLR, 2023.
David Stutz, Krishnamurthy Dj Dvijotham, Ali Taylan Cemgil, and Arnaud Doucet. Learning optimal conformal classifiers. In International Conference on Learning Representations, 2022.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. orXiv preprint arXiv:2302.1999 2023.
Weijie Tu, Weijian Deng, and Tom Gedeon. A closer look at the robustness of contrastive language-image pre-training (clip). Advances in Neural Information Processing Systems, 36, 2023.
Weijic Tu, Weijian Deng, Dylan Campbell, Stephen Gould, and Tom Gedcon. An empirical study into what matters for calibrating lasion-language models. In International Conference on Machine Learning (ICML), 2024.
16

---

Published in Transactions on Machine Learning Research (01/2026)
Vladimir Vovk, Alex Gammerman, and Glenn Shafer. Algorithmic Learning in Random World. Springer, 01 2005.
Volodya Vovk, Alexander Gammerman, and Craig Saunders. Machine- Learning Applications of Algorithmic Randomness. In Proceedings of the 16th International Conference Machine Learning, pp- 444-455 Morgan Kaufmann, 1999. ISBN 1-55860-612-2.
Haohan Wang, Songwei Ge, Zachary Lipton, and Eric P Xing. Learning robust global representations by penalizing local prodictive power. In Advonces in Neural Information Processing Systems, pp. 10506-10518, 2019.
Hongxin Wei and Jianguo Huang. Torchep: A library for conformal prediction based on pylorch, 2024.
Jianxiong Xiao, James Hays, Krista A. Ehinger, Aude Oliva, and Antonio Torralba. Sun database: Large- scale scene recognition from abbey to soo. In Proceedings of the IEEE/CVF Conference on Compuler Vision and Pattern Recognition (CVPR), pp. 3485 3492,, 2010.
Hu Xu, Saining Xie, Xiaoqing Tan, Po- Yao Huang, Russell Howes, Vasu Sharma, Shang- Wen Li, Gargi Ghosh, Luke Zettlemoyer, and Christoph Feichtenhofer. Demystifying clip data. In The Twelfth Interna- lional Conference on Learning Representalions, 2024.
Hee Suk Yoon, Eunseop Yoon, Joshua Tian Jin Tee, Mark A Hasegawa-Johnson, Yingzhen Li, and Chang D Yoo. C-TPT: Calibrated test-time prompt tuning for vision- laiiunl models via text feature dispersion. In The Twelfth International Conference Learning Representations, 2024.
Tao Yu, Zhihe Lu, Xin Jin, Zhibo Chen, and Xinchao Wang. Task residual for tuning vision-language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10899-10909, 2023.
Renrui Zhang, Rongyao Fang, Wei Zhang, Peng Gao, Kunchang Li, Jifeng Dai, Yu Qiao, and Hongsheng Li. Tip-adapter: Training-free free clip-adapter for better vision- langunla modeling. In European Conference on Computer Vision (FCCV), pp. 1-19, 2022.
Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Conditional prompt learning for vision- language models. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recog- nition (CVPR), 2022a.
Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Learning to prompt for vision-language models. International Journal of Compuler Vision (IJCV), 2022b.
17


Reviews:
[{"review_id": "TpQTqUkcS7", "reviewer": "Reviewer_2jeP", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "Summary of contributions:\n- Comprehensive and systematic study of conformal prediction on vision foundation models, covering 17 models and several representative datasets. \n- Actionable insights from the comparison of CP methods, including the relationship between accuracy, coverage, and efficiency; performance under distribution shift; relationship of CP performance to pre-training strategies; impact of calibration on CP performance\n\n\nStrengths:\n- Evaluation is comprehensive \n- The takeaway messages are practically useful\n- The paper is well organized\n\nWeakness\n- Some data and model categorization information is missing \n- Sometimes the writing can be a bit confusing", "claims_and_evidence_yn": "Yes", "claims_explanation": "The conclusions and messages are from the empirical patterns, and the empirical studies cover a wide range of models, training strategies, and datasets, so the evidence is convincing.", "audience_yn": "Yes", "audience_explanation": "Researchers in computer vision and uncertainty quantification can build on the insights from this paper to build robust conformal prediction sets in their tasks. Also, people working on conformal prediction may be able to extract interesting theoretical and methodological questions based on these empirical findings.", "requested_changes": "1. Please pay attention to \\cite{} and \\citep{} throughout the paper. \n2. In equations (4-6), what does $x_{k=y}$ mean? I suppose it is the predicted probability for label $y$, but the notation is a bit non-standard.\n3. How is the data folds split in the experiments? Are the calibration and test data randomly split so they are exchangeable? If this is the case, then the coverage of conformal prediction is guaranteed to be above $1-\\alpha$ on average, and the empirical coverage gap is mainly the artifact of randomness or ties. I was wondering whether coverage gap is a meaningful metric here. \n4. Could you explain this sentence on page 7: \"despite obtaining lower classification accuracy when using LP on the different foundation models, CP methods typically yield better performance than in ViTImageNet.\" I'm not sure I understand what you mean here. What are you comparing?\n5. The message from the results surrounding Table 1 could be made more clear. Do you mean prediction accuracy does not necessarily translate to efficient prediction sets, and CLIP methods (or LP?) often leads to more efficient prediction sets?\n6. Section 4.2.2 on distribution shifts is interesting. It might be interesting to further comment on why you choose ImageNet as calibration set and adapted version of ImageNet as test set, and not the opposite. Does this reflect practical deployment better?\n7. In Table 3, there is a missing link between the training strategy category and the model names, so it is a bit confusing when mapping the table metrics to the conclusion remarks. Please consider add this information here and for other similar occurrences.\n8. While APS has robust coverage, it seems its size is way larger than other methods. Would this be desired in practice?", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "qO6tn0w7Ms", "reviewer": "Reviewer_iyJB", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper aims to conduct a comprehensive investigation of conformal inference applied to vision models for classification tasks. Several conformity scores and vision models are compared in the paper, based on which the authors provide guidelines for choosing between the scores and vision models.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The paper investigates the performance of 17 vision models with 3 conformity scores.", "audience_yn": "Yes", "audience_explanation": "It would be of interest to the practitioners to understand the performance of uncertainty quantification methods on vision data.", "requested_changes": "1. The paper only compares the classification task in the paper; apparently, the Vision Foundation model is capable of other tasks, e.g., generating new content. Providing a comparison in such tasks may strengthen the paper. \n\n2. It would be informative to provide the SE of the quantities reported in the paper. \n\n3. There are conformity scores addressing distribution shifts, e.g., [1,2]. They should also be included in the comparison.\n\n\n[1] Tibshirani, Ryan J., et al. \"Conformal prediction under covariate shift.\" Advances in neural information processing systems 32 (2019).\n\n[2] Cauchois, Maxime, et al. \"Robust validation: Confident predictions even when distributions shift.\" Journal of the American Statistical Association 119.548 (2024): 3033-3044.", "broader_impact_concerns": "N/A", "additional_comments": "N/A", "extra_scores": {}, "extra_text": {}}, {"review_id": "I2QPVkmUcE", "reviewer": "Reviewer_FNpJ", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper, from an empirical aspect, studies a very interesting problem: how would vision foundation models’ predictive performance correlate with their performance in (distribution-free) uncertainty quantification? Using metrics such as sizes of prediction sets, class-conditional coverage, and coverage gap, this paper offers a novel way to benchmark foundation models.", "claims_and_evidence_yn": "No", "claims_explanation": "Overall, this paper conducts extensive numerical comparisons across various models and different choices of nonconformity scores, and also inspects the difference with/without distribution shifts and calibration. These experiments have enriched our knowledge on both vision foundation models and the practical performance of predictive inference.\nBut in some sections (e.g., Section 4.2.3 and discussions on page 7) are confusing in the sense that either the empirical results are hard to interpret or the intuition behind the results are not further theoretically justified by toy examples.\n\nI have include strengths in the next part and also include my questions in \"Requested Changes\".", "audience_yn": "Yes", "audience_explanation": "1. This paper conducts extensive numerical comparisons across various models and different choices of nonconformity scores, and also inspects the difference with/without distribution shifts and calibration. These experiments have enriched our knowledge on both vision foundation models and the practical performance of predictive inference.\n\n2. While some experimental results are in line with our intuition, there are some experiments, especially those related to minimum class-conditional coverage and calibration, that are bringing in new findings in the field and can motivate more theoretical thinking.\n\n3. This paper is clearly organized and experimental results are well presented.", "requested_changes": "1. First and foremost, the notion of “better conformalization” is a bit vague even in this concrete context. In my understanding, it can relate to more accurate prediction sets, or more evenly distributed class-conditional performance, or better cross-domain robustness. I would suggest that the authors could formally and rigorously clarify this notion.\n\n2. Regarding the metrics for evaluation, especially when combined with row 2 in Figure 1, the sign of the coverage gap would be very helpful in understanding the performance of different models. For example, it is unclear whether the increasing trend of the orange line in row 2-panel 1 in Figure 1 indicates a decreasing trend in coverage rate below 90% or otherwise.\n\n3. On page 6, in the paragraph related to RAPS’s class-conditional coverage, I wonder if there is a typo in the definition of C_M(y).\n\n4. The comparison between SSL models and models trained with supervised data on page 7 is a bit confusing. I also wonder how the performance on marginal coverage would be for two types of models? \n\n5. Section 4.2.2 is very interesting in that it has shown very different performance of these nonconformity scores under distribution shift. \nThe robustness of APS may suggest that it can capture some domain-invariant information (or, this score is nearly independent of X). It would be more helpful if some discussions on the theoretical explanation could be added.\n\n6. Section 4.2.3 on calibration offers a novel aspect of evaluating the performance of each model. However, the findings are very hard to interpret in the sense that, even with the “oracle” model (i.e. a model that captures the true distribution of Y|X), it may be not guaranteed to have the optimal performance in the class-conditional coverage. In this sense, the correlation between calibration and MCCC can be confusing without further theoretical justification (even with a toy setting).", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)