C-Adapter: Adapting Deep Classifiers for Efficient Conformal Prediction Sets

## Abstract

Abstract Conformal prediction, as an emerging uncertainty quantification technique, typically functions as post-hoc processing for the outputs of trained classifiers.
To optimize the classifier for maximum predictive efficiency, Conformal Training rectifies the training objective with a regularization that minimizes the average prediction set size at a specific error rate.
However, the regularization term inevitably deteriorates the classification accuracy and leads to suboptimal efficiency of conformal predictors.
To address this issue, we introduce Conformal Adapter (C-Adapter), an adapter-based tuning method to enhance the efficiency of conformal predictors without sacrificing accuracy.
In particular, we implement the adapter as a class of intra order-preserving functions and tune it with our proposed loss that maximizes the discriminability of non-conformity scores between correctly and randomly matched data-label pairs.
Using C-Adapter, the model tends to produce extremely high non-conformity scores for incorrect labels, thereby enhancing the efficiency of prediction sets across different coverage rates.
Extensive experiments demonstrate that C-Adapter can effectively adapt various classifiers for efficient prediction sets, as well as enhance the conformal training method.

## 1 Introduction

Quantifying the uncertainty of predictions is critical for artificial intelligence systems, particularly in high-stakes environments (e.g., financial decision-making and medical diagnostics).
Conformal prediction, a statistic framework for uncertainty estimation, converts an algorithm’s predictions into prediction sets containing the true class with a user-specified coverage rate (Balasubramanian et al., ; Shafer & Vovk, ).
Critically, the validity of sets is satisfied in a distribution-free sense: they possess explicit, non-asymptotic guarantees even without distributional assumptions or model assumptions. To obtain informative outputs, it is of great importance to improve the efficiency of conformal predictors, aiming for the prediction sets with minimal ambiguity (Sadinle et al., ).

Conformal prediction typically functions as post-hoc processing for the output of trained classifiers, which might already be either unnecessarily conservative or overconfident (Bellotti, ; Stutz et al., ).
To optimize the predictive efficiency, Conformal Training (Stutz et al., ) rectifies the training objective with a regularization that minimizes the average prediction set size at a specific error rate (e.g., 0.01).
However, the regularization term inevitably deteriorates the classifier accuracy by increasing the difficulty of converging to an optimal solution (Stutz et al., ), which in turn leads to the suboptimal efficiency of the conformal predictor.
This challenge is especially significant when dealing with many classes, making it difficult to apply to large-scale datasets such as ImageNet (Deng et al., ).
This motivates our methodology, which enables the efficient adaptation of trained classifiers for conformal prediction without sacrificing classification accuracy.

In this work, we propose Conformal Adapter (dubbed C-Adapter), an adapter-based tuning method to enhance the efficiency of conformal predictors.
In particular, we tune an adapter layer appended to trained classifiers for conformal prediction using the training data.
Our key idea is to adapt trained classifiers for conformal prediction while preserving the ranking of labels in the output logits, thereby maintaining the top-$k$ accuracy of the classifiers.
To achieve this, we implement the adapter as a class of intra order-preserving functions (Rahimi et al., ).
For the optimization of this adapter, we propose a loss function that enhances the discriminability of non-conformity scores between correctly and randomly matched data-label pairs.
In effect, the loss encourages the non-conformity scores of correctly matched data-label pairs to be lower than those of incorrectly matched ones, resulting in more efficient predictions across different coverage rates.
Equipped with C-Adapter, the predictor maintains top-$k$ accuracy and generates highly efficient prediction sets. The application of C-Adapter is illustrated in Figure 1.

Figure: Figure 1: Application of C-Adapter. C-Adapter adapts trained classifiers for conformal prediction while preserving the ranking of labels in the output logits. Compared to using the raw logits, this refinement improves the efficiency of prediction sets while maintaining the marginal coverage rate.
Refer to caption: x1.png

To validate the effectiveness of our method, we conduct extensive evaluations on three benchmarks of image classification, including CIFAR-100 (Krizhevsky et al., ), ImageNet (Deng et al., ), and ImageNet-V2 (Recht et al., ). The results demonstrate that C-Adapter can significantly enhance the efficiency of conformal predictors. For example, C-Adapter reduces the average size for APS from $9.21$ to $2.86$ on ImageNet (Deng et al., ) with DenseNet121 (Huang et al., ) at $\alpha=0.1$. This approach also generalizes effectively to different score functions, consistently improving their efficiency.
Moreover, C-Adapter can enhance the efficiency of prediction sets while simultaneously improving their conditional coverage.
Notably, our method is easy to implement in practice, as it is insensitive to the hyperparameter $T$ and incurs low computational costs.

We summarize our contributions as follows:

- •
We propose C-Adapter, a simple and effective method to enhance the efficiency of conformal predictors without sacrificing classifier accuracy. This approach serves as a distinctive complement to existing score-based and training-based conformal prediction algorithms.
- •
We theoretically demonstrate that enhancing the discriminability of non-conformity scores between correctly and randomly matched data-label pairs is equivalent to improving the overall efficiency of conformal predictors. To this end, we propose a loss function specifically designed to achieve this goal and apply it to optimize our conformal adapter.
- •
We empirically show that C-Adapter effectively adapts various classifiers for efficient prediction sets across different non-conformity score functions. Moreover, we validate that C-Adapter outperforms Conformal Training and can further enhance its performance.

## 2 Background

#### Setup

In this work, we consider the multi-class classification task with $K$ classes. Let $(X,Y)\sim\mathcal{P}_{\mathcal{X}\mathcal{Y}}$ denote a random data pair sampled from the joint distribution $\mathcal{P}_{\mathcal{X}\mathcal{Y}}$, where $\mathcal{X}\subset\mathbb{R}^{d}$ is the input space and $\mathcal{Y}:=\{1,\cdots,K\}$ is the label space.
Given a training set, we learn a classifier $f:\mathcal{X}\rightarrow\mathbb{R}^{K}$ with parameter $\bm{\theta}$.
Given an instance $\bm{x}$, we predict the probability of class $k$ by:

$$ $\hat{\pi}_{k}(\bm{x};\bm{\theta})=\psi(f_{k}(\bm{x};\bm{\theta}))=\frac{e^{f_{ k}(\bm{x};\bm{\theta})}}{\sum_{i=1}^{K}e^{f_{i}(\bm{x};\bm{\theta})}},$ (1) $$

where $\psi$ denotes the softmax function and $f_{k}(\bm{x};\bm{\theta})$ is the $k$-th element of the logits $f(\bm{x};\bm{\theta})$.
Deep classifiers usually suffer from the miscalibration issue: the estimated probabilities might be either conservative or overconfident, leading to inaccurate assessments of uncertainty (Guo et al., ).

#### Conformal Prediction

In uncertainty quantification, conformal prediction (Vovk et al., ) seeks to construct prediction sets $\mathcal{C}(X)\subseteq\mathcal{Y}$ such that $\mathbb{P}\{Y\in\mathcal{C}(X)\}\geq 1-\alpha$ for a pre-specified error rate $\alpha\in(0,1)$.
To satisfy the desired coverage rate $1-\alpha$, we take an independent conformal calibration dataset $\mathcal{D}_{\text{cal}}:=\{(\bm{x}_{i},y_{i})\}_{i=1}^{n}$, and then determine the threshold $\tau_{\alpha}$ such that the prediction sets are large enough to achieve the desired coverage level of $1-\alpha$ on this calibration set.
Specifically, we calculate the non-conformity score $s_{i}:=S(\bm{x}_{i},y_{i};\hat{\pi})$ for each sample $(\bm{x}_{i},y_{i})$ in the calibration set where $S$ is a pre-specified score function to measure non-conformity of each input sample. We then determine the threshold $\tau_{\alpha}$ as the $1-\alpha$ quantile of the set $\{s_{i}\}_{i=1}^{n}$, as follows:

$$ $\tau_{\alpha}=\inf\left\{s:\frac{|\{i\in\{1,\cdots,n\}:s_{i}\leq s\}|}{n}\geq \frac{\lceil(n+1)(1-\alpha)\rceil}{n}\right\}.$ $$

During testing, we calculate the non-conformity score $S(\bm{x}_{n+1},y;\hat{\pi})$ for a given instance $\bm{x}_{n+1}$ and each label $y\in\mathcal{Y}$. Then, the prediction set $\mathcal{C}(\bm{x}_{n+1};\tau_{\alpha},\hat{\pi})$ with $1-\alpha$ coverage is constructed by:

$$ $\mathcal{C}(\bm{x}_{n+1};\tau_{\alpha},\hat{\pi}):=\left\{y\in\mathcal{Y}:S( \bm{x}_{n+1},y;\hat{\pi})\leq\tau_{\alpha}\right\}.$ (2) $$

In other words, the final prediction sets achieve marginal coverage by containing all labels with non-conformity scores below the threshold (Vovk, ; Angelopoulos et al., ).
In addition to the coverage, we typically expect to optimize the size of prediction sets, which is referred to as efficiency.
Nevertheless, the length of the resulting prediction sets can vary dramatically depending on the design of $S(\bm{x},y;\hat{\pi})$.
In this work, we consider three popular score functions for classification, including THR (Sadinle et al., ), APS (Romano et al., ), and RAPS (Angelopoulos et al., ). We provide a detailed introduction to these score functions in Appendix B.1.

#### Conformal Training

Conformal prediction typically works as post-hoc processing for the outputs of trained classifiers. To optimize the classifier for maximum predictive efficiency, Conformal Training (ConfTr) (Stutz et al., ) rectifies the training objective with a regularization that minimizes the average prediction set size at a specific error rate $\alpha$. The loss function is formulated as:

$$ $\mathcal{L}_{\text{ConfTr}}(f(\bm{x};\bm{\theta}),y,\tau^{\text{soft}}_{\alpha })=\mathcal{L}_{\text{cls}}(f(\bm{x};\bm{\theta}),y)+\lambda\mathcal{L}_{\text {size}}(f(\bm{x};\bm{\theta}),\tau^{\text{soft}}_{\alpha}),$ (3) $$

Here, $\mathcal{L}_{\text{cls}}$ represents the classification loss, while $\mathcal{L}_{\text{size}}$ refers to the size loss, which approximates the size of the prediction set at a coverage rate of $1-\alpha$.
Here, $\tau^{\text{soft}}_{\alpha}$ denotes the soft threshold and $\lambda$ controls the strength of the regularization term.
We provide a detailed introduction to ConfTr in Appendix B.2.
Notably, while ConfTr with a tuned hyperparameter $\lambda$ may improve the efficiency of conformal predictors, the regularization term $\mathcal{L}_{\text{size}}$ inevitably deteriorates the accuracy of the classifier by increasing the difficulty of converging to an optimal solution (Stutz et al., ).

Figure: (a) CIFAR-100
Refer to caption: x2.png

To provide a straightforward view, we demonstrate the effect of the regularization term $\mathcal{L}_{\text{size}}$ on the accuracy and efficiency of conformal predictors in Figure 2.
We conduct experiments of ConfTr with various $\lambda$, using ResNet18 on CIFAR100 and ImagNet.
The results demonstrate that using this regularization continuously degrades the classification accuracy of the classifier as $\lambda$ increases.
For efficiency, ConfTr raises the average size of APS and RAPS after achieving the optimal performance on CIFAR-100.
On ImageNet, ConfTr offers only marginal benefits for the efficiency of conformal predictors.
The negative effect of ConfTr is especially noticeable on THR: the average size of THR is consistently increased over various $\lambda$.
The decrease in classification accuracy inevitably results in larger prediction sets, which in turn limits the efficiency on average.
We present a detailed description of the experimental setup and the effect of the regularization term $\mathcal{L}_{\text{size}}$ on top-$k$ classification accuracy in Appendix G.1.
We proceed by introducing our method, targeting this issue.

## 3 Method

In our previous analysis, we demonstrate that ConfTr deteriorates the classification accuracy, thereby hindering the efficiency of conformal predictors. To address this issue, our key idea is to adapt the trained classifiers for conformal prediction while preserving the ranking of labels in the output logits, thereby keeping the top-$k$ accuracy of the original classifier unchanged.

Figure: Figure 3: Flow of C-Adapter. The design follows the definition of intra order-preserving functions (Rahimi et al., , Theorem 1), ensuring that the refined logits maintain the ranking of the inputs.
Refer to caption: x4.png

#### Conformal Adapter

To this end, we propose a novel adapter-based tuning method – Conformal Adapter (dubbed C-Adapter), which appends an adapter layer to trained classifiers for conformal prediction.
Formally, we use $g:\mathbb{R}^{K}\rightarrow\mathbb{R}^{K}$ to denote the conformal adapter that takes the model outputs $f(\bm{x};\bm{\theta})$ as input.
Then, the final prediction of the model equipped with C-Adapter is:

$$ $\displaystyle\widetilde{\pi}(\bm{x};\bm{\theta},\bm{w})=\psi(g(f(\bm{x};\bm{ \theta});\bm{w})),$ $$

where $\bm{w}$ denotes the parameters of C-Adapter. While ConfTr alters the parameters of trained classifiers $\bm{\theta}$ through retraining or fine-tuning, we only update a few trainable parameters $\bm{w}$ added for conformal prediction. In addition to enhancing training efficiency, the adapter-based tuning method requires access only to the model outputs. This makes it compatible with black-box models (e.g., online APIs) and other modern neural networks (e.g., Radford et al. (, CLIP)).

Importantly, the adapter requires to be learned within a hypothesis space that can provably guarantee preserving the accuracy of the original network $f$.
To achieve that, we implement the adapter as a class of intra order-preserving functions (Rahimi et al., ), a family of functions that is both necessary and sufficient to keep the top-$k$ accuracy of the original network unchanged.
Formally, a function $h:\mathbb{R}^{K}\rightarrow\mathbb{R}^{K}$ is intra order-preserving, if, for all $i,j\in[K]$ and any vector $\bm{x}\in\mathbb{R}^{K}$, $\bm{x}_{i}>\bm{x}_{j}$ (or $\bm{x}_{i}=\bm{x}_{j}$) if and only if $h_{i}(\bm{x})>h_{j}(\bm{x})$ (or $h_{i}(\bm{x})=h_{j}(\bm{x})$).
For convenience, we use $\bm{f}$ to indicate the model output $f(\bm{x};\bm{\theta})$.
We denote $R:\mathbb{R}^{K}\rightarrow\mathbb{U}^{K}$ as the sorting function, where $\mathbb{U}^{K}\subset\{0,1\}^{K\times K}$ represents the set of $K\times K$ permutation matrices.
We have $\bm{r}=R(\bm{f})\bm{f}$ as the sorted $\bm{f}$, satisfying $\bm{r}_{1}>\cdots>\bm{r}_{K}$.
We use $U$ to denote the $K\times K$ upper-triangular matrix of ones.

To ensure that C-Adapter belongs to the class of intra order-preserving functions, we define it by

$$ $g(\bm{f};\bm{w})=R(\bm{f})^{-1}U\Psi(\bm{f}),$ $$

where the $i$-th term of $\Psi(\bm{f})$ is formulated as:

$$ $\Psi_{i}(\bm{f})=\begin{cases}\sqrt{(\bm{r}_{i}-\bm{r}_{i+1})}\sigma(\varphi_{ i}(\bm{f}))&\text{for }i<K,\\ \varphi_{K}(\bm{f})&\text{for }i=K.\end{cases}$ $$

Here, $\varphi(\bm{f})=\bm{w}\cdot\bm{f}+\bm{w}^{\prime}$, and $\sigma$ represents the sigmoid function. We denote $\varphi_{i}(\bm{f})$ as the $i$-th component of $\varphi(\bm{f})$.
We outline the workflow in Figure 3.
Since $\Psi(\bm{f})$ is continuous in $\bm{f}$, it is straightforward to verify that this structure satisfies the requirements of the intra order-preserving family (Rahimi et al., , Theorem 1).
We provide a detailed description of the intra order-preserving family and its relationship to our network structure in Appendix C.
This structure decouples the logit order from the adaptation for conformal prediction, allowing C-Adapter to focus on optimizing efficiency. We demonstrate the superiority of this adaptation strategy over others in Figures 7 and 6.

#### Training objective

ConfTr optimizes the efficiency of conformal predictors at a predetermined error rate (e.g., $\alpha=0.01$), which may result in suboptimal performance when predicting with a different coverage rate. To address this issue, we consider a general criterion for efficiency:

$$ $\mathbb{E}_{\bm{x}\sim\mathcal{P}_{\mathcal{X}}}\left[\int_{0}^{1}\left| \mathcal{C}(\bm{x};\tau_{\alpha},\widetilde{\pi}_{\bm{w}})\right|\,\mathrm{d} \alpha\right],$ (4) $$

which measures the definite integral of efficiency over $\alpha\in(0,1)$. For notation shorthand, we use $\widetilde{\pi}_{\bm{w}}$ to indicate that the underlying classifier $f$ is equipped with C-Adapter, parameterized by $\bm{w}$. This objective is analogous to the AUC in classification (Cortes & Mohri, ), as AUC reflects the classifier’s performance across all possible thresholds, while classification error considers only a single fixed one. However, the objective in Equation (4) cannot be directly computed from a given dataset. To address this issue, we translate it into an equivalent form that can be explicitly calculated.

From Equation (2), we can infer that we construct the conformal prediction set for $\hat{X}\sim\mathcal{P}_{\mathcal{X}}$ at $\alpha$ by comparing the non-conformity score $S(\hat{X},y;\widetilde{\pi}_{\bm{w}})$ with $\tau_{\alpha}$ for each $y\in\mathcal{Y}$. Therefore, it is straightforward to verify that the expected set size at the error rate $\alpha$ over the data distribution $\mathcal{P}_{\mathcal{X}}$ is determined by the probability of the event $\{\tau_{\alpha}\geq S(\hat{X},\hat{Y};\widetilde{\pi}_{\bm{w}})\}$, where $\hat{X}\sim\mathcal{P}_{\mathcal{X}}$ and $\hat{Y}\sim\text{Uniform}(\mathcal{Y})$. When extending to any $\alpha\in(0,1)$, the threshold $\tau_{\alpha}$ can be the non-conformity score of any observation $(X,Y)\sim\mathcal{P}_{\mathcal{X}\mathcal{Y}}$. This prompts us to consider the following probability:

$$ $\mathbb{P}\left(S(X,Y;\widetilde{\pi}_{\bm{w}})\geq S(\hat{X},\hat{Y}; \widetilde{\pi}_{\bm{w}})\right),\text{where }(X,Y)\sim\mathcal{P}_{\mathcal{X }\mathcal{Y}},\hat{X}\sim\mathcal{P}_{\mathcal{X}},\hat{Y}\sim\textrm{Uniform} (\mathcal{Y}).$ (5) $$

In particular, this probability quantifies the likelihood that the non-conformity score of a randomly matched data-label pair $(\hat{X},\hat{Y})$ is not greater than that of a correctly matched pair $(X,Y)$ . This probability approaches zero when the scores of correctly and incorrectly matched data-label pairs are well distinguishable, and approaches $1/2$ when they are not effectively distinguished. In the following, we present a formal analysis demonstrating that minimizing the probability in Equation (5) is equivalent to optimizing the overall efficiency defined in Equation (4).

###### Proposition 1 .

Let $\hat{\pi}$ and $\hat{\pi}^{\prime}$ be pre-trained classifiers with parameters $\theta$ and $\theta^{\prime}$, respectively, and let $S$ be a specific non-conformity score function. We denote $\mathcal{P}_{S_{\theta}}$ and $\mathcal{P}_{S_{\theta^{\prime}}}$ as the distributions of $S(X,Y;\hat{\pi})$ and $S(X,Y;\hat{\pi}^{\prime})$, where $(X,Y)\sim\mathcal{P}_{\mathcal{X}\mathcal{Y}}$. Let $F_{S_{\theta}}$, and $F_{S_{\theta^{\prime}}}$ be the CDF corresponding to $\mathcal{P}_{S_{\theta}}$ and $\mathcal{P}_{S_{\theta^{\prime}}}$. Given that $\hat{X}\sim\mathcal{P}_{\mathcal{X}}$ and $\hat{Y}$ follows a uniform distribution over $\mathcal{Y}$, we have

$$ $\mathbb{P}\left(S(X,Y;\hat{\pi})\geq S(\hat{X},\hat{Y};\hat{\pi})\right)> \mathbb{P}\left(S(X,Y;\hat{\pi}^{\prime})\geq S(\hat{X},\hat{Y};\hat{\pi}^{ \prime})\right)$ $$

holds if and only if

$$ $\mathbb{E}_{X\sim\mathcal{P}_{\mathcal{X}}}\left[\int_{0}^{1}|\mathcal{C}\left (X;F^{-1}_{S_{\theta}}(1-\alpha),\hat{\pi}\right)|\,\mathrm{d}\alpha\right]> \mathbb{E}_{X\sim\mathcal{P}_{\mathcal{X}}}\left[\int_{0}^{1}|\mathcal{C}\left (X;F^{-1}_{S_{\theta^{\prime}}}(1-\alpha),\hat{\pi}^{\prime}\right)|\,\mathrm{ d}\alpha\right].$ $$

Figure: (a) w/o C-Adapter
Refer to caption: x5.png

The proof of Proposition 1 is provided in Appendix D. Here, the inverse CDF calculates the $(1-\alpha)$-th quantile of the score distributions, which determines the threshold $\tau_{\alpha}$. Then, to optimize overall efficiency in Equation (4), we turn to minimize the following objective, rewritten from Equation (5):

$$ $\mathcal{L}(\bm{w})=\mathbb{E}\left[\mathds{1}_{\{S(X,Y;\widetilde{\pi}_{\bm{w }})>S(\hat{X},\hat{Y};\widetilde{\pi}_{\bm{w}}\}}\right],$ (6) $$

where $(X,Y)\sim\mathcal{P}_{\mathcal{X}\mathcal{Y}},\hat{X}\sim\mathcal{P}_{\mathcal
{X}}$, and $\hat{Y}\sim\text{Uniform}(\mathcal{Y})$.
Given the non-differentiability of the indicator function, it is common practice to utilize surrogate functions as differentiable approximations (Yan et al., ; Yuan et al., ). In this work, we apply the sigmoid function with a parameter $T$ as the surrogate, defined as $\sigma_{T}(x)=1/\left(1+\exp\left(-x/T\right)\right)$. For the score function utilized during training, we employ either THR or APS. The differentiable APS is implemented as outlined in ConfTr (Stutz et al., ). Ultimately, the convex relaxation of Equation (6) is given by

$$ $\widetilde{\mathcal{L}}(\bm{w})=\mathbb{E}\left[\sigma_{T}\left(S(X,Y; \widetilde{\pi}_{\bm{w}})-S(\hat{X},\hat{Y};\widetilde{\pi}_{\bm{w}})\right) \right].$ (7) $$

By optimizing this objective, the scores of correctly and incorrectly matched data-label pairs become more distinguishable: correctly matched pairs are encouraged to have relatively smaller non-conformity scores compared to incorrectly matched pairs. We visualize this effect in Figure 4. With C-Adapter, the APS scores of incorrect labels become significantly higher than those of correct labels, leading to more efficient prediction sets across varying coverage rates. Moreover, our proposed objective achieves superior average performance compared to the size loss of ConfTr (see Table 2).

###### Proposition 1 .

#### Batched optimization

In the $t$-th iteration, we construct an auxiliary batch $\hat{\mathcal{B}}_{t}$ by creating $K$ data-label pairs for each instance in $\mathcal{B}_{t}$. Each pair $(\hat{\bm{x}},\hat{y})$ in $\hat{\mathcal{B}}_{t}$ consists of an instance from $\mathcal{B}_{t}$ and one of the $K$ possible labels $\hat{y}\in\mathcal{Y}$. Subsequently, we update the parameters $\bm{w}$ of C-Adapter by

$$ $\bm{w}^{(t)}\leftarrow\bm{w}^{(t-1)}-\eta_{t}\cdot\nabla_{\bm{w}}\left[\frac{1 }{|\mathcal{B}_{t}|\cdot|\hat{\mathcal{B}}_{t}|}\sum_{(\bm{x},y)\in\mathcal{B} _{t}}\sum_{(\hat{\bm{x}},\hat{y})\in\hat{\mathcal{B}}_{t}}\sigma_{T}\left(S \left(\bm{x},y;\widetilde{\pi}_{\bm{w}}\right)-S\left(\hat{\bm{x}},\hat{y}; \widetilde{\pi}_{\bm{w}}\right)\right)\right].$ (8) $$

The optimization incurs low computational costs, as we only update the parameters of the linear layers in C-Adapter. In practical applications, we tune the parameters of C-Adapter using the training set for the trained classifier $f$. Our method can also be implemented with a hold-out set, which is explicitly validated in Figure 9. Noticeably, our method offers several compelling advantages:

- •
Flexible: C-Adapter can enhance the efficiency of conformal predictors across different non-conformity score functions, not limited to the one employed during its tuning (see Table 1 and Table 5). By default, we tune C-Adapter using THR.
- •
Easy to use: C-Adapter is insensitive to the parameter $T$ and requires minimal hyperparameter tuning (see Figure 8). Moreover, our method demonstrates high computational efficiency and a rapid convergence rate (refer to the convergence analysis in Appendix E).
- •
Model-agnostic: C-Adapter requires access only to the model outputs and integrates effortlessly with any classifier. Our method can effectively adapt trained classifiers for efficient prediction sets, regardless of the network architecture or pre-training strategy.

## 4 Experiments

### 4.1 Experimental Setup

#### Dataset

We evaluate our approach using three benchmarks of image classification: CIFAR-100 (Krizhevsky et al., ), ImageNet (Deng et al., ), and ImageNet-V2 (Recht et al., ). For CIFAR-100 and ImageNet-V2, we randomly split the test sets into calibration and test subsets, each containing 5,000 samples. For ImageNet, we partition the 50,000-sample test dataset into a calibration subset of 30,000 samples and a test subset of 20,000 samples.

#### Models

For our evaluations, we utilize four well-established deep image classifiers: ResNet101 (RN101) (He et al., ), two variants of DenseNet (DN121 and DN161) (Huang et al., ), and ResNeXt50 (RNX50) (Xie et al., ). Additionally, we employ the Vision-Language Model CLIP (Radford et al., ), which is based on a Vision Transformer architecture (ViT-B/16) (Dosovitskiy et al., ). For ImageNet, we leverage pre-trained deep image classifiers from TorchVision (Paszke et al., ), whereas for CIFAR-100, we train the classifiers from scratch using the entire training set. For CLIP, we rely on its inherent zero-shot capabilities to perform classification tasks.

#### Training details

C-Adapter is tuned for 240 iterations using Adam (Kingma & Ba, ), with a batch size of 256 and a learning rate of 0.1.
The parameter $T$ is set to 0.0001 by default.
We partition the calibration set into a validation subset and a calibration subset in an 20:80 ratio, with the validation set used for early stopping.
When a validation set is not necessary, the entire calibration set is employed for calibration, ensuring all methods have access to the same dataset.
To ensure the reliability of our results, each experiment is repeated 10 times, and the average result is reported. All experiments are conducted on an NVIDIA GeForce RTX 4090 using PyTorch (Paszke et al., ).

#### Evaluation metrics

The primary metrics for evaluating prediction sets are: (1) efficiency (Size) and (2) marginal coverage rate (Coverage). Moreover, we assess conditional coverage using (1) class-conditional coverage gap (CovGap) (Ding et al., ) and (2) size-stratified coverage violation (SSCV) (Angelopoulos et al., ). We detail these metrics in Appendix F.

### 4.2 Results

#### C-Adapter improves the efficiency of conformal predictors.

In Table 1, we present the performance of THR, APS, and RAPS with C-Adapter on ImageNet and CIFAR-100.
A salient observation is that our method drastically improves the efficiency of conformal predictors with the desired coverage rate.
For example, C-Adapter reduces the size of APS from 16.43 to 4.23 on ImageNet using DN161 with $\alpha=0.05$.
Notably, the improvements remain substantial when there is a mismatch between the score functions used during adapter tuning (THR) and those employed in conformal prediction (APS and RAPS).
When C-Adapter is tuned with APS, similar enhancements are observed with both THR and RAPS, as detailed in Appendix H.
This highlights the flexibility of our method.
Overall, empirical results show that C-Adapter can enhance the efficiency of conformal predictors across various score functions, regardless of model architectures and pre-training strategies.

**Table 1: Performance of C-Adapter on common benchmarks. $\downarrow$ indicates that a smaller value is better. Results in bold indicate superior performance. C-Adapter is tuned using THR.**
|  |  | w/o C-Adapter $\backslash$ w/ C-Adapter |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Score | Model | ImageNet | CIFAR-100 |  |  |  |  |  |  |
| $\alpha=0.05$ | $\alpha=0.1$ | $\alpha=0.05$ | $\alpha=0.1$ |  |  |  |  |  |  |
|  |  | Coverage | Size ($\downarrow$) | Coverage | Size ($\downarrow$) | Cover | Size ($\downarrow$) | Coverage | Size ($\downarrow$) |
| THR | RN101 | 0.95 $\backslash$ 0.95 | 4.03 $\backslash$ 3.82 | 0.90 $\backslash$ 0.90 | 1.91 $\backslash$ 1.89 | 0.95 $\backslash$ 0.95 | 3.64 $\backslash$ 3.17 | 0.90 $\backslash$ 0.90 | 1.87 $\backslash$ 1.76 |
| DN121 | 0.95 $\backslash$ 0.95 | 5.66 $\backslash$ 5.35 | 0.90 $\backslash$ 0.90 | 2.42 $\backslash$ 2.34 | 0.95 $\backslash$ 0.95 | 3.27 $\backslash$ 3.00 | 0.90 $\backslash$ 0.90 | 1.72 $\backslash$ 1.70 |  |
| DN161 | 0.95 $\backslash$ 0.95 | 4.03 $\backslash$ 3.69 | 0.90 $\backslash$ 0.90 | 1.89 $\backslash$ 1.82 | 0.95 $\backslash$ 0.95 | 2.91 $\backslash$ 2.75 | 0.90 $\backslash$ 0.90 | 1.72 $\backslash$ 1.69 |  |
| RNX50 | 0.95 $\backslash$ 0.95 | 4.26 $\backslash$ 3.87 | 0.90 $\backslash$ 0.90 | 1.87 $\backslash$ 1.85 | 0.95 $\backslash$ 0.95 | 3.41 $\backslash$ 3.09 | 0.90 $\backslash$ 0.90 | 1.78 $\backslash$ 1.76 |  |
| CLIP | 0.95 $\backslash$ 0.95 | 6.88 $\backslash$ 6.71 | 0.90 $\backslash$ 0.90 | 3.33 $\backslash$ 3.25 | 0.95 $\backslash$ 0.95 | 9.71 $\backslash$ 8.25 | 0.90 $\backslash$ 0.90 | 4.78 $\backslash$ 4.36 |  |
| Average | 0.95 $\backslash$ 0.95 | 4.97 $\backslash$ 4.69 | 0.90 $\backslash$ 0.90 | 2.29 $\backslash$ 2.23 | 0.95 $\backslash$ 0.95 | 4.59 $\backslash$ 4.05 | 0.90 $\backslash$ 0.90 | 2.37 $\backslash$ 2.25 |  |
| APS | RN101 | 0.95 $\backslash$ 0.95 | 14.73 $\backslash$ 3.98 | 0.90 $\backslash$ 0.90 | 7.23 $\backslash$ 2.30 | 0.95 $\backslash$ 0.95 | 7.60 $\backslash$ 3.19 | 0.90 $\backslash$ 0.90 | 3.95 $\backslash$ 1.86 |
| DN121 | 0.95 $\backslash$ 0.95 | 20.00 $\backslash$ 5.73 | 0.90 $\backslash$ 0.90 | 9.21 $\backslash$ 2.86 | 0.95 $\backslash$ 0.95 | 10.20 $\backslash$ 3.08 | 0.90 $\backslash$ 0.90 | 5.39 $\backslash$ 1.85 |  |
| DN161 | 0.95 $\backslash$ 0.95 | 16.43 $\backslash$ 4.23 | 0.90 $\backslash$ 0.90 | 6.82 $\backslash$ 2.33 | 0.95 $\backslash$ 0.95 | 9.90 $\backslash$ 2.86 | 0.90 $\backslash$ 0.90 | 5.42 $\backslash$ 1.80 |  |
| RNX50 | 0.95 $\backslash$ 0.95 | 21.54 $\backslash$ 4.26 | 0.90 $\backslash$ 0.90 | 8.92 $\backslash$ 2.32 | 0.95 $\backslash$ 0.95 | 9.95 $\backslash$ 3.26 | 0.90 $\backslash$ 0.90 | 5.14 $\backslash$ 1.91 |  |
| CLIP | 0.95 $\backslash$ 0.95 | 26.35 $\backslash$ 7.98 | 0.90 $\backslash$ 0.90 | 13.24 $\backslash$ 3.94 | 0.95 $\backslash$ 0.95 | 16.13 $\backslash$ 13.50 | 0.90 $\backslash$ 0.90 | 10.18 $\backslash$ 8.70 |  |
| Average | 0.95 $\backslash$ 0.95 | 19.81 $\backslash$ 5.24 | 0.90 $\backslash$ 0.90 | 9.08 $\backslash$ 2.75 | 0.95 $\backslash$ 0.95 | 10.76 $\backslash$ 5.18 | 0.90 $\backslash$ 0.90 | 6.01 $\backslash$ 3.22 |  |
| RAPS | RN101 | 0.95 $\backslash$ 0.95 | 7.13 $\backslash$ 3.75 | 0.90 $\backslash$ 0.90 | 4.60 $\backslash$ 2.25 | 0.95 $\backslash$ 0.95 | 5.16 $\backslash$ 4.43 | 0.90 $\backslash$ 0.90 | 3.25 $\backslash$ 1.81 |
| DN121 | 0.95 $\backslash$ 0.95 | 10.28 $\backslash$ 6.53 | 0.90 $\backslash$ 0.90 | 6.57 $\backslash$ 2.80 | 0.95 $\backslash$ 0.95 | 7.19 $\backslash$ 3.74 | 0.90 $\backslash$ 0.90 | 4.50 $\backslash$ 1.80 |  |
| DN161 | 0.95 $\backslash$ 0.95 | 7.31 $\backslash$ 4.10 | 0.90 $\backslash$ 0.90 | 4.63 $\backslash$ 2.27 | 0.95 $\backslash$ 0.95 | 7.10 $\backslash$ 3.15 | 0.90 $\backslash$ 0.90 | 4.59 $\backslash$ 1.79 |  |
| RNX50 | 0.95 $\backslash$ 0.95 | 7.87 $\backslash$ 4.11 | 0.90 $\backslash$ 0.90 | 5.20 $\backslash$ 2.26 | 0.95 $\backslash$ 0.95 | 7.20 $\backslash$ 3.94 | 0.90 $\backslash$ 0.90 | 4.47 $\backslash$ 1.89 |  |
| CLIP | 0.95 $\backslash$ 0.95 | 15.14 $\backslash$ 7.82 | 0.90 $\backslash$ 0.90 | 9.25 $\backslash$ 3.49 | 0.95 $\backslash$ 0.95 | 14.52 $\backslash$ 11.19 | 0.90 $\backslash$ 0.90 | 9.41 $\backslash$ 7.62 |  |
| Average | 0.95 $\backslash$ 0.95 | 9.55 $\backslash$ 5.26 | 0.90 $\backslash$ 0.90 | 6.05 $\backslash$ 2.61 | 0.95 $\backslash$ 0.95 | 8.24 $\backslash$ 5.45 | 0.90 $\backslash$ 0.90 | 5.24 $\backslash$ 2.98 |  |

#### C-Adapter outperforms ConfTr.

ConfTr (Stutz et al., ) can be employed as a fine-tuning method to adapt classifiers for conformal prediction.
Initially, the classifier is trained with cross-entropy loss, and then only the fully connected layer is tuned using the objective in Equation (3). We compare this approach with ours on CIFAR-100. For ConfTr, we set the learning rate to $0.001$ with a batch size of 256. A higher learning rate significantly decreases classification accuracy, leading to a dramatic decline in efficiency. The parameters $T$ and $\lambda$ are tuned from the sets $\{0.01,0.1,0.5,1\}$ and $\{0.005,0.01,0.05,0.1,0.2\}$, respectively. During training, we utilize THRLP (Stutz et al., ) for ConfTr, setting $\alpha$ to 0.01. For evaluation, we employ THR, APS, and RAPS with $\alpha=0.1$.

Our results in Figure 5 illustrate the superior performance of our approach. For APS and RAPS, both C-Adapter and ConfTr improve the efficiency of conformal predictors, with C-Adapter demonstrating superior performance. Furthermore, C-Adapter enhances the efficiency of THR, whereas ConfTr does not. Additionally, we apply C-Adapter to models that have already been fine-tuned using ConfTr. The results indicate that our approach can further improve the performance of ConfTr. Notably, Baseline+C-Adapter outperforms ConfTr+C-Adapter, suggesting that the accuracy decline associated with ConfTr limits the efficiency of conformal predictors. Overall, empirical results demonstrate that C-Adapter not only surpasses ConfTr but can also enhance its performance.

Figure: (a) THR
Refer to caption: x7.png

Figure: Figure 6: Accuracy of various adaptation strategies, on CIFAR-100. Both Retraining and Fine-tuning result in 3-5% lower accuracy compared to the baseline.
Refer to caption: x10.png

#### Ablation study on the adaptation strategy

To further demonstrate the significance of this adapter-based tuning method, this ablation compares our approach with two alternative strategies: (1) Retraining, which involves training the classifier from scratch with our proposed loss function, and (2) Fine-tuning, where the classifier is initially trained with cross-entropy loss and subsequently fine-tuned only on the fully connected layer with our loss.
The second strategy is analogous to ConfTr, but it employs a different loss function.
In our approach, we first train the classifier using cross-entropy loss and then adapt it for conformal prediction with C-Adapter.
This ablation employs a consistent loss function to ensure a fair comparison among different adaptation strategies.
We provide the detailed experimental setup for the competing methods in Appendix G.2.

As demonstrated in Figure 6, both Retraining and Fine-tuning result in 3-5% lower accuracy compared to the baseline. Our results in Figure 7 empirically demonstrate that this decline in accuracy limits overall efficiency: while all three adaptation strategies can enhance the efficiency of APS and RAPS, our method significantly outperforms the others. The negative impact of decreased accuracy is particularly evident in THR, where only our method achieves an improvement in efficiency. Overall, this ablation study further highlights the superiority of our adaptation strategy.

#### Ablation study on the loss function

The size loss from ConfTr can also be utilized to tune our conformal adapter. We conduct an ablation study on ImageNet using DN121, comparing C-Adapter with size loss against our proposed loss function. For the size loss, we maintain a consistent experimental setup and tune the parameter $T$ within the range $\{0.0001,0.001,0.01,0.1\}$, while setting the error rate $\alpha$ to $0.01$ during training. For evaluation, we use THR and APS at various coverage rates.

Our results in Table 2 indicate that C-Adapter effectively integrates with size loss, enhancing the efficiency of conformal predictors regardless of the employed score function. However, our proposed loss function achieves superior average performance. Notably, size loss performs poorly at small error rates $\alpha$; it exhibits inferior performance compared to the baseline when utilizing THR at $\alpha=0.01$ and $\alpha=0.02$, while our method consistently outperforms the baseline. Overall, this analysis highlights the flexibility of C-Adapter and the efficacy of our proposed loss function.

**Table 2: Comparison of C-Adapter with different loss functions, on ImageNet with DN121. Baseline represents the scenario without C-Adapter. Since each entry achieves the desired coverage, only Size is presented. Our loss achieves superior average performance compared to the size loss.**
|  | THR | APS |  |  |  |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| ${\bm{\alpha}}$ | 0.06 | 0.05 | 0.04 | 0.03 | 0.02 | 0.01 | Average | 0.06 | 0.05 | 0.04 | 0.03 | 0.02 | 0.01 | Average |
| Baseline | 4.35 | 5.66 | 7.26 | 10.46 | 15.91 | 33.84 | 12.91 | 15.94 | 20.00 | 24.42 | 32.62 | 48.13 | 91.49 | 38.77 |
| size loss | 4.26 | 5.33 | 7.04 | 9.93 | 17.44 | 43.16 | 14.53 | 4.48 | 5.71 | 7.39 | 10.82 | 18.82 | 42.63 | 14.98 |
| Ours | 4.27 | 5.35 | 6.94 | 9.75 | 15.01 | 30.31 | 11.94 | 4.44 | 5.73 | 7.37 | 10.70 | 17.30 | 36.24 | 13.63 |

Figure: (a) THR
Refer to caption: x11.png

#### C-Adapter can reduce conditional coverage violations.

As shown in Table 1, C-Adapter can enhance the efficiency of THR, APS, and RAPS. However, unlike THR, which seeks optimal efficiency with limited conditional coverage, APS is designed to improve the conditional coverage of prediction sets. RAPS also aims to enhance conditional coverage while simultaneously boosting efficiency. In this study, we further demonstrate that C-Adapter can enhance the conditional coverage of APS and RAPS. All experimental setups remain consistent, except that training concludes at the iteration corresponding to the optimal SSCV instead of Size on the validation set.

As detailed in Table 3, C-Adapter consistently reduces Size, SSCV, and CovGap in most cases under this setting. Notably, the reduction in Size is less substantial compared to the results in Table 1.
Thus, users can adopt an early stopping strategy that best aligns with their specific needs for efficiency and conditional coverage.
Overall, this experiment validates that C-Adapter can enhance the conditional coverage of APS and RAPS while simultaneously improving their efficiency.

**Table 3: Experimental results on conditional coverage. This experiment is conducted on ImageNet. $\downarrow$ indicates that smaller values are preferable. Since each entry achieves the desired coverage, Coverage is omitted. C-Adapter consistently reduces Size, SSCV, and CovGap in most cases.**
|  |  | Size $\downarrow$ / SSCV $\downarrow$ / CovGap $\downarrow$ |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- |
|  |  | RN101 | DN121 | DN161 | RNX50 | CLIP | Average |
| $\alpha=0.05$ | APS | 14.73 / 3.07 / 4.30 | 20.00 / 2.48 / 4.39 | 16.43 / 3.16 / 4.49 | 21.54 / 5.09 / 4.50 | 26.35 / 3.25 / 4.94 | 19.81 / 3.41 / 4.52 |
| +Ours | 11.00 / 2.05 / 4.27 | 13.39 / 1.79 / 4.37 | 10.87 / 2.69 / 4.38 | 12.98 / 2.93 / 4.38 | 17.21 / 2.23 / 4.90 | 13.09 / 2.34 / 4.46 |  |
| RAPS | 7.13 / 2.35 / 4.37 | 10.28 / 2.95 / 4.50 | 7.31 / 3.17 / 4.36 | 7.87 / 3.83 / 4.60 | 15.14 / 1.78 / 5.07 | 9.55 / 2.82 / 4.58 |  |
| +Ours | 6.98 / 1.99 / 4.27 | 8.87 / 1.88 / 4.61 | 6.48 / 2.34 / 4.31 | 6.79 / 2.67 / 4.52 | 12.55 / 1.71 / 4.97 | 8.41 / 2.12 / 4.54 |  |
| $\alpha=0.1$ | APS | 7.23 / 5.97 / 6.03 | 9.21 / 5.76 / 5.69 | 6.82 / 5.76 / 5.70 | 8.92 / 7.26 / 6.09 | 13.24 / 6.87 / 7.47 | 9.08 / 6.32 / 6.20 |
| +Ours | 5.75 / 5.85 / 5.93 | 6.56 / 1.89 / 5.75 | 5.21 / 1.91 / 5.77 | 6.48 / 4.36 / 6.05 | 9.38 / 3.51 / 7.44 | 6.68 / 3.78 / 6.19 |  |
| RAPS | 4.60 / 4.56 / 6.15 | 6.57 / 2.98 / 5.71 | 4.63 / 3.93 / 6.12 | 5.20 / 2.87 / 6.16 | 9.25 / 3.44 / 7.51 | 6.05 / 3.56 / 6.33 |  |
| +Ours | 4.45 / 4.12 / 6.02 | 5.89 / 2.87 / 5.78 | 4.31 / 3.73 / 6.18 | 4.75 / 2.67 / 6.13 | 7.82 / 2.40 / 7.51 | 5.44 / 3.16 / 6.32 |  |

Figure: (a) THR
Refer to caption: x14.png

#### How does the parameter T 𝑇 T italic_T affect the performance of C-Adapter?

In Figure 8, we ablate how the parameter $T$ introduced by the surrogate function affects the efficiency of conformal predictors, using THR, APS, and RAPS. We set the error rate $\alpha$ to 0.05. Notably, the average set size remains stable, showing no significant variations with changes in $T$ across different score functions. For simplicity, we use a moderate parameter range and set $T$ to $10^{-4}$ throughout our experiments.

#### C-Adapter shows robustness to distribution shifts.

We further investigate the robustness of C-Adapter to distribution shifts. Specifically, we tune C-Adapter using the training set of ImageNet and split ImageNet-V2 into two equal-sized calibration and test sets. Notably, coverage will not be affected under this setting, as the calibration and test sets remain exchangeable. We examine the performance of C-Adapter on APS, THR, and RAPS at $\alpha=0.1$ and $\alpha=0.2$. As demonstrated in Table 4, C-Adapter consistently reduces Size across various base classifiers on ImageNet-V2, regardless of the score function employed in conformal prediction or the predefined error rate $\alpha$. For example, when evaluated on DN161 with $\alpha=0.1$, C-Adapter reduces the Size of APS from 19.32 to 5.21. Overall, these results highlight the robustness of C-Adapter to shifts in data distribution.

Figure: Figure 9: Comparison of tuning C-Adapter using a hold-out set vs. the original training set on ImageNet with DN121.
Refer to caption: x17.png

#### Does C-Adapter perform better with a hold-out set for training?

In the original setup, we train C-Adapter using the training set for the classifier $f$. In this ablation, we investigate whether C-Adapter benefits from using a hold-out set for training. Specifically, we randomly divide the ImageNet test set into 10,000 samples for training, 20,000 for calibration and validation, and 20,000 for testing. C-Adapter is tuned for 2 epochs using the 10,000-sample training set with Adam, a batch size of 256, and a learning rate of 0.1, while the parameter $T$ is set to 0.0001. For evaluation, we utilize THR and APS with an error rate of 0.05.

Our results in Figure 9
indicate that C-Adapter demonstrates improved performance when utilizing a hold-out set for training, irrespective of the score function employed. However, this improvement is not statistically significant. We conclude that using the original training set is sufficient for adapting the classifier to achieve more efficient prediction sets, which also enables more efficient data utilization.

**Table 4: The robustness of C-Adapter to distribution shift. C-Adapter is tuned using ImageNet and tested on ImageNet-V2. Since each entry achieves the desired coverage, only Size is presented.**
|  | w/o C-Adapter $\backslash$ w/ C-Adapter |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
| Model | THR | APS | RAPS |  |  |  |
|  | $\alpha=$0.1 | $\alpha=$0.2 | $\alpha=$0.1 | $\alpha=$0.2 | $\alpha=$0.1 | $\alpha=$0.2 |
| RN101 | 6.03 $\backslash$ 5.43 | 2.11 $\backslash$ 2.01 | 19.65 $\backslash$ 5.59 | 7.17 $\backslash$ 2.57 | 10.90 $\backslash$ 7.01 | 5.67 $\backslash$ 2.29 |
| DN121 | 8.01 $\backslash$ 7.70 | 2.60 $\backslash$ 2.52 | 24.73 $\backslash$ 8.14 | 9.13 $\backslash$ 3.21 | 14.31 $\backslash$ 10.38 | 7.20 $\backslash$ 3.01 |
| DN161 | 5.41 $\backslash$ 4.72 | 2.06 $\backslash$ 1.91 | 19.32 $\backslash$ 5.21 | 6.31 $\backslash$ 2.52 | 10.27 $\backslash$ 5.98 | 5.18 $\backslash$ 2.18 |
| RNX50 | 6.80 $\backslash$ 5.78 | 2.07 $\backslash$ 2.05 | 26.27 $\backslash$ 6.11 | 8.58 $\backslash$ 2.63 | 11.43 $\backslash$ 7.83 | 6.14 $\backslash$ 2.38 |
| CLIP | 5.66 $\backslash$ 5.59 | 2.31 $\backslash$ 2.29 | 20.73 $\backslash$ 14.88 | 8.21 $\backslash$ 6.60 | 10.60 $\backslash$ 8.67 | 6.35 $\backslash$ 6.10 |
| Average | 6.38 $\backslash$ 5.84 | 2.23 $\backslash$ 2.16 | 22.14 $\backslash$ 7.99 | 7.88 $\backslash$ 3.51 | 11.50 $\backslash$ 7.97 | 6.11 $\backslash$ 3.19 |

## 5 Conclusion

In this paper, we introduce C-Adapter, an adapter-based tuning method to enhance the efficiency of conformal predictors.
Our key idea is to adapt the trained classifiers for conformal prediction while preserving the ranking of labels in the output logits, thereby maintaining the top-$k$ accuracy of the classifiers.
To achieve this, we implement the adapter as a class of intra order-preserving functions.
For the optimization of C-Adapter, we propose a loss function that enhances the discriminability of non-conformity scores between correctly and randomly matched data-label pairs.
Extensive experiments demonstrate that C-Adapter effectively adapts various classifiers for efficient prediction sets and enhances the conformal training method.
Our method is user-friendly, as it is insensitive to hyperparameters and computationally efficient.
We hope the insights from this work will inspire future research to explore more effective model adaptation strategies for conformal prediction.

#### Limitation

Although our adaptation strategy demonstrates promise, we focus solely on using it to optimize the efficiency of conformal predictors. Developing targeted loss functions to adapt deep classifiers for other aspects of conformal prediction (e.g., conditional coverage or robustness) is not explored in this work and offers an interesting direction for future research.

## Appendix A Related Work

Conformal prediction is a statistical framework for uncertainty quantification that provides a finite-sample coverage guarantee (Angelopoulos & Bates, ; Balasubramanian et al., ; Shafer & Vovk, ). This framework has found diverse applications across various domains, including classification (Sadinle et al., ), regression (Romano et al., ), and more specialized areas such as large language models (Su et al., ; Cherian et al., ), graph neural networks (Zargarbashi et al., ), image generative models (Horwitz & Hoshen, ), hyperspectral imaging (Liu et al., ), robotic control (Wang et al., ), and autonomous systems (Lindemann et al., ).

In this work, we focus on the split conformal prediction framework (Vovk et al., ; Angelopoulos & Bates, ), where the training and calibration sets are disjoint. In addition, we specifically target the classification task. A key research direction in this area is the design of powerful score functions. Various score functions have been proposed to enhance the efficiency or adaptiveness of prediction sets, including THR (Sadinle et al., ), APS (Romano et al., ), RAPS (Angelopoulos et al., ), SAPS (Huang et al., ), and RANK (Luo & Zhou, ). However, despite significant progress in developing score functions, conformal prediction is typically applied as a post-hoc process for trained classifiers, which may already be either unnecessarily conservative or overconfident. This separate processing can result in suboptimal efficiency of conformal predictors.

#### Adapting deep classifiers for conformal prediction.

To address the aforementioned issue, several works propose training (fine-tuning) time regularizations to improve the performance of conformal predictors (Stutz et al., ; Einbinder et al., ; Correia et al., ; Huang et al., ). The uncertainty-aware conformal loss function (Einbinder et al., ) optimizes the performance of conformal predictors by encouraging the non-conformity scores to follow a uniform distribution, specifically focusing on optimizing APS. To optimize the classifier for maximum predictive efficiency, ConfTr (Stutz et al., ) modifies the training objective by introducing a regularization term that minimizes the average prediction set size at a specific error rate. However, this regularization term can negatively impact classifier accuracy by making it more challenging to converge to an optimal solution, thereby limiting the overall efficiency of the conformal predictor. Similar works (Huang et al., ; Correia et al., ) adopt the ConfTr framework to enhance the efficiency of conformal predictors, yet they still encounter the limitations of ConfTr. Motivated by this, we propose C-Adapter, which enables the efficient adaptation of trained classifiers for conformal prediction without sacrificing classification accuracy. Additionally, C-Adapter can broadly improve various score functions and enhance ConfTr, serving as a unique complement to existing approaches.

## Appendix B Vital Techniques in Conformal Prediction

### B.1 Key Score Functions

Score functions play a crucial role in conformal prediction. With a fixed underlying classifier, the usefulness of the prediction sets is entirely dependent on the chosen score function. Thresholding (THR) (Sadinle et al., ) is a commonly used one, which is formulated as:

$$ $S_{\textrm{THR}}(\bm{x},y;\hat{\pi})=1-\hat{\pi}_{y}(\bm{x}).$ $$

THR tends to generate efficient prediction sets. However, this score function frequently undercovers hard examples while overcovering trivial ones, resulting in high conditional coverage violations.

To mitigate this issue, a popular alternative is the series of adaptive prediction sets. Adaptive Prediction Sets (APS) (Romano et al., ), the pioneering work in this series, was specifically designed to reduce conditional coverage violations in classification tasks. It is formulated as follows:

$$ $\displaystyle S_{\textrm{APS}}(\bm{x},y,u;\hat{\pi})=\sum_{y_{i}\in\mathcal{Y} }\hat{\pi}_{y_{i}}(\bm{x})\cdot\mathds{1}_{\{\hat{\pi}_{y_{i}}(\bm{x})>\hat{ \pi}_{y}(\bm{x})\}}+u\cdot\hat{\pi}_{y}(\bm{x}),$ $$

where $u$ is an independent random variable following a uniform distribution on $[0,1]$. The prediction set is constructed by adding classes in descending order of probabilities, starting from the most likely to the least, until the cumulative probability exceeds $1-\alpha$.

However, APS always results in large prediction sets since tail classes with low probabilities are easily included. To alleviate this limitation, Regularized Adaptive Prediction Sets (RAPS) (Angelopoulos et al., ) penalizes classes based on their rank information with a predefined threshold, thereby promoting the formation of efficient prediction sets. RAPS is formulated as follows:

$$ $\displaystyle S_{\textrm{RAPS}}(\bm{x},y,u;\hat{\pi})=S_{\textrm{APS}}(\bm{x}, y,u;\hat{\pi})+\lambda\cdot(o(y,\hat{\pi}(\bm{x}))-k_{reg})^{+},$ $$

where $o(y,\hat{\pi}(\bm{x}))$ is the label ranking of $y$, $\lambda$ and $k_{reg}$ are hyperparameters, and $(z)^{+}$ denotes the positive part of $z$. This regularization encourages more efficient prediction sets. In this work, we evaluate the performance of C-Adapter on THR, APS, and RAPS. For RAPS, we consistently set $k_{reg}$ to 1 and $\lambda$ to 0.001 across all experiments.

### B.2 Conformal Training

The core concept of ConfTr (Stutz et al., ) is to render the entire conformal prediction pipeline differentiable, thereby enabling direct optimization of the average prediction set size during classifier training. This process involves simulating both the calibration and prediction phases in each mini-batch. Specifically, mini-batch $\mathcal{B}$ is divided into a calibration subset $\mathcal{B}_{\text{cal}}$ and a test subset $\mathcal{B}_{\text{test}}$. The subset $\mathcal{B}_{\text{cal}}$ is used to compute the soft threshold $\tau^{\text{soft}}$, while $\mathcal{B}_{\text{test}}$ is used to obtain the soft prediction sets $\mathcal{C}_{\text{soft}}(\bm{x};\tau^{\text{soft}},\hat{\pi})$ for loss calculations. The detailed operations are as follows:

Soft threshold: During the calibration step, a non-differentiable quantile operation is required to determine the threshold $\tau$. To make this operation differentiable, smooth sorting techniques (Blondel et al., ; Cuturi et al., ; Williamson, ) are employed, as follows:

$$ $\tau^{\text{soft}}_{\alpha}=\mathcal{Q}_{\text{soft}}(\{S(\bm{x},y;\hat{\pi}) \}_{(\bm{x},y)\in\mathcal{B}_{\text{cal}}},1-\alpha),$ (9) $$

where $\mathcal{Q}_{\text{soft}}$ denotes the differentiable quantile operator, derived using smooth sorting techniques.

Soft conformal prediction set: The calculation of conformal prediction sets involves a non-differentiable hard-thresholding operation, as shown in Equation (2). To address this limitation, ConfTr employs the sigmoid function as a differentiable surrogate for the thresholding:

$$ $\mathcal{C}_{\text{soft}}(\bm{x};\tau^{\text{soft}}_{\alpha},\hat{\pi})=\left \{\sigma\left(\frac{\tau^{\text{soft}}_{\alpha}-S(\bm{x},y;\hat{\pi})}{T} \right)|y\in\mathcal{Y}\right\},$ (10) $$

where $\sigma$ denotes the sigmoid function and $T$ is a hyperparameter. The $k$-th term in this set represents a soft assignment of class $k$, indicating the probability of class $k$ being included in the prediction set. By taking the limit as $T\rightarrow 0$, this operator becomes

$$ $\lim_{T\rightarrow 0}\sigma\left(\frac{\tau^{\text{soft}}_{\alpha}-S(\bm{x},y; \hat{\pi})}{T}\right)=\begin{cases}1,~{}~{}S(\bm{x},y;\hat{\pi})\leq\tau^{ \text{soft}}_{\alpha},\\ 0,~{}~{}S(\bm{x},y;\hat{\pi})>\tau^{\text{soft}}_{\alpha}.\end{cases}$ $$

For loss calculation, after $\tau^{\text{soft}}$ is computed using $\mathcal{B}_{\text{cal}}$ as specified in Equation (9), Equation (10) is applied to each instance in $\mathcal{B}_{\text{test}}$ to compute the soft prediction sets. The size of each prediction set is approximated by summing the values in the set $\mathcal{C}_{\text{soft}}(\bm{x})$, which is optimized during training. Additionally, a standard classification loss, such as cross-entropy loss, is incorporated to enhance classification accuracy. The total loss function is then formulated as follows:

$$ $\mathcal{L}_{\text{ConfTr}}(f(\bm{x};\bm{\theta}),y,\tau^{\text{soft}}_{\alpha })=\mathcal{L}_{\text{cls}}(f(\bm{x};\bm{\theta}),y)+\lambda\mathcal{L}_{\text {size}}(f(\bm{x};\bm{\theta}),\tau^{\text{soft}}_{\alpha}),$ $$

where $\mathcal{L}_{\text{cls}}$ represents the classification loss, and $\mathcal{L}_{\text{size}}$ refers to the size loss, which approximates the size of the prediction set at a specific error rate (e.g., 0.01). Here, $\lambda$ controls the strength of $\mathcal{L}_{\text{size}}$.

## Appendix C Intra Order-Preserving Functions

###### Definition 1 .

A function $h:\mathbb{R}^{K}\rightarrow\mathbb{R}^{K}$ is considered intra order-preserving if, for any vector $\bm{x}\in\mathbb{R}^{K}$, the relative ordering of the elements in $\bm{x}$ is preserved in $h(\bm{x})$. Formally, $h_{i}(\bm{x})>h_{j}(\bm{x})$ (or $h_{i}(\bm{x})=h_{j}(\bm{x})$) holds if and only if $\bm{x}_{i}>\bm{x}_{j}$ (or $\bm{x}_{i}=\bm{x}_{j}$).

An intra order-preserving function maintains all ties and inequalities among the input elements. A typical example is the softmax operator presented in Equation (1). The following theorem outlines the necessary and sufficient conditions for constructing continuous intra order-preserving functions.

###### Theorem 1 ( Rahimi et al. ( 2020 ) ) .

Let $\mathbb{U}^{K}\subset\{0,1\}^{K\times K}$ denote the set of $K\times K$ permutation matrices, and let $R:\mathbb{R}^{K}\rightarrow\mathbb{U}^{K}$ represent the sorting function. For any vector $\bm{x}\in\mathbb{R}^{K}$, the vector $\bm{r}=R(\bm{x})\bm{x}$ satisfies $\bm{r}_{1}\geq\cdots\geq\bm{r}_{K}$. A continuous function $h:\mathbb{R}^{K}\rightarrow\mathbb{R}^{K}$ is intra order-preserving if and only if it can be written as $h(\bm{x})=R(\bm{x})^{-1}Ut(\bm{x})$, where $U$ is an upper-triangular matrix of ones, and $t:\mathbb{R}^{K}\rightarrow\mathbb{R}^{K}$ is a continuous function that satisfies the following condition: $t_{i}(\bm{x})>0$ (or $t_{i}(\bm{x})=0$) if $\bm{r}_{i}>\bm{r}_{i+1}$ (or $\bm{r}_{i}=\bm{r}_{i+1}$), for all $i<K$. The value of $t_{K}(\bm{x})$ is arbitrary.

This theorem provides a pathway for learning within this function family using backpropagation. To maintain the top-$k$ accuracy of the original network, we implement our conformal adapter as a class of intra order-preserving functions, defined as follows:

$$ $g(\bm{f};\bm{w})=R(\bm{f})^{-1}U\Psi(\bm{f}),$ $$

where the $i$-th term of $\Psi(\bm{f})$ is formulated as:

$$ $\Psi_{i}(\bm{f})=\begin{cases}\sqrt{(\bm{r}_{i}-\bm{r}_{i+1})}\sigma(\varphi_{ i}(\bm{f}))&\text{for }i<K,\\ \varphi_{K}(\bm{f})&\text{for }i=K.\end{cases}$ $$

Here, $\varphi(\bm{f})=\bm{w}\cdot\bm{f}+\bm{w}^{\prime}$, and $\sigma$ denotes the sigmoid function. We denote $\varphi_{i}(\bm{f})$ as the $i$-th term of $\varphi(\bm{f})$.
Since $\Psi(\bm{f})$ is continuous in $\bm{f}$, it is straightforward to verify that this structure satisfies the requirements outlined in Theorem 1. This structure decouples the logit order from the adaptation for conformal prediction, enabling C-Adapter to focus on optimizing efficiency. Notably, to improve convergence and facilitate easier optimization of the structures, we can apply a residual function:

$$ $g(\bm{f};\bm{w})=R(\bm{f})^{-1}U\Psi(\bm{f})+\bm{f}.$ $$

Additionally, rescaling the input $\bm{f}$ to the range (0, 1) (e.g., by using the softmax function) can also benefit optimization.

Figure: (a) Loss
Refer to caption: x18.png

###### Definition 1 .

###### Theorem 1 ( Rahimi et al. ( 2020 ) ) .

## Appendix D Proof for Proposition 1

###### Proof.

Considering $(X,Y)\sim\mathcal{P}_{\mathcal{X}\mathcal{Y}},\hat{X}\sim\mathcal{P}_{\mathcal
{X}},\hat{Y}\sim\textrm{Uniform}(\mathcal{Y})$, and letting $\mu(\hat{\pi}):=\mathbb{P}\left(S(X,Y;\hat{\pi})\geq S(\hat{X},\hat{Y};\hat{
\pi})\right)$, we have

$$ $\displaystyle\mu(\hat{\pi})={}$ $\displaystyle\mathbb{E}_{(X,Y)\sim\mathcal{P}_{\mathcal{X}\mathcal{Y}},\hat{X} \sim\mathcal{P}_{\mathcal{X}},\hat{Y}\sim\textrm{Uniform}(\mathcal{Y})}\left[ \mathds{1}_{\{S(X,Y;\hat{\pi})\geq S(\hat{X},\hat{Y};\hat{\pi})\}}\right]$ $\displaystyle={}$ $\displaystyle\mathbb{E}_{\hat{X}\sim\mathcal{P}_{\mathcal{X}},\hat{Y}\sim \textrm{Uniform}(\mathcal{Y})}\left[\mathbb{E}_{(X,Y)\sim\mathcal{P}_{\mathcal {X}\mathcal{Y}}}\left[\mathds{1}_{\{S(X,Y;\hat{\pi})\geq S(\hat{X},\hat{Y}; \hat{\pi})\}}\right]\right]$ $\displaystyle={}$ $\displaystyle\frac{1}{K}\sum_{\hat{Y}\in\mathcal{Y}}\mathbb{E}_{\hat{X}\sim \mathcal{P}_{\mathcal{X}}}\left[\mathbb{E}_{(X,Y)\sim\mathcal{P}_{\mathcal{X} \mathcal{Y}}}\left[\mathds{1}_{\{S(X,Y;\hat{\pi})\geq S(\hat{X},\hat{Y};\hat{ \pi})\}}\right]\right]$ $\displaystyle={}$ $\displaystyle\frac{1}{K}\mathbb{E}_{\hat{X}\sim\mathcal{P}_{\mathcal{X}}}\left [\mathbb{E}_{(X,Y)\sim\mathcal{P}_{\mathcal{X}\mathcal{Y}}}\left[\sum_{\hat{Y} \in\mathcal{Y}}\mathds{1}_{\{S(X,Y;\hat{\pi})\geq S(\hat{X},\hat{Y};\hat{\pi}) \}}\right]\right]$ $\displaystyle={}$ $\displaystyle\frac{1}{K}\mathbb{E}_{\hat{X}\sim\mathcal{P}_{\mathcal{X}}}\left [\mathbb{E}_{s_{\theta}\sim\mathcal{P}_{{\mathcal{S}_{\theta}}}}\left[\sum_{ \hat{Y}\in\mathcal{Y}}\mathds{1}_{\{s_{\theta}\geq S(\hat{X},\hat{Y};\hat{\pi} )\}}\right]\right].$ $$

Assuming that the CDF of $\mathcal{P}_{\theta}$, denoted as $F_{S_{\theta}}$, is monotonically increasing, we have

$$ $\displaystyle\mathbb{E}_{\hat{X}\sim\mathcal{P}_{\mathcal{X}}}\left[\mathbb{E} _{s_{\theta}\sim\mathcal{P}_{{\mathcal{S}_{\theta}}}}\left[\sum_{\hat{Y}\in \mathcal{Y}}\mathds{1}_{\{s_{\theta}\geq S(\hat{X},\hat{Y};\hat{\pi})\}}\right ]\right]$ $\displaystyle={}$ $\displaystyle\mathbb{E}_{\hat{X}\sim\mathcal{P}_{\mathcal{X}}}\left[\int\sum_{ \hat{Y}\in\mathcal{Y}}\mathds{1}_{\{t\geq S(\hat{X},\hat{Y};\hat{\pi})\}} \mathrm{d}F_{S_{\theta}}(t)\right]$ $\displaystyle(\text{let }t=F_{S_{\theta}}^{-1}(1-\alpha))={}$ $\displaystyle\mathbb{E}_{\hat{X}\sim\mathcal{P}_{\mathcal{X}}}\left[\int_{1}^{ 0}\sum_{\hat{Y}\in\mathcal{Y}}\mathds{1}_{\{F_{S_{\theta}}^{-1}(1-\alpha)\geq S (\hat{X},\hat{Y};\hat{\pi})\}}\ \mathrm{d}(1-\alpha)\right]$ $\displaystyle={}$ $\displaystyle\mathbb{E}_{\hat{X}\sim\mathcal{P}_{\mathcal{X}}}\left[\int_{0}^{ 1}\sum_{\hat{Y}\in\mathcal{Y}}\mathds{1}_{\{F_{S_{\theta}}^{-1}(1-\alpha)\geq S (\hat{X},\hat{Y};\hat{\pi})\}}\mathrm{d}\alpha\right]$ $\displaystyle={}$ $\displaystyle\mathbb{E}_{\hat{X}\sim\mathcal{P}_{\mathcal{X}}}\left[\int_{0}^{ 1}|\mathcal{C}\left(\hat{X};F^{-1}_{S_{\theta}}(1-\alpha),\hat{\pi}\right)|\, \mathrm{d}\alpha\right].$ $$

Thus,
$\mu(\hat{\pi})>\mu(\hat{\pi}^{\prime})$
if and only if

$$ $\mathbb{E}_{X\sim\mathcal{P}_{\mathcal{X}}}\left[\int_{0}^{1}|\mathcal{C}\left (X;F^{-1}_{S_{\theta}}(1-\alpha),\hat{\pi}\right)|\,\mathrm{d}\alpha\right]> \mathbb{E}_{X\sim\mathcal{P}_{\mathcal{X}}}\left[\int_{0}^{1}|\mathcal{C}\left (X;F^{-1}_{S_{\theta}}(1-\alpha),\hat{\pi}^{\prime}\right)|\,\mathrm{d}\alpha \right].$ $$

∎

###### Proof.

## Appendix E Convergence Analysis

Our method is computationally efficient as it updates only a limited number of parameters and can converge rapidly. To demonstrate this, we conduct an experiment on ImageNet using DN121, visualizing the changes in loss and efficiency over iterations. We tune C-Adapter with Adam, using a learning rate of 0.1, a batch size of 256, and a weight decay of 0.0001. We apply THR during adapter tuning. We also use THR for evaluation and set the error rate to 0.5.
As shown in Figure 10, our method converges rapidly within 200 iterations, with the efficiency of the conformal predictor improving quickly and approaching nearly optimal performance in just 50 iterations. This convergence analysis highlights the computational efficiency of our proposed approach.

## Appendix F Evaluation Metrics

Size refers to the average number of labels in the prediction sets, while Coverage indicates the percentage of test samples where the prediction sets contain the ground-truth labels:

$$ Size $\displaystyle=\frac{1}{|\mathcal{D}_{\text{test}}|}\sum_{(\bm{x},y)\in\mathcal {D}_{\text{test}}}|\mathcal{C}(\bm{x})|,$ Coverage $\displaystyle=\frac{1}{|\mathcal{D}_{\text{test}}|}\sum_{(\bm{x},y)\in\mathcal {D}_{\text{test}}}\mathds{1}_{\{y\in\mathcal{C}(\bm{x})\}}.$ $$

CovGap (Ding et al., ) and SSCV (Angelopoulos et al., ) are defined as follows:

$$ CovGap $\displaystyle=100\times\frac{1}{|\mathcal{Y}|}\sum_{y\in\mathcal{Y}}|\hat{c}_{ y}-(1-\alpha)|,$ SSCV $\displaystyle=100\times\sup_{j}\left|{(1-\alpha)-\frac{\left|\left\{i:y_{i}\in \mathcal{C}\left(\bm{x}_{i}\right),i\in\mathcal{J}_{j}\right\}\right|}{\left| \mathcal{J}_{j}\right|}}\right|.$ $$

For CovGap, $\hat{c}_{y}$ denotes the coverage rate for class $y$ and quantifies the deviation of class-conditional coverage from the desired level of $1-\alpha$. For SSCV, $\mathcal{J}$ represents the partitioned sets, with the prediction sets categorized by their sizes. This metric evaluates the maximum deviation of the observed coverage rate from $1-\alpha$ across different set size categories. In our experiment, the partitioning of set sizes for SSCV is defined as {0-1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11-100, 101-1000}.

## Appendix G Detailed Experimental Setup

### G.1 Detailed Setup for Figure 2

For CIFAR100, ResNet18 is trained using the full training set of 50,000 samples. The test set of 10,000 samples is divided into a calibration subset of 5,000 samples and a test subset of 5,000 samples. The calibration subset is further split into a validation set and a calibration set in an 20:80 ratio for parameter tuning. The network is trained for 200 epochs using SGD with a momentum of 0.9, a weight decay of 0.0005, and a batch size of 256. The initial learning rate is set to 0.1 and is reduced by a factor of 5 at 60, 120, and 160 epochs. The hyperparameters $T$ and $\lambda$ of ConfTr are tuned from the ranges {0.01, 0.1, 0.5, 1} and {0.005, 0.01, 0.05, 0.1, 0.2}, respectively.

For ImageNet, instead of training from scratch, we fine-tune only the fully connected layer of a pre-trained ResNet18 using the training set. The test set is divided into a calibration subset of 30,000 samples and a test subset of 20,000 samples, with the calibration subset further split into a validation set and a calibration set in a 20:80 ratio for parameter tuning. The fully connected layer is tuned for 240 iterations using Adam with a batch size of 256 and a learning rate of 0.001. A larger learning rate significantly decreases classification accuracy, thereby reducing efficiency. The hyperparameters $T$ and $\lambda$ for ConfTr are selected from the ranges {0.01, 0.1, 0.5, 1} and {0.001, 0.005, 0.01, 0.05, 0.1}, respectively; a larger $\lambda$ also leads to a substantial decline in accuracy.

For evaluation, we use THR, APS, and RAPS, with the error rate $\alpha$ set to 0.1. During model training, we utilize the THRLP score function (Stutz et al., ), setting the error rate $\alpha$ to 0.01. We also present the top-2, top-3, and top-5 accuracy of ConfTr on CIFAR100 and ImageNet in Figure 11.

Figure: (a) CIFAR100
Refer to caption: x20.png

### G.2 Detailed Setup for Figure 7

Retraining: Classifiers are trained using the complete training set of 50,000 samples, with the objective defined in Equation (7). The network is trained for 200 epochs using SGD with a momentum of 0.9, a weight decay of 0.0005, and a batch size of 256. The initial learning rate is set to 0.1 and reduced by a factor of 5 at epochs 60, 120, and 160. The parameter $T$ is tuned within the range {0.001, 0.01, 0.1, 1} using the validation set. We utilize THR for classifier training.

Fine-tuning: Classifiers are trained using the same training set of 50,000 samples with cross-entropy loss. The network is trained for 200 epochs using SGD, with a momentum of 0.9, a weight decay of 0.0005, and a batch size of 256. The initial learning rate is set to 0.1, reduced by a factor of 5 at epochs 60, 120, and 160. During fine-tuning, only the fully connected layer of the pre-trained classifier is updated, training for 240 iterations with Adam, a batch size of 256, and a learning rate of 0.001. Notably, a larger learning rate results in a significant decrease in classification accuracy. The parameter $T$ is tuned from the range {0.001, 0.01, 0.1, 1} using the validation set.

## Appendix H Additional Experimental Results

**Table 5: Performance of C-Adapter on common benchmarks. $\downarrow$ indicates that a smaller value is better. Results in bold indicate superior performance. C-Adapter is tuned using APS.**
|  |  | w/o C-Adapter $\backslash$ w/ C-Adapter |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Score | Model | ImageNet | CIFAR-100 |  |  |  |  |  |  |
| $\alpha=0.05$ | $\alpha=0.1$ | $\alpha=0.05$ | $\alpha=0.1$ |  |  |  |  |  |  |
|  |  | Coverage | Size ($\downarrow$) | Coverage | Size ($\downarrow$) | Coverage | Size ($\downarrow$) | Coverage | Size ($\downarrow$) |
| THR | RN101 | 0.95 $\backslash$ 0.95 | 4.03 $\backslash$ 3.66 | 0.90 $\backslash$ 0.90 | 1.91 $\backslash$ 1.86 | 0.95 $\backslash$ 0.95 | 3.64$\backslash$ 3.11 | 0.90 $\backslash$ 0.90 | 1.87 $\backslash$ 1.80 |
| DN121 | 0.95 $\backslash$ 0.95 | 5.66 $\backslash$ 5.39 | 0.90 $\backslash$ 0.90 | 2.42 $\backslash$ 2.40 | 0.95 $\backslash$ 0.95 | 3.27 $\backslash$ 3.03 | 0.90 $\backslash$ 0.90 | 1.72 $\backslash$ 1.69 |  |
| DN161 | 0.95 $\backslash$ 0.95 | 4.03 $\backslash$ 3.88 | 0.90 $\backslash$ 0.90 | 1.89 $\backslash$ 1.86 | 0.95 $\backslash$ 0.95 | 2.91 $\backslash$ 2.71 | 0.90 $\backslash$ 0.90 | 1.72 $\backslash$ 1.71 |  |
| RNX50 | 0.95 $\backslash$ 0.95 | 4.26 $\backslash$ 3.93 | 0.90 $\backslash$ 0.90 | 1.87 $\backslash$ 1.84 | 0.95 $\backslash$ 0.95 | 3.41 $\backslash$ 3.16 | 0.90 $\backslash$ 0.90 | 1.78 $\backslash$ 1.77 |  |
| CLIP | 0.95 $\backslash$ 0.95 | 6.88 $\backslash$ 6.90 | 0.90 $\backslash$ 0.90 | 3.33 $\backslash$ 3.28 | 0.95 $\backslash$ 0.95 | 9.71$\backslash$ 9.67 | 0.90 $\backslash$ 0.90 | 4.78 $\backslash$ 4.69 |  |
| Average | 0.95 $\backslash$ 0.95 | 4.97 $\backslash$ 4.75 | 0.90 $\backslash$ 0.90 | 2.29 $\backslash$ 2.25 | 0.95 $\backslash$ 0.95 | 4.59 $\backslash$ 4.34 | 0.90 $\backslash$ 0.90 | 2.37 $\backslash$ 2.33 |  |
| APS | RN101 | 0.95 $\backslash$ 0.95 | 14.73 $\backslash$ 3.82 | 0.90 $\backslash$ 0.90 | 7.23 $\backslash$ 2.07 | 0.95 $\backslash$ 0.95 | 7.60 $\backslash$ 3.16 | 0.90 $\backslash$ 0.90 | 3.95 $\backslash$ 1.80 |
| DN121 | 0.95 $\backslash$ 0.95 | 20.00 $\backslash$ 5.64 | 0.90 $\backslash$ 0.90 | 9.21 $\backslash$ 2.74 | 0.95 $\backslash$ 0.95 | 10.20 $\backslash$ 4.12 | 0.90 $\backslash$ 0.90 | 4.44 $\backslash$ 2.35 |  |
| DN161 | 0.95 $\backslash$ 0.95 | 16.43 $\backslash$ 4.13 | 0.90 $\backslash$ 0.90 | 6.82 $\backslash$ 2.05 | 0.95 $\backslash$ 0.95 | 9.90 $\backslash$ 3.14 | 0.90 $\backslash$ 0.90 | 5.42 $\backslash$ 1.87 |  |
| RNX50 | 0.95 $\backslash$ 0.95 | 21.54 $\backslash$ 4.10 | 0.90 $\backslash$ 0.90 | 8.92 $\backslash$ 2.07 | 0.95 $\backslash$ 0.95 | 9.95 $\backslash$ 3.19 | 0.90 $\backslash$ 0.90 | 5.14 $\backslash$ 1.90 |  |
| CLIP | 0.95 $\backslash$ 0.95 | 26.35 $\backslash$ 7.42 | 0.90 $\backslash$ 0.90 | 13.24 $\backslash$ 3.43 | 0.95 $\backslash$ 0.95 | 16.13 $\backslash$ 12.94 | 0.90 $\backslash$ 0.90 | 10.18 $\backslash$ 8.10 |  |
| Average | 0.95 $\backslash$ 0.95 | 19.81 $\backslash$ 5.04 | 0.90 $\backslash$ 0.90 | 9.08 $\backslash$ 2.47 | 0.95 $\backslash$ 0.95 | 10.76 $\backslash$ 5.31 | 0.90 $\backslash$ 0.90 | 6.01 $\backslash$ 3.20 |  |
| RAPS | RN101 | 0.95 $\backslash$ 0.95 | 7.13 $\backslash$ 4.43 | 0.90 $\backslash$ 0.90 | 4.60 $\backslash$ 2.01 | 0.95 $\backslash$ 0.95 | 5.16 $\backslash$ 4.71 | 0.90 $\backslash$ 0.90 | 3.25 $\backslash$ 1.81 |
| DN121 | 0.95 $\backslash$ 0.95 | 10.28 $\backslash$ 7.38 | 0.90 $\backslash$ 0.90 | 6.57 $\backslash$ 2.66 | 0.95 $\backslash$ 0.95 | 7.19$\backslash$ 4.00 | 0.90 $\backslash$ 0.90 | 4.50 $\backslash$ 1.83 |  |
| DN161 | 0.95 $\backslash$ 0.95 | 7.31 $\backslash$ 5.01 | 0.90 $\backslash$ 0.90 | 4.63 $\backslash$ 2.00 | 0.95 $\backslash$ 0.95 | 7.10 $\backslash$ 3.22 | 0.90 $\backslash$ 0.90 | 4.59 $\backslash$ 1.81 |  |
| RNX50 | 0.95 $\backslash$ 0.95 | 7.88 $\backslash$ 5.05 | 0.90 $\backslash$ 0.90 | 5.20 $\backslash$ 2.01 | 0.95 $\backslash$ 0.95 | 7.20 $\backslash$ 3.64 | 0.90 $\backslash$ 0.90 | 4.47 $\backslash$ 1.79 |  |
| CLIP | 0.95 $\backslash$ 0.95 | 15.14 $\backslash$ 8.74 | 0.90 $\backslash$ 0.90 | 9.25 $\backslash$ 3.41 | 0.95 $\backslash$ 0.95 | 14.52 $\backslash$ 13.61 | 0.90 $\backslash$ 0.90 | 9.41 $\backslash$ 8.92 |  |
| Average | 0.95 $\backslash$ 0.95 | 9.55 $\backslash$ 6.12 | 0.90 $\backslash$ 0.90 | 6.05 $\backslash$ 2.42 | 0.95 $\backslash$ 0.95 | 8.24  $\backslash$ 5.84 | 0.90 $\backslash$ 0.90 | 5.24 $\backslash$ 3.23 |  |

#### Results when tuning C-Adapter using APS

We report the detailed results of Coverage and Size when C-Adapter is tuned using APS. Empirical results in Table 5 demonstrate that C-Adapter consistently enhances the efficiency of conformal predictors, regardless of the model architectures and pre-training strategies, highlighting the flexibility of our approach.

## Peer Reviews

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: 1. The authors are encouraged to include some related methods in the comparisons to provide a more comprehensive evaluation.
2. The motivation and insight for employing the Adapter module should be further emphasized to clarify its significance in the proposed approach.
soundness: 3
strengths: 1. The results demonstrate that the proposed method significantly reduces prediction set sizes while maintaining accuracy.
2. C-Adapter is versatile, working effectively with a range of classifiers and showing strong compatibility with black-box models.
3. Empirical results indicate that C-Adapter performs consistently across various datasets, models, and evaluation metrics.
4. Minimal hyperparameter tuning and high computational efficiency make C-Adapter highly practical for deployment.
summary: The paper introduces C-Adapter, an adapter-based tuning method designed to improve the efficiency of conformal predictors without compromising classification accuracy. This approach is highly relevant to uncertainty quantification, where conformal prediction frameworks generate prediction sets that, with a specified coverage rate, are likely to include the true class. C-Adapter seeks to optimize prediction efficiency while preserving or enhancing model accuracy, which holds significant potential for high-stakes applications such as medical diagnostics.
weaknesses: 1. The primary concern with this paper is the lack of comparison with related methods. The authors tested the proposed C-Adapter across various benchmarks (Table 1), loss functions (Table 2), values of α (Table 3), and distribution shifts (Table 4), but did not include comparisons with other approaches in conformal prediction.
2. While the use of adapters for conformal prediction is a novel application, the concept of adapters itself is well-established. The insight for choosing adapters over other modules, such as LoRA, is not sufficiently discussed and would benefit from further elaboration.

---

confidence: 3
contribution: 4
initial_rating: 8
presentation: 3
questions: While C-Adapter is shown to outperform Conformal Training (ConfTr), what are the specific conditions or datasets where ConfTr might still be preferable or complementary to C-Adapter?

The results indicate that C-Adapter improves conditional coverage. What are the underlying mechanisms that enable this improvement and how it compares to methods specifically tailored for conditional coverage?

The evaluation focuses on standard score functions (THR, APS, RAPS). How would C-Adapter perform with more specialized or non-standard score functions used in specific domains?
soundness: 4
strengths: Originality: The paper is original in proposing C-Adapter, an adapter-based method for improving the efficiency of conformal predictors while maintaining accuracy. Unlike traditional methods such as Conformal Training, which can compromise classifier performance, C-Adapter innovatively integrates an adapter layer that preserves label ranking to maintain top-k accuracy. The introduction of intra order-preserving functions and a new loss function tailored for conformal prediction is novel and adds depth to the methodology.

Quality: The quality of the work is strong, supported by both theoretical justifications and comprehensive empirical results. The authors provide a solid mathematical foundation for their approach, including proofs and detailed discussions on the properties of the proposed method. The experiments are well-designed and conducted across various benchmarks, such as CIFAR-100 and ImageNet, using multiple classifiers. This extensive evaluation highlights the robustness and effectiveness of C-Adapter. The paper also compares its method with existing solutions like Conformal Training and demonstrates clear improvements.

Clarity: The paper is generally clear, with a well-organized structure that guides the reader through the problem, methodology, and experimental results. The introduction and related work sections set the stage effectively, and the results are presented with informative figures and tables. However, the clarity could be further enhanced by simplifying some complex mathematical sections and providing more intuitive explanations. This would make the paper more accessible to readers who are not specialists in conformal prediction or the specific mathematical frameworks used.

Significance: The paper's contribution is significant, particularly for the field of uncertainty quantification in machine learning. C-Adapter presents a practical and adaptable solution that can be applied to a variety of classifiers and settings, including black-box models. Its ability to maintain classification accuracy while reducing prediction set sizes has practical implications for high-stakes applications such as medical diagnostics and financial forecasting, where efficient and reliable uncertainty estimates are crucial. The method's flexibility and minimal computational overhead further enhance its significance, positioning it as a valuable tool for both research and practical implementations.
summary: The paper "C-Adapter: Adapting Deep Classifiers for Efficient Conformal Prediction Sets" introduces C-Adapter, a method that improves the efficiency of conformal predictors while preserving classification accuracy. By adding an adapter layer to trained classifiers, C-Adapter maintains top-k accuracy through label ranking preservation. It optimizes a unique loss function to enhance non-conformity score separation between correct and incorrect predictions, resulting in more efficient prediction sets. Tested on CIFAR-100 and ImageNet, C-Adapter significantly reduces prediction set sizes and outperforms existing methods like Conformal Training, adapting well across various classifiers and scoring functions with minimal computational cost.
weaknesses: While the paper provides strong theoretical support, certain sections, particularly those involving the mathematical underpinnings of intra order-preserving functions, may be difficult for readers unfamiliar with this concept. To improve accessibility, the authors could include a simplified overview or illustrative examples to help readers intuitively grasp the key ideas without needing extensive background knowledge. This would broaden the paper’s reach and make it more appealing to a wider audience.

While the paper briefly addresses distribution shifts using ImageNet-V2, a more detailed exploration or comparison with other methods in this context would strengthen the claim of C-Adapter’s robustness. Further experiments with synthetic or real-world data shifts could provide deeper insights into its performance under more varied conditions.

While the paper claims C-Adapter is insensitive to hyperparameters, the provided analysis on the parameter T is limited. A more comprehensive exploration of hyperparameter sensitivity, including the impact of different tuning strategies and settings, would help verify this claim. Showing how C-Adapter behaves under a variety of hyperparameter configurations can reassure practitioners of its reliability in different scenarios.

---

confidence: 3
contribution: 2
initial_rating: 6
presentation: 3
questions: I tried to cover most of my concerns and questions in the *Weaknesses* section. I kindly request the authors to review that section.
soundness: 3
strengths: **S1.** The paper is very well written and presented. 

**S2.** The methodological development is well written, with a comprehensive background and related works. 

**S3.** I found the experimentation well-motivated, covering important ablation studies, including alpha, training strategy, adaptation strategy, and parameter $T$. The authors achieved considerable improvement across different feature extractor backbones. Besides, the experiments on alpha under THR and APS demonstrate the robustness of their C-Adapter strategy.
summary: The authors proposed an adapter-based tuning strategy to enhance conformal prediction performance without sacrificing the model's performance. They implemented this adapter as a class of intra-order-preserving functions to maximize the discriminability of non-conformity scores between correctly and randomly matched data-label pairs. This approach achieved high non-conformity scores for incorrect labels, enhancing the efficiency of prediction sets across different coverage rates.
weaknesses: **W1. Methodological Novelty.** I found the contributions made by the authors are somewhat limited. The intra order-preserving function is adapted from (Rahimi et al., 2020). Overall, the complete approach is somewhat like a combination of the existing SOTAs, including the intra order-preserving function, conformal training (Rahimi et al., 2020, Stutz et al., 2021). Other than the theoretical demonstration and an additional learnable layer (the adapter layer), I would suggest the authors to specifically highlight any other methodological contribution. 

**W2. Loss function explanation.** One of the limitations of their work is the quality of the approximation depends heavily on the choice of $T$, which seems to be not affected by the prediction set size that much, according to the experimental findings. What is the rationale behind the insensitiveness of their approach to this parameter $T$? Besides, they approximated the loss function with sigmoid which might not be strictly convex over the entire domain. Hence, this approximation might introduce non-convexities that could impact the convergence and optimization process.

While there are some concerns about the proposed methodology and the extent of its novelty, the current rating reflects the accumulated efforts in problem definition, motivation, presentation, diverse experimentation, and ablation studies. I would strongly suggest to consider the findings and address them in their rebuttal. Good luck.

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: With that being said, I have several concerns and questions for the authors:
1. The main motivation behind his paper is that increasing \lambda decreases the classification accuracy, leading to larger average size of prediction sets. However, this is not the case in Fig 1 (blue lines) presented - CIFAR-100 has a U shape, and ImageNet is almost flat. Please explain why this could be the case and why the relationships could differ between datasets.
2. The authors present a training objective designed to optimize the efficiency of conformal predictors over the entire range of alpha values (0, 1). Please explain why this was done. Practically, maybe only one \alpha value could be used. Does only optimizing for one \alpha value improve performance compared to all?
3. The authors test C-Adapter's performance when trained on ImageNet and then tested on ImageNet-V2. This approach evaluates how well the adapted model generalizes to a different but related dataset, simulating a distribution shift scenario. However, the authors say in page 10 - “Notably, coverage will not be affected under this setting, as the calibration and test sets remain exchangeable”. A distribution shift means that calibration and test sets are non-exchangeable. If this is the case, the claim that C-adapter is robust to distribution shift is not warranted. Please explain. Furthermore, quantification of the distribution shift (if there is) would significantly strengthen this claim.
4. Will the code and data be publicly available?
soundness: 3
strengths: To the best of my knowledge, no other work has introduced an “adapter” for conformal training to improve the efficiency of prediction sets. It combines two interesting works and presents interesting empirical results, albeit in limited settings. And could be interesting to the community. Given the clarifications in the authors’ response, I would be willing to increase the score.

The submission is clear, mostly technically correct, and experimentally rigorous. The main strength of the paper is its empirical findings. Although only applied to limited settings/tasks, the empirical results support their claims. Evaluating the effectiveness of C-Adapter on other tasks (NLP or time series), more SOTA architectures and potentially more impactful applications would provide a more comprehensive understanding of its capabilities and limitations.
summary: The submission presents a method to improve conformal training (Stutz et al 2021) for classification through an adapter-based tuning method. The authors claim significant efficiency improvements. The authors draw attention to an existing with conformal training - specifically that the regularization term in conformal training may deteriorate classifier accuracy, and offer a method to alleviate this. The key idea is to use intra-order-preserving functions (Rahimi et al. 2020) as an extra component applied to a pre-trained model. The authors demonstrate the synthesis of two methods through CIFAR-100 and ImageNet classification tasks.
weaknesses: The main weakness of the submission is that it lacks theoretical insight into the efficacy of the method. When does C-adapter perform better/worse than conformal training? Is this always guaranteed? Furthermore, the submission could use an in-depth theoretical analysis of the robustness of C-Adapter - its robustness to distribution shifts, adversarial examples, and noisy data. Theoretical investigations would significantly strengthen the claims of robustness made based on empirical observations.

Meta-review: The paper introduces the Conformal Adapter (C-Adapter), a new method designed to improve the efficiency of conformal prediction sets while maintaining classification accuracy. This post-processing layer retains label ranking in output logits and optimises conformal prediction efficiency. It outperfroms existing methods like Conformal Training (ConfTr) in efficiency and robustness across various datasets and models.

The paper has several contributions, summarised as follows:
1) Utilises an "intra order-preserving function" to retain classification accuracy and introduces a loss function to enhance data-label pair discriminability.
2) Demonstrates significant improvements in prediction set efficiency across benchmarks such as CIFAR-100, ImageNet, and ImageNet-V2, and is compatible with multiple score functions and model architectures.
3) Requires minimal hyperparameter tuning, is computationally efficient, and easy to integrate with existing models, including black-box models.
4) Provides detailed experiments and analyses, with a strong theoretical foundation connecting the loss function to efficiency optimisation.

Overall, the paper makes a good contribution to conformal prediction literature by offering a practical method for enhancing prediction set efficiency, suggesting future research might explore broader aspects of conformal prediction.

Decision: Accept (Poster)