Revisiting Confidence Calibration for Misclassification Detection in VLMs
Authors: Jincheng Huang
Jie Xu
Xiaoshuang Shi
Ping Hu
Lei Feng
Xiaofeng Zhu
Venue: ICLR 2026 Poster (2026)
Keywords: Vision-language models
Confidence calibration
Misclassification Detection
Primary area: interpretability and explainable AI

Abstract: Confidence calibration has been widely studied to improve the trustworthiness of predictions in vision-language models (VLMs). However, we theoretically reveal that standard confidence calibration inherently _impairs_ the ability to distinguish between correct and incorrect predictions (i.e., Misclassification Detection, MisD), which is crucial for reliable deployment of VLMs in high-risk applications. In this paper, we investigate MisD in VLMs and propose confidence recalibration to enhance MisD. Specifically, we design a new confidence calibration objective to replace the standard one. This modification theoretically achieves higher precision in the MisD task and reduces the mixing of correct and incorrect predictions at every confidence level, thereby overcoming the limitations of standard calibration for MisD. As the calibration objective is not differentiable, we introduce a differentiable surrogate loss to enable better optimization. Moreover, to preserve the predictions and zero-shot ability of the original VLM, we develop a post-hoc framework, which employs a lightweight meta network to predict sample-specific temperature factors, trained with the surrogate loss. Extensive experiments across multiple metrics validate the effectiveness of our approach on MisD.

Published as a conference paper at ICLR 2026
REVISITING CONFIDENCE CALIBRATION FOR MISCLAS- SIFICATION DETECTION IN VLMs
Jincheng Huang' Jie Xu Xiaoshuang Shi'
Ping Hu'
Lei Feng**
Xiaofeng Zhu
Schooo of Computer Science and Engineering, University of Electronic Science and Technology of China
Southeeatt University, 'Hainan University, 'Singapore University of Technology and Design
ABSTRACT
Confidence calibration has been widely studied to improve the trustworthiness of predictions in vision-languee models (VLMs). However, theoretically reveal that standard confidence calibration inherently impairs the ability to distinguish between correct and incorrect predictions (i.e., Miselassification Detection, MisD), which is crucial for reliable deployment of VLMs in high-risk applications. this paper, we investigate MisD in VLMs and propose confidence recalibration enhance MisD. Specifically, we design a new confidence calibration objective to replace the standard one. This modification theoretically achieves higher precision in the MisD task and reduces the mixing of correct and incorrect predictions at every confidence level, thereby overeoming the limitations of standard calibra- tion for MisD. As the calibration objective is not differentiable, we introduce a differentiable surrogate loss to enable better optimization. Moreover, to preserve the predictions and zero-shot ability of the original VLM, develop post-hoc framework, which employs lightweight meta network to predict sample-specific temperature factors, trained with the surrogate loss. Extensive experiments across multiple metrics validate the effectiveness of our approach on MisD
INTRODUCTION
Pretrained Vision-language models (VLMs) (Radford et al. 2021 Zhou et al. 2022b a Khattak et al. 2023), such as CLIP (Radford et al., (021). have demonstrated impressive zero-shot capabilities. Owing to their strong generalization ability and pretrained nature, they have been applied to wide range of downstream tasks, including autonomous driving (Cui et al.) (024), medical diagnosis (Zhao al. 2023), and 3D scene understanding (Chen al. 2023). While these models improve flexibility and accuracy, ensuring their reliability remains essential, which is crucial for real-word world deployment and safety-critical applications. Consequently, confidence calibration (Guo et al.) (0077)
which adjusts model confidence to better match true correctness, is therefore an important component developing reliable VLM--bsed systems.
Confidence calibration aims to align the model's predicted confidence with the true likelihood of
correctness. A classical calibration method is Temperature Scaling (Guo et al. 2017). which adjusts the sharpness of the output probabilities using temperature coefficient to better align confidence with empirical accuracy. However, a single global temperature overlooks the instance wwie variation in confidence miscalibration. Therefore, many instance-wise calibration methods have been proposed (Huang et al. 2025] Krishnan & Tickoo 2020). In VLMs, confidence calibration becomes more challenging due modality-speciff factors the process of fine-tuning. To address these issues, several works (Wang et al.) (2024) Lv et al. 2025) show that distances in the text embedding space play crucial role in calibration errors, and they incorporate this text-modality signal as an additional temperature adjustment. In contrast, to mitigate miscalibration introduced by fine-tuning, DOR (Wang et al., 2025) uses large vocabulary set to preserve the semantic structure of the pretrained CLIP, thereby reducing the shift in text features caused by prompt tuning.
Although previous methods can achieve great calibration for VLMs. many high-risk tasks rely more on the model's s ability to rank predictions by correctness than confidence calibration performance.
Conresponding authors

---

Published as a conference paper at ICLR 2026
Relabilty curce Precision
Correct predicted sample
Incorrect predicted sample
Accuracy level
Precision (incorrect
High calibrated model
Low calibrated model
AcE
1.0
Accuracy
0.33
0.66
Confidence
0.0
0.5
1.0
Confidence
Unclearly distinguish between @ and
Clearly distinguish between and @
Confidance
(b)
Figure I: (a) Comparison between calibrated model and one with improved MisD performance. Left: a calibrated model aligns predicted confidence with accuracy. Right: a model with higher MisD performance that better ranks correct predictions above incorrect ones. (b) Illustration of the relationship between the reliability curve and the MisD, Given reliability curve and a confidence threshold r, samples with confidence above r regarded correct predictions. Their precision can be derived from the area under the curve within that interval. Conversely, samples with confidence below r are treated as incorrect predictions, and their precision can be derived from the area above the curve.
Concretely, in the misclassification detection task, it is more desirable for the VLMSs to consistently assign lower confidence scores to misclassified samples than to correctly classified ones, rather than merely aligning predicted probabilities with accuracy, which is illustrated in Figure Iaa.
In this work, we first analyze the reliability diagram of confidence calibration for MisD, and find that the region under (above) the reliability curve is related to the precision of detecting correct (incorrect) prediction, as shown Figure [b). Motivated by this insight, we reveal that standard calibration inherently limits the upper bound of this precision by analyzing its reliability curve. To remedy this limitation, we introduce new target curve. It increases the area under the curve in the high- confidence region and enlarges the area above the curve in the low-confidence region, as illustrated Figure This design theoretically improves precision and reduces the mixing between correctly and incorrectly classified samples, thereby encouraging better separation between them. Although the proposed reliability curve is ideal for MisD, it cannot be directly expressed as a differentiable loss function. To bridge this gap, design surrogate loss that faithfully captures its effect, making the objective practically optimizable. In practice, to avoid interfering with the underlying VLM predictions, adopt post-hoc framework. Specifically, that predicts the temperature scaling factor to each sample, conditioned on the VLM logits, image emheddings, and predicted text embeddings. This meta network trained using our surrogate loss. Compared to previous works, our contributions are listed as follows:
introduce lightweight meta network
We make the first attempt to address MisD in VLMs from a calibration perspective. By analyzing the reliability diagram for MisD, we uncover key limitations of standard calibration and introduce new perspective to systematically analyze MisD and guide its improvement.
We propose a new calibration objective tailored to improving MisD performance. The theoretical analysis guarantees that the proposed objective can improve the precision in the MisD task and reduce mixing correct and incorrect predictions on every confidence level.
To ensure that the proposed loss does not interfere with the pretrained capabilities of VLMs, we develop post-hoc calibration framework that learns individualized temperature coefficients for each sample. Empirical results show that the proposed method consistently improves MisD over the recent uncertainty estimation methods across diverse settings.
RELATED WORK
This section briefly reviews the topics related to this work, including confidence calibration and misclassification detection, and the recent works in VLMs.
2

---

Published as a conference paper at ICLR 2026
2.1 CONFIDENCE CALIBRATION
Trustworthy machine learning Yang et al., 2025c b Fang et al. 2025 Zhuo et al. 2025 Yang et al. 2025a Fang et al., 2026) encompasses various approaches, among which confidence calibration is a key component for ensuring reliable predictive confidence. The calibration methods can be divided into two categories: training-timm time calibration and post-hoc calibration. For the training- taining-time calibration, a notable example is focal loss (Mukhoti et al. 2020), with subsequent works such as adaptive focal loss (Ghosh et al. 2022) modifying hyperparameters for different sample groups based on prior training knowledge, For the post-hoc methods, the most commonly used method is temperature scaling (TS) (Guo et al. 2017), which proposes to use a hyperparameter (i.e., temperature coefficient) as the denominator of logits, making the confidence adjustable. However, TS is not flexible enough, since TS uses a unified temperature coefficient for every sample. Therefore, many subsequent methods (Wang et al., 2021] Xiong et al. 2023 Yang et al. 2023a b) aim to improve TS by applying adaptive temperature parameters, treating samples differently for a more effective maximum-entropy regularizer.
In VLMs, calibration becomes more challenging due to coadalitt-speccfffe factors or the process of fine-tuning (Wang et al. 2024 2025). Consequently, DAC (Wang et al. (2024) utilizes the proximity of the text modality to adjust confidence accordingly. DOR (Wang et al. 2025) mitigating miscalibration
introduced during fine-tuning by introducing an extra-large vocabulary set. Although many calibration methods are proposed, the fundamental goal of calibration is to enable a model to distinguish between correct predictions and incorrect predictions. However, this work, we reveal that even perfectly calibrated model remains fundamentally limited, and thus may still fail to identify the misclassified samples or correctly predicted samples that bring the potential risk in many scenarios.
2.2 MISCLASSIFICATION DETECTION
Misclassification detection (Hendrycks & Gimpel 2016), which is also called failure prediction, and the goal is to detect incorrect predictions from correct predictions. It is crucial for machine learning models deployed in high-risk scenarios. Calibration methods (Zhang et al.) 2023) are often viewed as an effective strategy for misclassification detection, as aligning predicted confidence with the true likelihood of correctness allows the model to distinguish between reliable and unreliable predictions. However, empirical studies (Zhu et al. 2022) have shown that commonly used calibration methods only bring limited improvement for misclassification detection. Yet, no theoretical analysis has been conducted to uncover the underlying reasons, and consequently, the proposed method of this work has not been optimized from a calibration perspective. Therefore, to the best of our knowledge, no prior work has systematically investigated misclassification detection from the calibration perspective. Instead, existing approaches typically focus on confidence regression (Corbière et al., (019), exposing outlier samples (Zhu et al. 2023), flat minima (Zhu et al. 2022). among other techniques.
In VLMs, a few recent efforts have been devoted to misclassification detection. For instance, FSMisD (Zeng et al. 2025) adopts prompt-based strategy, which, however, overlooks the use of confidence information, limiting flexibility and making it difficult to integrate with existing prompt- tuning models. ViLU (Lafon et al. 2025). the other hand, formulates uncertainty modeling as a binary classification problem, which essentially ignores the role of confidence in distinguishing low-confidence correct predictions from high-confidence misclassifications and lacks theoretical guarantees. While these approaches provide useful heuristics, they mainly rely on task-specific designs empirical observations, resulting in limited ability to capture the fundamental role of confidence in misclassification detection.
3 PRELIMINARIES
Contrastive Language-Iman Pretraining (CLIP). CLIP is a powerful vision- lision-languuge model that aligns image and text representations in shared embedding space through contrastive learning (Radford et al. 202T). Owing to its contrastive training on large-scale image-text pairs, CLIP demon- strates strong sero-shot generalization and is readily deployable in various downstream scenarios. Let & x - Rd and v :t + Rd denote the image and text encoders of CLIP, respectively. Given an image instance and text label c, the output logit of CL.IP can be formulated as:
Z0,c Telip sim(E(x.), ((.o)))((((())))))) (t.)),
(1)
3

---

Published as a conference paper at ICLR 2026
where sim(...) denotes cosine similarity, te is hand-crafted prompt, typically set to "a photo of a {class}" and Telip is a fixed constant, usually set to 100. In multi-class classification, let C' = {co, C1, .I...., ) denote the set of candidate classes. The predicted label corresponds to the class with the highest predicted probability: which can be formally expressed as follows:
EE. e*),
ÿ = arg max(e"*
(2)
aEC
where I is the predicted class and the associated probability (i.e,, s= max(e* e²))) is referred to as the confidence of the prediction.
Confidence Calibration. Confidence calibration particularly important in high-risk and open- world scenarios (Guo et al.] 2017). It refers to alignment between model's predicted confidence and its actual accuracy, making the predicted confidence more trustworthy. For example, if the average confidence is 0.8, then approximately 80% of the predicted examples should be correct. Formally, the objective of confidence calibration can be defined as follows:
P yls p) = p, Vp E [0, 1],
where y the ground-truth label. The performance of confidence calibration can be evaluated using a reliability diagram, where the vertical axis represents accuracy and the horizontal axis represents confidence, as shown Figure [[)). Points closer the diagonal indicate better calibration. The reliability curve is obtained by smoothly connecting these points, with the diagonal line representing the perfect-calibration curve (i.e. f(x)) = a). Confidence calibration aims make the model's reliability curve as close as possible to the perfect calibration curve.
Misclassification detection (MisD). Misclassification detection is a critical safeguard for deploying models in real-world applications (Hendrycks & Gimpel, 2016), aiming to distinguish incorrect predictions from correct predictions based on confidence ranking. Formally, given a confidence threshold E [0, 1], predictions with confidence above are detected as correct predictions, while those with confidence below r are detected as incorrect predictions.
4 METHOD
The overall research framework can be divided into three parts: We first analyze the upper bound of calibration for MisD, demonstrating its inherent limitation. To overcome this limitation, we redesign the calibration reliability curve tailored for MisD. In addition, we establish several favorable properties of the proposed curve, highlighting its theoretical soundness and practical advantages in improving MisD. As the reliability curve not differentiable, we design a surrogate loss (i.e,, LSUR) to realize its optimization. Finally, we present a post-hoc and lightweight implementation, which adjusts confidence without modifying the VLMs' parameters.
Overview.
4.1 REVISITING CALIBRATION WITH MIsD
The reliability diagram is key tool for evaluating the calibration of probabilistic classifiers, where the diagonal (ace(s) = s) denotes perfect calibration. Thus, calibration quality is assessed by how closely the reliability curve aligns with this line. So far, it has been used exclusively for this purpose. In this work, we find that the reliability diagram also encodes information relevant to MisD, in particular that the regions under and above the reliability curve relate to the precision of detecting correct and incorrect predictions. This insight reveals direct link between the reliability diagram and MisD.
Specifically, we first analyze the meaning of the region under the reliability curve. Since the vertical axis of reliability diagram denotes accuracy, each curve value reflects the proportion of correct predictions among all samples at that confidence. The enclosed region naturally accumulates these proportions across the confidence spectrum. Building on this view, we formalize the following relationship. The formal proof listed in Appendix I
Lemma 4.1. Given confidence interval [a, b), let w(s) denote the density of the sample at confidence level s. Then, the precision of correct predictions within [a, b) can be derived from region under the reliability curve f(s) over [a, b), i... Prec* = (S" (s))s Sh w(s)ds. Similarly, the precision of incorrect predictions over (a, b) can be derived from the region above the curve via Prec ( (1))(1 f(8))ds)/ So w(s)ds.
4

---

Published as a conference paper at ICLR 2026
Lemma 4.1 establishes a strong connection between the region under (above) the reliability curve and the task of detecting correct (incorrect) predictions. Since MisD can be quantified by the joint precision of detecting high-confidence predictions (i.e., s 2 r) as correct and low-confidence predictions (i.e, 8 < r) as incorrect, it can be characterized by the region below the reliability curve in the high-confidence interval and the region above it in the low-confidence interval.
Based on the above observations, we can examine the effectiveness of confidence calibration for MisD. Specifically, we analyze the precision of the perfect calibration curve detecting correct and incorrect predictions, as formalized in the following theorem (the proof is listed in Appendix A.33:
Theorem 4.2. Let [0, 1] be confidence threshold. Under perfect calibration, the precision for the correct prediction detection and incorrect prediction detection tasks Prec = =Eesw(eleelraanl and Prec
E. (= [1 s), respectively.
Theorem 4.2 demonstrates that, even under the goal of confidence calibration (i.e., perfect calibration), the precision of correct prediction detection equals the conditional expectation of the confidence in [r. 1]. In practice, this expectation is strictly less than unless all samples are concentrated at confidence 1, which rarely oceurs. Similarly, the precision of incorrect prediction detection corresponds to the conditional expectation of 1 - in [0. and is strictly less than unless all samples are concentrated at confidence 0. In realistic scenarios, such as with pretrained CLIP models, test samples are often drawn from diverse distributions, causing confidence values to be widely spread across [0. 1] (see Appendix D for empirical validation). Consequently, both the precision of correct and incorrect prediction detection are far below 1. This observation explains why strict calibration alone is insufficient to achieve high MisD performance.
Motivated by Lemma 4.1 and Theorem 4.2 we seek a reliability curve as the calibration objective that explicitly guides optimization toward better MisD. Let f: [o, 1] + [o. 1] denote the reliability curve. Based on the above discussion, we therefore formulate the following MisD-oriented objective:
w(s)[1 - f(x))dr + L (ss)f())dxx
max JEF
where F = (f 00 f(.) 0, f() = f nondecreasing) is function family. This objective maximizes the region above the curve in the low-confidence region [0, 0.5). and the region under the curve in the high-confidence region [0.5, 1], thereby improving the precision of correct and incorrect prediction detection. However, directly maximizing Eq, 4 over J causes overly aggressive probability adjustments and instability. To balance improvement in MisD with smoothness, we impose an additional requirement that f should have a controllable and gradual transition. As a practical and analytically tractable instantiation, we adopt the normalized sigmoid curve:
(2-0.5) o (=..5
V (s)
(2)) 1+c=='
(5)
0.5 (-1-
where À E R+ controls the smoothness of the transition, enabling a trade-off between separation strength and stability. Figure Jillustraas the curve for different values of X, highlighting its flexibility, We summarize the key properties of the proposed curve that are effective for MisD as follows.
Property la: Perfect calibration as special case (A oc).
As x -> oc, (z) converges uniformly to the diagonal (x) = on [0, 1], which corresponds to perfect calibration.
1.0
1e-05
0.8
0.05
Accuracy
Property 1b: Step-function limit that maximizes MisD ef-
0.6-
A 0.5
fectiveness (A + 0). As X - O, (z) converges to the closed-form solution of Eq. (i.e., step function), then the following corollary holds. The proof is listed in Appendix A.
ECE
0.2
0.0
0.0
0.2
0.8
1.0
Corollary 4.3. When + O, the proposed normalized sig- moid curve assigns higher confidence to all correctly classified samples than to any misclassified sample, i.e.,
Confidence
Figure 2: Visualization of the pro- posed normalized sigmoid curve with different À values alongside ECE in the reliability diagram.
P(Conf(N*+) > Conf(N-)) 11
(6)
where N and N denote the sets of correctly classified and misclassified samples, respectively. In this case, the reliability curve approaches step-like curve (ef. Figure 2 with À le 5).
5

---

Published as a conference paper at ICLR 2026
Properties la and 1b characterize the effect of the limit behavior as À varies. A smaller À leads to stronger separation and hence better MisD performance, but pushing À - 0 results in overly aggressive probability updates, causing instability and potential over-fitting on the calibration set. Therefore, in practice, an intermediate is chosen to balance separation strength and stability.
Precision dominance antees strictly higher precision than perfect calibration for correctly classified samples in the high- confidence region, and symmetrically outperforms perfect calibration for misclassification detection in the low-confidence region, Consequently, given the distribution of samples over confidence values (i.e., w())), we arrive at the following theorem (The proof is listed in the Appendix (.44)
Property 2:
perfect calibration. The proposed reliability curve guar-
Theorem 4.4, Let r E (0.5, 1) and let w(s) 2 0 be any weight function on [0, 1] with J w(s) ds > O. The precision for correct-pred corecct-rediction detection (i.e., Prec*) satisfies the following inequality:
Jt w()) (s) ds
Jr¹ (.((s) ss =: Prec (r: w).
Prece (r: w)
S. w(s) ds
([)())) ds
Moreover, for the incorrect prediction detection with (0, 0.5), the precision (i.e., Prec satisfies the following inequality:
Prece (r; w) = So w()) (1 (s)) ds So (s)ssd))) ds
So ((s)) (1 8) ds
2
=: Proc (r; w).
So w(s) ds
Less tolerance for prediction mixing. MisD emphasizes separating correctly and incorrectly classified samples; hence, less mixing between these two groups at the confidence level critical. Entropy is natural metric that can be used to measure the degree of mixing. Therefore, can characterize the tolerance of the reliability curve for prediction mixing at every confidence level by the entropy. We then have the following theorem (The proof is listed in Appendix (A.6]:
Property 3:
Theorem 4.5. Given arbitrary confidence then the corresponding entropy for the proposed reliability curve(i.e., (()) and perfect-calibration curve satisfy the following inequality.
Mv(s)= & Maiag(s). M(s) == -P((N)) (og((P(NN))) - IPN,)) log(P(())
(7)
where P(NT) and P(N,)) denote the probabilities of being correctly and incorrectly classified given confidence 8, respectively; i.e., IPNN+ P(correct/com conf = 8) and P(N; P(N)).
This theorem indicates that the proposed reliability curve (i.e., (s)) has less tolerance for mixing of correctly and incorrectly classified samples.
Taken together, Properties 1-3 provide coherent theoretical picture: the proposed curve continu- ously interpolates between perfect calibration and ideal separation (Property 1), provably achieves higher precision for both correct and incorrect prediction detection (Property 2), and admits lower entropy at each confidence level, indicating less tolerance for prediction mixing (Property 3). These results highlight that (s) aligns better with the goal of improving MisD. Building these insights, we next introduce a practical post-hoo hoc method that operationalizes the proposed reliability curve.
4.2 SURROGATE LOSS
Despite the advantages of the proposed reliability curve (i.e., (s))), directly realizing it as a training objective is challenging. In particular, the reliability curve cannot be straightforwardly expressed as differentiable loss, and an alternative would be approximate it through binning on a held-out calibration set, similar to ECE (Guo et al., 2017). However, calibration sets are usually small in practice, which causes large variance the empirical bin estimates and makes such optimization unreliable. To tackle this issue, we propose simple alternative to achieve it.
Specifically, we approximate the desired behavior the reliability curve through a differentiable surrogate penalty. Recall our proposed curve (s(s,, which smoothly maps the confidence score $ E [0. to an expected accuracy curve, describing how prediction accuracy is expected to vary with confidence. Under this formulation, each prediction contributes observed correctness signal. When a misclassified sample receives high confidence (e.g.. s), its observed accuracy (0) lies far below the expected value (s) the sigmoid-shaped reliability curve at that confidence level. The

---

Published as a conference paper at ICLR 2026
resulting gap therefore induces a penalty proportional to (s) - 0, reducing upward distortion in the high-confidence region of the reliability curve. Similarly, when a correctly classified sample is assigned low confidence, its observed accuracy (1) lies far above the expected value V'(s) in the low-confidence region. The corresponding gap, - (s), induces a penalty that encourages its confidence to increase toward the expected accuracy, thereby correcting distortion in the low- confidence region. However, (s() only constrains the confidence, i.e., the highest probability, while ignoring the distribution of the remaining probabilities. This may lead to optimization difficulties and suboptimal performance. Therefore, we further incorporate a constraint on the full probability distribution to make the constraint more comprehensive:
LSUR= =min B | EIEN- 011- -y. log fxx))
(8)
+(1 - 3) IN- ΣjEN- ((V(s), -11/e...., 1/c] log f(x))},
where 3 is a hyperparameter balancing the constraints on correctly and incorrectly predicted samples, and of--.3 is fusion function, which can be chosen as summation or multiplication. The first term of of encourages the reliability curve to align with the (s). The second term of Φ .} provides a full probability constraint to avoid the limitation of only constraining the predicted class probability from the first term. Specifically, for correctly predicted samples, it is standard Cross Entropy, which enforces a low-entropy probability distribution, while for incorrectly predicted samples, it imposes a constraint by comparing the predicted probability vector with the uniform distribution [1/e, 1/c], thereby encouraging a high-entropy probability distribution, In summary, the first term drives the main alignment with the proposed reliability curve (i.e., (s)). and the second term complements with full- probability regularization, jointly yielding a more stable and effective objective.
4.3 LIGHTWEIGHT META NETWORK
To avoid altering the model predictions and introducing significant overhead, we propose a post-hoc method. It employs lightweight meta network (i.e., LMN) to learn the temperature coefficients for refining the VLM's confidence. For instance, given an image v, its logits can be refined as:
2 = [Tu 2,,1, Fv.2. Ev,el,
Zμ,i Telip sim(E(x,), (tt)))
To obtain the instance-specific temperature coefficient T., we design lightweight network that takes as input the information from both modalities, without modifying the parameters of the pretrained or fine-tuned model. Concretely. the model's 's output logits Zu, the image embedding {(x,), and the predicted text embeddings ((pp) are each passed through a separate fully connected (FC) layer to map them into common latent space. These representations are then concatenated and fed into another FC layer, which projects them to scalar value corresponding to T. Formally, we have:
To = o (h. h. (b.)), h (z.)), h = FC, (E(x.)) h, = FC, (w),t)), (10)
where [-II)] denotes concatenation and a+ (2) log(1 ( (x))(())) is an element wise softplus activation (Dugas al. 2000). Finally, by substituting Eq. into the objective function in Eq.(8), we update only the parameters of these FC layers the calibration set, making the overall training procedure lightweight and efficient. The flowchart of the proposed meta network can be found in Figure 5
5 EXPERIMENT
In this section, we conduct experiments six public datasets to evaluate the proposed method in terms of different settings. Details of experiments are shown Appendix C and additional experiments are shown in Appendix D
The code is released at Code Link
5.1 EXPERIMENTAL SETUP
Datasets. We conduct the analysis six datasets covering specialized and
domains,
which include DTD (Cimpoi et al. 2014). Flowers102 (Nilsback & Zisserman 2008). EA (Helber al. 2019) RESICS45 (Cheng et al., 2017), MNIST (Deng 2012), and CUB (Wah et al.
7

---

Published as a conference paper at ICLR 2026
Table 1: Comparison results of recent confidence calibration OOD detection methods in the few-shot setting. Note that 1 indicates higher is better, J indicates lower is better.
Plowers
Methods
AUROCT
AUIPR-ST
AUPR-FF
FPR90-S.
FPRMN-FF
AUROCI
ALIPR-ST
AUIR-FF
PPR90-SU
FPR90-F)
Zero-shot CLIP
0.762
0.740
0.771
0.669
0.572
0.864
0.922
0.759
0.135
0.354
FesturCClippng
0,749
0.935
0.695
0.416
SCT
0.259
0.74]
0.776
0.685
0.557
0.868
0.926
0.761
0.429
0.337
VILU
0.769
0.759
0.762
0.678
0.521
0.875
0.913
0.772
0.101
0.329
LMN (Curs)
0,802
0,800
0.804
0.636
0.457
0.886
0.937
0.799
0.378
0.305
BurSAT
RESICS45
Methods
AUROC1
AUPR-ST
HPR90-E2
AUPR-ST
AUPR-ET
HPR90-SU
HPR90-BU
Zero-shor CLIP
0.65
0.501
0.782
0.742
0.824
0.711
0.636
0.508
PeatureClipping
0.685
0.613
0.727
0.729
0.536
0.781
0.827
0.705
0.638
0.501
SCT
0.681
0.534
0.791
0.754
0.682
0.784
0.826
0.716
0.633
0.501
0.723
LMN (Curs)
0,788
0,698
0,885
0.685
0.468
0.808
0.845
0.741
0.597
0.445
MNIST
Methods
AUROCT
AUPR-S1
AUPR-E5
FPR90-S.
FPR90-EL
AUROC1
AUPR-ST
AUPR-ET
FPR90-S]
FPR90-EL
Zero-sbot CLIP
0.813
0.565
0,919
0
0.482
0.807
0.839
0.758
0.767
0.554
PeatureClipping
0.756
0.768
0.541
SCT
0.837
0.661
0.933
0.486
0.123
0.808
0.839
0.759
0.766
0.547
VILL
0.877
0.769
0.954
0.350
0.263
0,801
0.827
0.753
0.769
0.563
LMN (Cuns)
0.915
0,779
0.965
0,200
0,205
0.812
0.846
0.764
0.756
0.532
For each dataset, we adopt the official training and test splits provided in Li ct al. (2022). and construct a few-shot calibration set by randomly sampling a small subset from the training set.
2011p.
Baselines. We evaluate our method using pretrained CLIP as the base model and compare it with two recent calibration approaches (FeatureClipping (Tao al. (025) and DOR (Wang et al.) (2055), strong VLM-based OOD detector (SCT (Yu et al. 2024)). ) and the latest MisD-oriented method for VLMs (ViLU (Lafon et al. (0255). We also acknowledge ledge FSMisD (Zeng et al., 2025), a recent MisD-oriented method for VLMs, but its code is not publicly available and cannot be included. In
addition, we evaluate two commonly used prompt-tuning CLIP variants: CoOP (Zhou et al. 2022b)
(textual) and VPT (Jia et al., 2022) (visual).
Implementation details. We use CLIP ViT-B/32 as the visual backbone. For both visual and textual
prompt learning, set the prefix size to 16 (Zou al. 2022b Jia et al. 2022). We use SGD the optimizer, with the number of training epochs selected from 100, 150, 200] and the learning rate selected from (0.001, 0.002, 0.005). We adopt 16-shot setting for the calibration set. When the base model is prompt-tuning CLIP, we split the original 16-shot calibration set into two parts: one is used for learning the prompts, and the other is reserved for training our post-hoc calibration network.
Evaluation metrics. To evaluate the MisD, we measure the ranking capability of confidence scores, i.e., the ability rank correctly classified samples ahead misclassified ones. Following existing works (Corbière et al. 2019 Hendrycks et al. 2018). we adopt several widely used metrics: AUROC, AUPR-Success, AUPR-Error,
SBR90%%-Scceess TPR, and FPR90% 1er---Error-PP- -PR. Since our MisD evaluation simultaneously considers both correct and incorrect predictions. the positive class in AUPR and FPR redefined according to the detection target: for Success, correctly predicted samples are treated as positives, whereas for Error, incorrectly predicted samples are treated as positives.
5.2 MAIN RESULTS
We first evaluate the MisD performance of recent confidence calibration methods, the OOD detection method, and our proposed method applied to the pretrained CLIP. The results on five commonly used metrics (ie., AUROC, AUPR-S Success,,,s,, AUPR-Error, FPR90- Succes,, and FPR90-Error) are reported in Table from which make the following observations:
First, the confidence calibration methods cannot help CLIP to distinguish between the correctly predicted samples and the incorrectly predicted samples. For example, average across all datasets, FeatureClipping improves AUROC by only about 0.7% over the pretrained CLIP. Moreover, such marginal improvements are further undermined by their inconsistency, enhance CLIP uniformly across all evaluation metrics. This aligns with Theorem [.2] confirming that confidence calibration imposes fundamental limit achievable MisD performance.
both methods fail to
8

---

Published as a conference paper at ICLR 2026
AUROC
AUROC
FPR90-E
FPR90-E
0.50
0.86
0.50
9.55
QUTSETP
0.68
0.45
0.50
0.86
0.40
0.35
3.45
0.50
0.78
0.30
2.40
Mew 8.0% 45A MWM
060 Bach M55 MW5
015 Mlarr
KINE PRNST
010 Woure Sanck
Figure 3: Bar plot comparison of textuul-prompt (i.., TP in Figure) and visual-prompt (i.e.. VP in Figure) CLIP before and after applying our proposed method, evaluated in terms of AUROC and FPR90-Error across DTD, Flowers 102, EuroSAT, RESICS45, MNIST, and CUB datasets.
Second, compared to the VLM-based calibration and OOD detection methods, the proposed method always outperforms them by large margins across all evaluation metries. For example, the proposed method on average improves by 6.1%, 10.5%, 4.7%, 13.4%. and 22.9%, compared to the pretrained CLIP, in terms of AUROC, AUPR-SUnn -Seucsss,,ss, AUPR Error, FPR90- more, even when compared with the strongest baseline ViLU, our method still achieves substantial gains of 2.8%, 5.1%. 2. 1%,,5 5.4%, and 18.7% on the same metrics. These results demonstrate the superiority of the proposed method.
and FPR90-Error. Further-
5.3 EVALUATION ON VARIOUS BASE MODELS
To evaluate the effectiveness of the proposed method on more base models, we adopt two types of prompt-based CLIP models (i.e., textual prompt and visual prompt). Since prompt-based methods require training set learn the prompt embeddings, we leverage the few-shot calibration set, which
is originally divided into training and validation portions. Specifically, following (Wang et al. 2021 the training portion is used to learn the prompt embeddings, while the validation portion serves both to validate the learned prompts and to train the proposed post-hoc calibration model.
The results are shown in Figure 3 We have the following observations: our method consistently improves the performance both types of prompt-base based CLIP across multiple datasets. For example, on average, our method brings an improvement of 1.2% in AUROC and a reduction of 6.2% in FPR90-Error for the textual prompt, and 1.1% and 4.4%, respectively, for the visual prompt. These results demonstrate the applicability of our model different forms of base models.
5.4 EVALUATION ON OPEN-VOCABULARY SETTING
A key advantage of pretrained CLIP lies in its strong zero-shot capability, enabling it to generalize well to unseen classes. However, fine-tuning operations, including prompt learning, may compromise this ability by overfitting to the training classes. To examine whether our proposed post-hoe method preserves the zero-shot capability, we design an open-vocabulary evaluation and compare LMN with both zero-shot CLIP and the open-vocabulary calibration method DOR (Wang al., 2025 (a CoOP-based calibration method). Specifically, we randomly sample subset of classes for calibration, while the remaining unseen classes are reserved for testing. The results are shown in Table
The experimental results demonstrate that our method not only avoids degrading CLIP's performance under the open- vocabulary setting, but also leads to consistent improvements across almost all datasets and evaluation metrics. For example, on the DTD dataset, our method achieves a 7. % and 5.9% relative improvement over CLIP and DOR in terms the FPR90-S metric. This indicates that our approach is capable of enhancing model performance while preserving CLIP's inherent strengths. We attribute this to the post-hoc nature of our method, which does not modify the pretrained parameters of CLIP and thus maintains the intrinsic generalization ability of its image and text embeddings. By retaining this property, the proposed calibration method can leverage the strong representational power of CLIP while improving its reliability on unseen classes.

---

Published as a conference paper at ICLR 2026
Table 2: Results of open-vocabulary evaluation six datasets, measured by AUROC, FPR90-S, and
FPR90-E. 1 denotes that higher values are better, while & denotes that lower values are better.
DTD
Flowers102
EuroSAT
Methods
AUROCT
FPR90-S)
FPR90-EL
AUROC
FPR90-S]
FPR00-E]
AUROCT
FPR90-S.
FPR90-E,
Zero shot CLIP
0.760
0.642
0.604
0.853
0.407
0.397
0.608
0.747
0.882
LMN (Ouns)
0.770
0.596
0.598
0,858
0.396
0.387
0.614
0.743
0.875
RESICS45
MNIST
CUB
Methads
AUROC1
FPR90-S)
FPR90-EL
AUROC
FPR90-S, S.}
FPR90-E.
AUROC1
FPR90-5, S.
FPR90-FL
Zero-shot CLIP
0.780
0.601
0.536
0.854
0.545
0.322
0.789
0.641
0.490
DOR
0.764
0.621
0.543
0.842
0.573
0.331
0.779
0.667
0.512
LMN (Ouns)
0.782
0.599
0.527
0
0.539
0.301
0.790
0.653
0.472
(P96641
148 181
210 10C
Figure 4: Bar plot comparison AUROC, FPR90-S, and FPR90-E metrics CLIP, CLIP with full probabilities constraint, and CLIP with our full method (i.e., first three figures) the RESICS45 dataset. The corresponding reliability diagrams are shown in the last three figures.
5.5 ABLATION STUDIES
The key component of the proposed method is the surrogate loss LSUR, which consists of two parts: (i) a confidence regularization term that encourages the predictions align with the target reliability curve (i.e., (())), and (ii) full-probas -ull-probability constraint (FPC) that complements the confidence regularization by regularizing the entire probability distribution, preventing the training instability cause by single probability (i.e., confidence) were optimized. Therefore, we do not report separate ablation with only the confidence regularization term, as it intended to be used together with FPC. To verify the effectiveness of these components, we visualize the reliability diagrams on the RESICS45 dataset under three settings: the original CLIP, CLIP with FPC, and CLIP with the full proposed method. The visualizations together with the corresponding quantitative results are reported in Figure Additional ablation studies on other datasets are provided Appendix D.I
From Figure 4 draw the following observations. First, the complete objective achieves the best overall performance, and using FPC alone also improves MisD performance compared to the pretrained CLIP, indicating that both components of the surrogate loss contribute effectively. Second, from the reliability diagrams, we see that the pretrained CLIP model exhibits overconfidence. Incorporating FPC alleviates this issue to some extent; however, the resulting curve still deviates from the target (s)) Finally, with the proposed confidence regularization term, the reliability curve aligns much more closely with (s),, demonstrating that our surrogate loss indeed possesses the desired ability to calibrate predictions toward the target normalized sigmoid reliability curve.
6 CONCLUSION
In this work, we revisited the reliability diagram of confidence calibration and established its connec- tion with MisD. We showed that the standard calibration objective inherently limits MisD performance, and proposed new reliability curve the calibration objective. This reliability curve theoretically improves MisD performance and reduces the mixing correct and incorrect predictions. As the value of the reliability curve is statistical result and thus non-differentiable, we address this issue by designing a differentiable surrogate loss. Furthermore, to preserve the predictive power of VLMs, we developed lightweight post-hoc framework that employs meta network to produce sample-specins temperature factors. Both theoretical analysis and extensive experiments confirmed that our approach consistently enhances MisD performance while maintaining model accuracy.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGEMENT
Xiaofeng Zhu was supported in part by the National Key Research and Development Program of China under Grant N0.2022YFA 1004100)) 2F02222A10000000) Ping Hu was supported in part by the National Natural Science Foundation of China under Grant (No.624766 62466447048 and the Sichuan Science and Technology Program under Grant 200.2025YFMSO000 Xiaoshuang Shi was supported in part by the National Natural Science Foundation of China (No. 62276052).
REPRODUCIBILITY STATEMENT
We have made extensive efforts to ensure the reproducibility of our work. The model architecture, training procedure, and hyperparameter settings described in Appendix [C2 Complete proofs of the theoretical results are presented in Appendix [A] The datasets used in our experiments are publicly available, and the preprocessing steps explained in the Appendix [ Moreover, we provide an anonymous link to the source code in the supplementary materials to facilitate reproduction of our experiments (Code Link).
REFERENCES
Runnan Chen, Youquan Liu, Lingdong Kong, Xinge Zhu, Yuexin Ma, Yikang Li, Yuenan Hou, Yu Qiao, and Wenping Wang. Clip2scene: Towards label-efficient 3d scene understanding by clip. In CVPR, pp. 7020-7030, 2023.
Gong Cheng, Junwei Han, and Xiaoqiang Lu. Remote sensing image classification: Benchmark and state of the art. Proceedings of the IEEE, 105((0)):865-1883, 2017.
Mircea Cimpoi, Subhransu Maji, lasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. De- scribing textures in the wild. In CVPR, pp. 3606-3613, 2014.
Charles Corbière, Nicolas Thome, Avner Bar-Hen, Matthieu Cord, and Patrick Pérez. Addressing failure prediction by learning model confidence. In NeurlPS, volume 32, 2019.
Can Cui, Yunsheng Ma, Xu Cao, Wenqian Ye, Yang Zhou, Kaizhao I Liang, Jintai Chen, Juanwu Lu, Zichong Yang. Kuei-Da Liao, et al. A survey multimodal large language models for autonomous driving. In CVPR, pp. 958-979, 2024.
Li Deng. The mnist database of handwritten digit images for machine learning research [best of the web]. IEEE signal processing magazine, 29(6): 141-142, 2012.
Charles Dugas, Yoshua Bengio, François Bélisle, Claude Nadeau, and René Garcia. Incorporating second-order functional knowledge for better option pricing. In NewrlPS, volume 13, 2000.
Ruiyi Fang, Bingheng Li, Zhao Kang, Qiuhao Zeng, Nima Hosseini Dashtbayaz, Ruizhi Pu, Boyu Wang, and Charles Ling. On the benefits of attribute-driven graph domain adaptation. The Thirteenth International Conference on Learning Representations, 2025.
Ruiyi Fang, Shuo Wang, Ruizhi Pu, Qiuhao Zeng, Hao Zheng, Ziyan Wang, Jiale Cai, Zhimin Mei, Song Tang, Charles Ling, al. Graph domain adaptation via homophily-agnostic reconstructing structure. In AAAI, 2026.
Arindam Ghosh, Thomas Schaaf, and Matthew Gormley. Adafocal: Calibration-awwaee adaptive focal loss. In NeurlPS, volume 35, pp. 1583-1595, 2022.
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Weinberger. On calibration of modern neural networks. In ICML, pp. 1321-1330. PMLR, 2017.
Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth. Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 12(7):2117-2266, 2019.
Dan Hendrycks and Kevin Gimpel. baseline for detecting misclassified and out-of out-of-distribution examples in neural networks. arXiv preprint arXiv: 16/0.0//0000200000 2016.

---

Published as a conference paper at ICLR 2026
Dan Hendrycks, Mantas Mazeika, and Thomas Dietterich. Deep anomaly detection with outlier exposure. arXiv preprint 1.12:08110004000 2018.
Jincheng Huang, Jie Xu, Xiaoshuang Shi, Ping Hu, Lei Feng, and Xiaofeng Zhu. The final layer holds the key: A unified and efficient gnn calibration framework. arXiv preprint arXiv:2505. 11335, 2025.
Menglin Jia, Luming Tang, Bor-Chun Chen, Claire Cardie, Serge Belongie, Bharath Hariharan, and Ser-Nam Lim. Visual prompt tuning, In ECCV, pp. 709-727. Springer, 2022.
Ulf Johansson, Cecilia Sonstrod, Tuwe Lofstrom, and Henrik Bostrom. Confidence classifiers with guaranteed accuracy or precision. In Conformal and Probabilistic Prediction with Applications, pp. 513- 533. PMLR, 2023.
Muhammad Uzair Khattak, Hanoona Rasheed, Muhammad Maaz, Salman Khan, and Fahad Shahbaz Khan. Maple: Multi-modal prompt learning. In CVPR, pp. 19113-19122, 2023.
Ranganath Krishnan and Omesh Tickoo. Improving model calibration with accuracy versus uncer- tainty optimization. In NeurIPS, volume 33, pp. 18237-18248, 2020.
Mare Lafon, Yannis Karmim, Julio Silva- Rodriguz,, Paul Couairon, Clément Rambour, Raphaël Ismail Ben Ayed, Jose Dolz, and Nicolas Thome. Vilu: Learning vision- language uncertainties for failure prediction. arXiv preprint arXiv:2507.0763 2025.
Chunyuan Li, Haotian Liu, Liunian Li, Pengchuan Zhang, Jyoti Aneja, Jianwei Yang, Ping Jin, Houdong Hu, Zicheng Liu, Yong Jae Lee, al. Elevater: A benchmark and toolkit for evaluating Singuag--augmetter visual models. In NeurIPS, volume 35, pp. 9287-9301, 2022.
Song-Lin Lv, Yu- Yang Chen, Zhi Zhou, Yu-Feng Li, and Lan-Zhe Guo. Contrast-aware calibration for fine-tuned clip: Leveraging image-text alignment. arXiv preprint arXiv:2501.19066 2025.
Jishnu Mukhoti, Viveka Kulharia, Amartya Sanyal, Stuart Golodetz, Philip Torr, and Puneet Dokania. Calibrating deep neural networks using focal loss. NeurIPS, volume 33, pp. 15288-15299, 2020.
Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. In ICCV, pp. 722-772-7777 729, 2008
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In ICML, pp. 8748-8763. PmLR, 2021.
Linwei Tao, Minjing Dong, and Chang Xu. Feature clipping for uncertainty calibration. In AAAI, volume 39, pp.
2025.
Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie. The caltech-ucsd birds-200-2011 dataset. 2011.
Shuoyuan Wang, Jindong Wang, Guoqing Wang, Bob Zhang, Kaiyang Zhou, and Hongxin Wei. Open-vocabulary calibration for fine-tuned clip. In ICML, 2024.
Shuoyuan Wang, Yixuan Li, and Hongxin Wei. Understanding and mitigating miscalibration in prompt tuning for vision-language models. In ICML, 2025.
Xiao Wang, Hongrui Liu, Chuan Shi. and Cheng Yang. Be confident! towards trustworthy graph neural networks via confidence calibration. In NeurlPS, volume 34, pp.
2021.
Miao Xiong, Ailin Deng, Pang Wei W Koh, Jiaying Wu, Shen Li, Jianqing Xu, and Bryan Hooi. moxximtty-informe calibration for deep neural networks. In NeurIPS, volume 36, pp. 68511- 68538, 2023.
Jia-Qi Yang, De-Chuan Zhan, and Le Gan. Beyond probability partitions: Calibrating neural networks with semantic aware grouping. In NeurlPS, volume 36, pp.
2023a.
12

---

Published as a conference paper at ICLR 2026
Liang Yang, Xin Chen, Jiaming Zhuo, Jin, Chuan Wang, Xiaochun Cao, Zhen Wang, and Yuanfang Guo. Disentangled graph spectral domain adaptation. In ICML, 2025a.
Xihong Yang, Jin Jiaqi, Siwei Wang, Ke Liang, Yue Liu, Yi Wen, Suyuan Liu, Sihang Zhou, Xinwang Liu, and En Zhu, Dealmve: Dual contrastive calibration for multi-view clustering. In Proceedings of the 31 ACM International Conference on Multimedia, pp. 337-346, 2023b.
Xihong Yang, Siwei Wang, Fangdi Wang, Jiaqi Jin, Suyuan Liu, Yue Liu, En Zhu, Xinwang Liu, and Yueming Jin. Automatically identify and rectify: Robust deep contrastive multi-view clustering in noisy scenarios. In ICML. PMLR, 2025b.
Xihong Yang, Yiqi Wang, Jin Chen, Wenqi Fan, Xiangyu Zhao, En Zhu, Xinwang Liu, and Defu Lian. Dual test-time training for out-o--distribution recommender system. IEEE Transactions on Knowledge and Data Engineering, 37(6) 2025c. doi: 10.1 10.10099 600
Geng Yu, Jianing Zhu, Jiangchao Yao, and Bo Han. Self-calibrated tuning of vision- lasounn models for out-of- -i-or---istiiuttion detection. In NeurIPS, volume 37, pp. 56322-56348, 2024.
Fanhu Zeng, Zhen Cheng, Fei Zhu, and Yao Zhang. Towards efficient and general-purpose few-shot misclassification detection for vision-language models. arXiv preprint arXiv:2503.20492 2025.
Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. In CVPR, pp. 11975-11986, 2023.
Xu-Yao Yao Zhang, Guo-Sen Xie, Xiuli Li, Tao Mei, and Cheng-Lin Liu. A survey on learning to reject. Proceedings of the IEEE, 11 (11 215, 2023.
Zihao Zhao, Yuxiao Liu, Han Wu, Mei Wang, Yonghao Li, Sheng Wang, Lin Teng, Disheng Liu, Zhiming Cui, Qian Wang, al. Clip in medical imaging: A comprehensive survey, arXiv preprint arXiv:2312.07353, 2023.
Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Conditional prompt learning for vision-language models. In CVPR, pp. 16816-16825, 2022a.
Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Learning prompt for vision- language models. ICCV, 130(9):2337-2338, 2022b.
Fei Zhu, Zhen Cheng, Yao Zhang, and Cheng-Lin Liu. Rethinking confidence calibration for failure prediction. In ECCV, pp. 518-536. Springer, 2022.
Fei Zhu, Zhen Cheng, Xu-Yao Zhang, and Cheng Liu. Openmix: Exploring outlier samples for misclassification detection. In CVPR, pp. 12074-1200 12083, 2023.
Jiaming Zhuo, Yuwei Liu, Yintong Lu, Ziyi Ma, Kun Fu, Chuan Wang, Yuanfang Guo, Zhen Wang, Xiaochun Cao, and Liang Yang. Dualformer: Dual graph transformer. In ICLR, 2025.
13

---

Published as a conference paper at ICLR 2026
A THEORETICAL PROOF
A.1 PROOF FOR LEMMA
Lemma A.I. Given a confudence interval (a, bl, let w(s) denote the density of the sample at confidence level Then, the precision of correct predictions within (a. b] can be derived from region under the reliability curve f((s) over (a, b), i.e., Prec* (So (()){(()dd)) Sa w(s)ds. Similarly, the precision of incorrect predictions over [a, can be derived from the region above the curve via Sh w(s)ds.
Prec = U*"
Proof. The formula of precision is defined as:
correct predictions with E [a. b)
Precision all predictions with E [a, bj
(11)
The reliability curve can be represented as f(s) = [[coorret/conffideeee = s), thus the number of correct predictions in the interval (a, b] can be represented
= S. (() (ccuraccsy)))))d
f" w(s) (ss))((corrott confidence s)ds. (12)
correct predictions =
The number of the whole sample is:
all predictions = / w(s)ds.
(13)
Combining the above three equations, we have the following equation:
Prec So w(())
(ooPPcccrrrettccofieennee s)ds
So w())s(s)ds)
(14)
0 wrs)ds
J w(s)
where -e [P(correet/coonfdeecce = s)ds is the area under the reliability curve over given confidence
interval [a, b). Then the precision equals that area divided by the length of the interval.
For the incorrect prediction detection, which just needs replace the numerator of Eq. III with the number incorrect predictions, thus we have:
5 (incorertr [fincooreectcooffdeeneee confidence s)ds
Sa (wwss((1 f(s))ds)
Prec (a
(15)
()
J w(s)ds
S w()dds
The proof is completed.
A.2 PROOF FOR COROLLARY 4.3
Corollary A.2 (Ideal separation case). When X - O, the proposed normalized sigmoid curve assigns higher confidence to all correctly classified samples than to any misclassified sample, i.e.,
P ((Conf(N*+) > Conf(N =1
(16)
where N and N denote the sets of correctly classified and misclassified samples, respectively. In this case, the reliability curve approaches ttke curve (cf. Fig. 2 with X = 1e - 5).
Proof. As A + O, Figure shows that all samples with confidence greater than 0.5 are correctly classified (accuracy 100%), ), while all samples with confidence below 0.5 are misclassified (accuracy 0%). This implies that the confidence of every correctly classified sample is strictly greater than 0.5, and the confidence of every misclassified sample is strictly less than 0.5. Therefore, the confidence of any correctly classified sample is always higher than that of any misclassified sample.
14

---

Published as a conference paper at ICLR 2026
A.3 PROOF FOR THEOREM 4.2
Theorem A.3. Let r E [0,1] be a confidence threshold. Under perfect calibration, the preci- sion for the correct prediction detection and incorrect prediction detection tasks is Precia.1) = Es-m(s)ss)r.111s)) and Prec =lsss s). respectively.
Proof. Given perfect calibration model, which reliability curve is diagonal line (i.e., f(s)
8). For the correct prediction detection task, predictions with confidence above the threshold (i.e., E (r, 1)) are considered detected correct predictions. By Lemma the precision of correct prediction [r, 1]
frr (s))S(s)ds)
Prec
(17)
Sr w(()ds
Substituting f(s) 5, we have:
Prect: J w(s)f(s)ds) E (s S (()dds
(18)
Similarly, the precision of incorrect predictions is:
Prec S6d (css(1(11)) f(s)) (a s).
(19)
So w(s)ds
The proof is completed.
A.4 PROOF FOR THEOREM
4.4 (restated)). Let r E (0.5,1) 1) and let w 2 be any weight function [0, with f w(s) ds The precision for correct-pres prrercttpeeiitiio detection (i.e., Prec*) satisfies the following inequality:
Theorem A.4 (Theorem
Prec (r; w) = ft w(s) ((s ds
Sr (w(s) sds
=: Prec (ri w).
S w(s) ds
Sr¹ (((ss ds
Moreover, for the incorrect prediction detection with E (0, 0.5), the precision (i.e., Prec satisfies the following inequality:
(r, = So w(s) (1 (x)) ds
2 So (((s) (1 8) ds
=: Proc (r; w).
So w(s) ds
So w(s) ds
Proof. Before proving the Theorem, state the following lemma (proof in Appendix A.5
Lemma A.5 (Above-diage) diigooal on the high-confidence side), Given the proposed normalized sigmoid
curve fi.e., (xx)) and diagonal line (i.e., y 2). then V is strictly concave [0.5, 1], satisfies
(0.5) = 0.5 and (1(1) = 1, and hence:
v (x) 2 x for all x [0.5, 1],
V(x) > for all (0.5, 1).
By Lemma A.5 let D(s)= = V(s)) for E (r. and D(s) > 0 on (r, 1). Therefore
J.
w(s) V (s) ds L. w(s)) s ds = f w(s) (s)) ds 2 0,
and the inequality is strict if w places positive mass on some subset of (r, 1) where > 0. Division by the common normalizer f w(s) ds > 0 yields the claim, then the inequality Prec (r; > Prec diag holds.
15

---

Published as a conference paper at ICLR 2026
For the incorrect prediction detection, since (s) [0, 0.5] (strict on (0,0.5) by symmetry of the above argument), an analogous statement holds for incorrect prediction detection on [0, r] with 0,
r < 0.5: for any nonnegative w with Jo ur(s) 6) ds
So w(s) (1 (()) ds
(()) (1 s) ds
So w())
So w(s) ds
Then the incquality Prece (r; w) > Prec diag holds.
A.5 PROOF FOR LEMMA A.S
Lemma (Lemma A.5 (restated)). Given the proposed normalized sigmoid curve (i.e., v(x)) and
diagonal line (i.e., y x), then V is strictly concave on [0.5, 1J. satisfies V (0.5) = 0.5 and (11 1, and hence:
(xx) 2 x for all x E [0.5, 1],
v(x) for all (0.5,
Proof. Let o(x) == o( ( Since o (2) = =z) -(()) and o" d"(z) = o o(x))
2x(z)), we have for x > 0.5 that I 0.5 Hence
x > 0 and thus o ( z. implying a" (= <0 0.
(x(x)) = xt 0" 0.5
1
0 (a E (0.5, 1)),
ie., ø is strictly concave on [0.5, 1]. The normalization
<xx) x(0)
( -00)= ( 0.5 (-0a)) <(1) (0.5)
0.5
T (x)
6(1) 60))
an affine transform and therefore preserves strict concavity. Direct evaluation gives
((1) 1, (0.5) = o(0) a--a) 1 (1)o(a)) (aa) - o( ((a) - (1-o((a) 2' a
0.5 A > 0.
=
Define F(x) == (xx) - x. Since V is strictly concave and - linear (hence both concave and
convex), F is strictly concave on [0.5, 1], and F(0.5) F(1) O. For any (0.5, 1), write I == A 0.5 + (1 A) with X E (0, 1); strict concavity yields
F(x) > XF(0.5) + (1 - A)F((1 0.
Thus ((x) > x on (0.5, 1) and V (x) 2 x on 0055, 1].
A.6 PROOF FOR THEOREM 4.5
Theorem A.6 (Theorem 4.3 (restated)). Given arbitrary confidence s, then the corresponding entropy for the proposed reliability curve(i.e., (s)) and perfect-calibration curve satisfy the following inequality.
My (s) S (s)ag(s). M(s) : == 1OW,
(20)
where N and N, denote the conditional probabilities of being correctly and incorrectly classified given confidence s, respectively; i.e., N. P(correct conf = s) and N = N
Proof. Since the vertical axis of the reliability diagram is accuracy, and given an arbitrary confidence we have P(N+)) Acc(s) and P(N,) = - Acc(s). Due to P(N+) + P(N.) = 1, then the entropy can written binary entropy form:
M(s) -pp Inp, (1 In(1 ps)),
(21)
where p. P.
16

---

Published as a conference paper at ICLR 2026
Then, obtain the first-order derivation of ps:
(M(s) = -(In ps + 1 (In(1 - p) + 1)) = -ln
(22)
Öp.
When Ps 0.5, ((s) reaches its maximum value in the interval [0, 1] and df((s) 0p3
0. Thus, the
closer Ps is to 0.5, the larger the entropy becomes.
Let us recall the Lemma A.5
when
[0, 0.5] have (s(s) $ - 0.5, then:
(8) - 0.5 = 0.5 1/ (8) 2 0.5 8 == |s 0.5/.
(23)
Similarly, we can easily obtain that when [0.5, 1] have (s(s)
0.5, then: (24)
( 0.5 = T 0.5 2 0.5 = 0.5/.
Based on Eq. (23) and Eq. 24p. (s) 0.5| |s 0.5| holds for all s E [0, Therefore, the ps of the perfect-calibratio curve lies closer to 0.5 than that of the proposed (s). As result, the perfect calibration curve exhibits higher entropy, thereby tolerating greater degree of mixing between correctly and incorrectly predicted samples.
B
MODEL DETAILS
FLOWCHART VISUALIZATION
This section gives a flowchart (i.e., Figure of the proposed lightweight meta networks (LMN).
Proj
Proj
T
LSUR
CLIP Logits
Softplus
a photo of "cat"
Predicted text embedding
Image embedding
Proj
Proj
Figure 5: Flowchart of the proposed lightweight meta network (LMN). Given the output logits of CLIP, image embedding, and predicted text embedding, LMN first projects them into shared subspace using distinct FC layers (indicated by different colors in the figure). The resulting vectors are concatenated and mapped to scalar via another FC layer, followed by a Softplus activation to produce the sample-specins т. Finally, the surrogate loss LSUR updates the FC layers.
B.2 UNDERSTANDING THE CONTRIBUTION OF EACH MODALITY LMN
The meta-network indeed leverages three types of signals (i.e., logits, image embeddings, and predicted text embeddings) and each contributes complementary information for predicting the effective temperature factor.
Logits. Logits as the primary calibration signal. Logits provide the most direct evidence for confidence misalignment, they encode the inter-class margins and the overall shape of the predictive distribution. This is the core quantity used by traditional temperature scaling. Here we describe the complementary roles of each modality and why all of them contribute meaningfully MisD:
Image embeddings. Image embeddings capture sample difficulty. Even when two samples share similar logits, their underlying visual characteristics may differ substantially. Image embeddings help
17

---

Published as a conference paper at ICLR 2026
DTD
Flowers102
140
Predicted
300
Predicted Count
Correct
120
100
Sanuee CAust
150
100
80
Prudiced Coorted frequencc)
Prodicted Gees Iserted frequency!
(a)
(h)
Figure 6: Predicted count (yellow) and correct count (blue) for each predicted class, sorted by prediction frequency.
LMN identify hard or visually atypical samples (e.g., unusual textures, crowded scenes, rare visual patterns). This allows LMN to incorporate sample-level visual cues.
Predicted text embeddings. Predicted text embeddings reveal systematic category-leve patterns. To further illustrate the semantic confusion patterns captured by the predicted text embeddings, we visualize the distribution of predicted samples across classes Figure ( The figure shows that CLIP tends to cluster certain misclassified samples around a few semantically related text prototypes. Incorporating text embeddings enables LMN capture these category tendencies and selectively increase correction strength for categories prone systematic confusion.
Taken together, the three modalities provide complementary and non-redundant information: log- its characterize confidence geometry, image embeddings characterize sample difficulty, and text embeddings capture semantic misalignment patterns.
B.3 MOTIVATION FOR CHOOSING
set the midpoint parameter r = 0.5 because corresponds to natural neutral threshold separating "uncertain" from "confident" predictions, which aligns with standard confidence-biat beaeeeeeee-eeddddddd calibration intuition (Johansson et al.) 2023). Moreover, placing the inflection point at the center of the interval (0, 1] avoids introducing asymmetry favoring either low- high-confidence regions, providing balanced and unbiased target curve.
C EXPERIMENTAL SETTING
C.1 DATASETS DETAILS
Table 3: The statistics of the used datasets.
Num. classes (1/)
Size training data
Avg. labeled data per class
Size test
DTD
47
3760
64
1880
Flowers 102
102
2040
6149
EuroSAT
10
27000
2200
5000
RESICS45
45
6300
110
25200
MNIST
10
60000
4696
10000
CUB
200
5594
5794
In the experiment section, we use six datasets. The statistics of the used datasets are reported in the Table Here, we provide description each them:
DTD (Cimpoi et al. 2014) The Describable Textures Dataset (DTD) is a continuously expanding collection of texture images captured in unconstrained environments. The
18

---

Published as a conference paper at ICLR 2026
annotations based Humaan-nterprettbbe attributes, reflecting perceptual characteristics textures.
Flowers 102 (Nilsback Zisserman 2008) a benchmark dataset that collects images of 102 distinct flower categories, each representing species commonly occurring in the United Kingdom.
3. EuroSAT (Helber al., 2019) EuroSAT focuses on the classification of satellite scene images, providing 10 predefined categories for evaluation.
4. RESICS45 (Cheng et al. 2017)
Designed for remote sensing image scene classification,
this benchmark dataset provides 45 categories of scenes and is publicly accessible.
5. MNIST (Deng 2012) MNIST serves benchmark collection of handwritten digit images, in which all samples are normalized terms of size and aligned to the center of uniform image grid.
6. CUB (Wah al. 2011) CUB is fine-grained bird species classification dataset, containing 200 categories with large intra-class variation and subtle inter-class differences. It serves as benchmark for subordinate categorization tasks and provides comprehensive annotations to support research in fine-grained visual recognition.
Table 4: Settings for the proposed method.
learning rate
B
Num. hidden
À
DTD
0.002
0.6
16
0.05
summation
Flowers
0,001
0.8
16
0.05
summation
EUroSAT
0.002
0.6
16
0.1
summation
RESICS45
0.001
0.8
64
0.05
multiply
MNIST
0.005
0.4
16
0.05
summation
CUB
0.002
0.9
8
0.05
multiply
C.2 IMPLEMENTATION DETAILS
We use CLIP ViT-B/32 as the visual backbone and report results averaged over 3 runs. For both visual and textual prompt learning, we set the prefix size (Zhou al., [022b] Jia et al. 2022). We use SGD the optimizer, with the number of training epochs selected from (100, 150, 200 and the learning rate selected from {0.001, 0.002, 0.005). We adopt 16 labeled samples per class (i.e,, 16-shot) as the calibration set. When the base model is prompt-tuning CL.PP, we split the original 16-shot calibration set into two parts: one used for learning the prompts, and the other reserved for training our post-hoc calibration network. Table describes describss the detailed settings and architecture for the proposed lightweight meta network based on pretrained CLIP.
D ADDITIONAL EXPERIMENTS
D.1 ABLATION STUDY
The key component of the proposed method the surrogate loss Lsu Rs which consists of parts: (i) confidence regularization term that encourages the predictions to align with the target reliability curve (i.e.. ((s)), and (ii) full-probability constraint (FPC) that complements the confidence regularization by regularizing the entire probability distribution, preventing the training instability cause by single probability (i.e., confidence) were optimized. Therefore, we do not report separate ablation with only the confidence regularization term, it is intended to be used together with FPC. To verify the effectiveness these components, we investigate the performance
the following variants: the original CLIP, CLIP with FPC, and CLIP with the full proposed method. The results reported in Table 5
According Table the complete objective function achieves the best performance. Notably, employing only partial components (i.e.. +FPC) also leads to improvements. Taken together, these findings verify the effectiveness of all proposed modules.
19

---

Published as a conference paper at ICLR 2026
Table 5: Results of the ablation study six datasets, measured by AUROC, FPR90-S, and FPR90-E.
1 denotes that higher values are better, while denotes that lower values are better.
DTD
Flowers
EuroSAT
Methods
AUROCT
FPR90-S]
FPR90-EL
AUROC
FPR90-S]
FPR00-E]
AUROCT
FPR90-S.
FPR90-E,
Zero shot CLIP
0.762
0.669
0.572
0.864
0.435
0.354
0.650
0.782
0.742
#FPC
+ALL
0.802
0.636
0.457
0.886
0.378
0.305
0.788
0.665
0.468
RESICS45
MNIST
CUB
Methads
AUROC1
FPR90-S]
PPR90-EL
AUROCT
FPR90 S.j
FPR90-E.
AUROC1
FPR90-5. S.
FPR90.FL E.
Zero-sbot CLIP
0.779
0.636
0.508
0.813
0.511
0.482
0.807
0.839
0,758
#PPC
0.794
0.615
0.482
0.908
0.228
0.235
0.808
0.613
0.440
+ALL
0,808
0.597
0.445
0.915
0.200
0.205
0.812
0.602
0.438
D.2 PARAMETER ANALYSIS
AUROC vs PT
AUPR-S VT
AUPRLE VS ST
0.90
0.35
DTC
0.85
RESKES45
AUROC
0.80
AUPR-S
AUPR-
0.75
0.7
0.50
0.70
BardAT
flowrit2
0.75
9.65
RES
8.60
0.2
0.1
0.6
0.4
0.6
5.8
0.
0.1
0.5
0.8
1.0
AUROC VT
AUPR-S VT
USAT
0.35
3.900
0.55
0.075
0.85
0.50
AUROC
0.850
0.825
0.80
0.800
0.75
20D
0.775
0.70
12512545
0.750
0.65
0.75
Figure 7: Line plots of model performance across six datasets (i.e., DTD, Flowers ETrOSAT, RESICS45, MNIST, and CUB) under varying hyperparameters. The first row shows the effect of 3. and the second row shows the effect of A. From left to right, each subfigure corresponds one evaluation metric: AUROC, AUPR-S, and AUPR-E. The curves illustrate how performance changes as the hyperparameter values increase.
In the proposed method, we employ the non-negative parameters(i.e. 3 and A). For B, it is used to achieve trade-off between the two terms of the objective function (i.e., LSUR). For X, which is used to control the smoothness of the reliability curve. To investigate the impact of 8 and À with different settings, we conduct experiments on all six datasets by varying the value of 3 in the range of 1. 1.0] and the value of À in the range of [0.01, 0.2). The results are shown in Figure] 7
From Figure 7 we make the following observations. First, for the hyperparameter (3, the proposed method achieves peak performance around @ = 0.6. Performance deteriorates when / is too large or too small, as the method then fails to balance focus between correctly and incorrectly predicted samples. Second, for the hyperparameter X, the method consistently performs well when À is set appropriately (e.g., [0.05, 0.1]). Similarly to 3, extreme values of À lead to inferior performance. This is because À controls the smoothness of & (s): if (()) is too sharp, the overly aggressive probability updates can cause unstable optimization; it is too smooth, the model loses the ability to effectively adjust predictions. In practice, we find that the parameter is highly stable across different datasets.
20

---

Published as a conference paper at ICLR 2026
As summarized in Table 4 five out of six datasets (f.e., spanning textures, fine- grained objects, satellite imagery, handwriting, and birds) select the same value À = 0.05. This consistency indicates that À does not require dataset-specific tuning. Therefore, recommend À 0.05 reliable default, which already achieves near-optimal performance in our experiments, with adjustments needed only in rare cases.
D.3 ANALYSIS THE LEARNED TEMPERATURE CORFFICIENT
DTD
Rlwars102
EUISSAT
RESICS4S
MNST
CUB
1.4
2.3
1.0
1.2
2.0
0.8
0.6
2.6
0.4
0.6
0.4-
0.2
0.2
0.2
0.0
Cor
Incor
0.0
6.0
0.0
Incor
0.3
Cor
Incor
0.0
Cor
In00
Figure 8: Comparison of learned T values for correct versus incorrect predictions across six datasets. Each bar represents the average for the corresponding sample type. The blue bars represent the correctly predicted samples, and the red bars represent the incorrectly predicted samples.
To verify that the learned mnstance-wise T indeed increases the confidence of correctly predicted samples compared with incorrectly predicted ones. Since the temperature coefficient is multiplied by the logits, a larger value of т results in higher confidence. We compare the average learned values between correct and incorrect predictions Figure In Figure we can observe that the average value of the learned r for correct predictions is larger than that for incorrect predictions, with larger gap on the gap is about 68.5% of the incorrect predictions. This demonstrates that the proposed method is able to capture the intrinsic differences between correctly and incorrectly classified samples, thereby enabling their discrimination.
all six datasets. The results visualized in
datasets. For example, on MNIST,
D.4 RESULTS ON SIGLIP BACKBONE
To further examine whether LMN generalizes beyond the CLIP-B/32, we additionally evaluate the method using CLIP-L/14 and SigLIP-B/16 Zhai et al. (2023) the backbones. SigLIP represents a modern vasion-language architecture trained with a different alignment objective and embedding structure, offering stronger testbed for assessing the Backbone-agnoste nature of LMN. Because most existing calibration and MisD-oriented baselines do not provide publicly available or repro- ducible implementations for SigLIP, SCT the only method that be feasibly adapted this backbone, and we therefore adopt it as the comparison baseline in this setting. The results are shown Table T and Table
From and Table 7 we can observe that, across all datasets and all backbones, LMN continues deliver consistent improvements over SCT under both AUROC and FPR90E, demonstrating that its effectiveness is not restricted to the CLIP-B/32 backbone and extends naturally to more advanced VLM architectures. These results confirm that LMN's surrogate objective and meta-network design retain their advantages even when the underlying visual encoder and multimodal representation mechanism differ substantially from those of CLIP-B/32.
D.5 CALIBRATION DATA SENSITIVITY ANALYSIS
To evaluate whether the effectiveness of LMN depends on the specific 16-shot calibration configura- tion used in the main experiments, we additionally conduct a sensitivity study by varying the number consider 4-shot, 8-shot, 16-shot, 32-shot, and
of calibration samples per class. Specifically, 64-shot settings on datasets with sufficiently large training sets, including DTD, EuroSAT, RESICS45, and MNIST. For Flowers 102 and CUB, the available training samples too limited support
21

---

Published as a conference paper at ICLR 2026
Table 6: MisD Performance comparison under CLIP-L/14. For each dataset, AUROC (1) and FPR90E (L) are reported.
DTD
Flowers102
EuroSAT
Method
AUROCT
FPR90-E 4
AUROC1
FPR90-E
AUROC1
FPR90-E !.
CLIP-L/14
0.789
0.521
0.887
0.289
0.730
0.605
SCT
0.808
0.433
0.903
0.203
0.787
0.497
LMN
0.843
0.373
0.942
0.132
0.862
0.351
RESICS45
MNIST
CUB
Method
AUROCT
FPR90-E 4
AUROC1 t
FPR90-E
AUROC
FPR90-E +
CLIP-L/14
0.798
0.469
0.910
0.211
0.809
0.465
SCT
0.821
0.382
0.942
0.154
0,821
0.396
LMN
0.864
0.329
0.958
0.110
0.840
0.368
Table 7: MisD Performance comparison under SigLIP-B/16. For each dataset, AUROC (f) and FPR90E (J) reported.
DTD
Flowers102
EuroSAT
Method
AUROC1 f FPR90-E |
AUROC
FPR90-E 1
AUROC1
FPR90-E +
SigLIP
0.783
0.376
0.881
0.337
0.631
0.746
SCT LMN
0.804
0.334
0.898
0.271
0.641
0.744
0.843
0.292
0.940
0.107
0.674
0.713
RESICS45
MNIST
CUB
Method
AUROC t
FPR90-E L AUROC1
FPR90-E &
AUROC f
FPR90-E |
SigLIP
0.801
0.413
0.914
0.154
0.907
0.218
SCT
0.814
0.376
0.956
0.093
0.909
0.220
LMN
0.856
0.243
0.978
0.064
0.935
0.174
larger-shot configurations beyond 16-shot, and therefore, we only report results up to 16-shot for these two datasets. The results reported in Table and Table
From the Tables, we can observe that across all datasets with adequate training data, both AUROC and FPR90E exhibit stable and progressively improving trends the number of calibration samples increases. This behavior is consistent with the design of LMN: as a lightweight meta-network, LMN naturally benefits from richer calibration signals, while its overall performance remains robust under very low-shot conditions, such as 4-shot and 8-shot. These observations indicate that LMN's effectiveness does depend on the specific 16-shot setup used in the main experiment.
D.6 ANALYZING THE COMPUTATIONAL EFFICIENCY OF LMN
To quantify the computational overhead of the proposed Lightweight Meta Network (LMN), we report the number of parameters, training time, and inference time on all six datasets in the Table TT]
First, for the number of parameters, acToss all datasets, LMN contains only 17K-20K parameters, which is less than 0.02% of the 151M parameters in CL.PP ViT-B/32. This demonstrates that LMN is extremely lightweight relative to the backbone model.
Second, for the training and test time, the results show that LMN is highly efficient in practice. All reported timings already include the one-time CLIP forward pass required to compute image and text embeddings, making the measurements fully reflect the real end-to-end cost. Training all datasets completes within one minute, with most datasets requiring only a few seconds (e.g., 7.05s on EuroSAT and 13.31s on DTD). This confirms that LMN introduces negligible computational burden during optimization. Inference similarly lightweight. Even on RESICS45, which contains over 20,000 test samples, LMN requires only 339 seconds minutes) process the entire test set.
22

---

Published as a conference paper at ICLR 2026
Table AUROC (1) under different numbers of calibration shots. + be obtained.
denotes that the result cannot
Dataset
CLIP
4-shot
8-shot
16-shot
32-shot
64-shot
DTD
0.762
0.787
0.779
0.792
0.804
0.807
Flowers102
0.864
0.873
0.884
0.886
EuroSAT
0.650
0.732
0.765
0.788
0.792
0.797
RESICS45
0.779
0.793
0.804
0.808
0.810
0,815
MNIST
0.813
0.883
0.901
0.915
0.936
0.941
CUB
0.807
0.810
0.810
0.812
Table 9: FPR90E (4) under different numbers of calibration shots. baseline.
CLIP denotes the zero-shot
Dataset
CLIP
4-shot
8-shot
16-shot
32-shot
64-shot
DTD
0.572
0.512
0.474
0.504
0.438
0.431
Hloeess022
0.354
0.329
0.317
0.305
ET
0.742
0.619
0.522
0.468
0.450
0.439
RESICS45
0.508
0.477
0.462
0.445
0.446
0.421
MNIST
0.482
0.301
0.264
0.205
0.160
0.142
CUB
0.554
0.549
0.538
0.532
For smaller datasets, inference finishes within tens seconds (e.g., 15.77s on DTD and 53.83s on FuroSAT). These results validate that LMN remains practical and efficient even when applied to large-scale test sets.
Overall, the empirical measurements demonstrate that LMN maintains its lightweight property in both training and inference, making it suitable for real-world and large-scale deployments.
D.7 EVALUATION UNDER DISTRIBUTION SHIFT
To further assess the robustness of the proposed LMN under distribution shift, conduct an additional evaluation using the ImageNet- Val to train the LMN and evaluating two distribution- shifted benchmarks (i.e., ImageNet-A and mageeNe-SSketeh Specifically, ImageNet-Sketch which introduces substantial style and texture shift, and ImageNet-A, which contains adversarially curated natural images designed to induce model failures. We report both misclasiiiiatioon-dttttiin metrics (AUROC and FPR90) and calibration metrics (Brier score) to obtain comprehensive view of the model's 's behavior under shift. The results are provided in Table
From the Table I1 across both distribution-ssiifeed datasets, indicating stronger discrimination between correct and incorrect predictions in the presence of distribution shifts. Meanwhile, LMN maintains competitive calibration performance. These findings confirm that LMN remains effective even when the input distribution deviates significantly from the calibration set, further demonstrating the practicality of the proposed approach.
observe that LMN consistently improves AUROC and reduces FPR90-E
Table 10: Training/inference time and number of LMN parameters across six datasets. LMN is highly lightweight, with only 77K200 parameters and second-level training time on all datasets.
Time (s)
DTD
Flowers102
EuroSAT RESICS45
MNIST
CUB
Train 13.31
29.63
7.05
13.14
6.81
48.35
Test
15.77
113.58
53.83
339.01
96.07
95.48
Num. Parameters
DTD
Flowers102
EuroSAT
RESICS45
MNIST
CUB
LMN
17.2K
18.8K
17.3K
18.1K
17.3K
20.3K
23

---

Published as a conference paper at ICLR 2026
Table 11: Performance under distribution shift ImageNet-A and ImageNet-Sketch.
ImageNet-A FPR90-E4
ImageNet-Skktch FPR90-F4
Method
AUROCT
Brien
AUROCT
Brier]
Zero-shot CLIP
0.657
0.740
0.814
0.805
0.495
0.732
LMN
0.682
0.716
0.796
0.813
0.484
0.721
EuroSAT
175
Density
Density
125
100
100
75
50
50
25
0.2
0.4
0.6
0.8
1.0
0.2
0.4
0.6
0.8
1.0
Confidence
Confidence
Figure 9: Histogram of test-sample confidence levels for CLIP the CUB (left) and EuroSAT (right) datasets.
D.8 ADDITIONAL EMPIRICALLY SUPPORT
We empirically examine the confidence distribution on two benchmark datasets: CUB and EuroSAT, respectively. For the test set, we record the confidence assigned by the pretrained CLIP model to each sample. We then visualize the distribution using histograms to visualize the density of each confidence level. The results are shown in Figure
Empirically, as shown in Figure 9) CLIP confidence values on test samples are widely spread across the [0, interval, with many samples falling the middle range. As result, the conditional expectations in [r, for correct prediction detection and in [0. r] for incorrect prediction detection are far below 1, confirming that strict confidence calibration alone cannot achieve high MisD performance.
LLM USAGE STATEMENT
In this work, large language model (LLM) was used polish the writing. The LLM assisted in improving clarity and grammar, but all scientific content, interpretations, and conclusions were generated solely by the authors.
24


Reviews:
[{"review_id": "nOyerWHTWw", "reviewer": "Reviewer_GAVj", "summary": "This paper addresses misclassification detection in VLMs, showing that standard confidence calibration—though improving ECE can theoretically and empirically hurt MisD. The authors analyze reliability diagrams, proving that standard calibration caps MisD precision. They introduce a new reliability curve, a normalized sigmoid, to reshape calibration toward better MisD separation, and design a differentiable surrogate loss to optimize it. Finally, they propose a lightweight post-hoc meta-network (LMN) that predicts sample-specific temperature coefficients, ensuring compatibility with pretrained CLIP models. Experiments on six datasets show MisD gains across metrics.", "questions": "1. Please include some recent MisD-oriented methods as the comparison baselines.\n\n2. Add SigLIP (e.g., SigLIP-B/16) or other popular VLMs as the backbone to show the generalization of the proposed method.\n\n3. Beyond the performance on MisD, please also include the calibration performance (ECE and Brier).\n\n4. To train a lightweight neural network, it would be interesting to show the sensitivity against data quantity of the proposed method. Also, would LMN benefit from having more data instances?\n\n5. Train/Calibrate on ImageNet-val, then evaluate MisD + calibration on ImageNet-Sketch and ImageNet-A; report AUROC/AUPR/FPR90, ECE/NLL/Brier.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The motivation of this paper is well-stated and supported with some empirical observations.\n\n2. The paper provides theoretical proofs (Appendix B) showing the new curve’s precision dominance and lower entropy.", "weaknesses": "1. The paper mostly compares against calibration (FeatureClipping, DOR) and an OOD method (SCT), while noting FSMisD and ViLU without quantitative results. This leaves open whether LMN is competitive for the actual MisD task.\n\n2. All core results use CLIP ViT-B/32 as the base model. It remains unclear whether LMN generalizes to modern VLMs.\n\n3. While the method is motivated by calibration–MisD tension and uses reliability-curve theory, the paper does not directly report calibration metrics for LMN (ECE/NLL/Brier) on the main setups.\n\n4. The paper fixes 16-shot calibration and evaluates on DTD, Flowers102, EuroSAT, RESICS45, MNIST, CUB. This is a useful starting point but too narrow to support general claims.\n\n5. Given that dealing with distribution shift is important to a calibration or MisD method, please also include setups to show the robustness of proposed method under distribution shift. For example, training on data from ImageNet-Val, and evaluate the ECE and MisD on ImageNet-Sketch and ImageNet-A etc.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5ueQuDMPU7", "reviewer": "Reviewer_jRJV", "summary": "This paper studies misclassification detection (MisD) for VLMs and argues that standard confidence calibration can hinder MisD. Specifically, the author argues that the objective of aligning confidence with accuracy is theoretically at odds with the goal of MiSD, which requires a clear ranking between correct and incorrect predictions. To address this issue, this paper proposes a recalibration objective based on a normalized sigmoid reliability curve, which is designed to maximize the separation between correct and incorrect predictions at every confidence level. Since this objective is non-differentiable, the authors introduce a differentiable surrogate loss. To preserve the original VLM's capabilities, they implement this via a post-hoc, lightweight meta-network that predicts instance-specific temperature scaling factors. Empirical experiments on multiple datasets demonstrate the effectiveness of the proposed method.", "questions": "1. In the surrogate loss (Eq. 8), the fusion function $\\phi$ can be summation or multiplication. Table 4 shows it is set to \"summation\" for most datasets but \"multiply\" for RESICS45 and CUB. What is the intuition behind this choice? Is there a clear rule or characteristic of a dataset that would guide this selection?\n2. In Table 1, the performance of the proposed method on Flowers102, RESICS45, and CUB seems marginal. Could you analyze the reason? And does this mean the effectiveness of the proposed method is dataset dependent?\n3. The meta-network uses features from both image and text modalities. Could you provide an ablation or insight to understand the relative importance of each modality's features (image embedding, text embedding, logits) for predicting the effective temperature factor?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The core argument that perfect calibration inherently limits MiSD performance is technically sound.\n2. The empirical results demonstrate the effectiveness of the proposed method.\n3. The paper is well written.", "weaknesses": "1. The parameter $\\lambda$, which controls the sharpness of the sigmoid curve, is crucial for balancing separation strength and stability. While a parameter analysis is provided, the paper could offer more concrete guidance on how to select $\\lambda$ in practice for a new dataset, rather than treating it purely as a hyperparameter to be tuned.\n2. Results focus on CLIP ViT-B/32 (plus prompt variants). Including B/16, L/14, or OpenCLIP variants, and at least one non-CLIP VLM would benefit this paper.\n3. The “lightweight” claim is plausible, but concrete numbers for training/inference overhead would still benefit the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "r1zHGvUG0E", "reviewer": "Reviewer_Mme6", "summary": "This paper investigates the connection between confidence calibration and misclassification detection (MisD), revealing that standard calibration methods limit misclassification detection (MisD) performance. The authors propose a new reliability curve designed to improve the precision of detecting both correct and incorrect predictions by better separating them. To address the non-differentiability of this new curve, a surrogate loss is introduced, and a lightweight post-hoc framework employing a meta network for sample-specific temperature scaling is developed to preserve the predictive power of Vision-Language Models (VLMs). Both theoretical analysis and experiments demonstrate that this approach consistently enhances MisD performance without sacrificing model accuracy.", "questions": "- line 32: \"easily deployed\" please remove such heuristic andcolloguial statements\n- line 35: \"the reliability of VLMs is often overlooked\" this is true for many deep learning models, please remove\n- line 36: \"has emerged as a key research direction for building reliable VLMs.\" this is true for many deep learning models, please remove\n- line 44-49: \"To address the former, ...\" such relative statements are hard to follow. Please reformulate these lines for better readability and clarity\n- page 2, Fig 1:\n    + many coloers in the plot, this hinders readability\n    + remove superfluous backgrounds (grey)\n    + remove emojis, the reader should make interpretations not the figure\n    + axis title of fig 1b, should be confidence*s*\n    + the threshold in fig 1b is very small\n    + remove superflous boxes with rounded corners (this distracts the very complicated figure)\n    + caption: remove the literal reference to concrete confidence scores, these should go into a table if important\n    + line 67-71: this is one long sentence, please reformulate\n- line 75-80: \"To remedy\" is one long sentence, please split it up for readibility\n- line 87-96: the bullet points repeat the text above, please keep only one to avoid confusion and leave space for other improvements\n- line 106: `sim` is not explained anywhere\n- line 108: Start sentence, \"Where the \"{class}\" is filled.\n- line 148: \"diagonal (y=x)\" which should be \"diagonal (acc = s)\", possibly repeat that you mean accuracy versus confidence\n- page 3: lemma 3.1 appears to be independent of `r` - if I confused this, please consider reformulating as the math suggests as much\n- page 4, equation 5: $\\lambda$ should be included in the function definion left of $\\Psi$ to indicate a dependency\n- page 5, line 254/255: \"and is more flexible than perfect calibration\" this appears to be unwarranted claim, please remove or reformulate\n- page 6, line 283-292: very hard to follow paragraph. Please consider reformulating it to highlight central ideas.\n- Table 1: no error bars or statistical treatment which elucidates competitiveness, not discussion of variability of metric results in the text too\n- line 360: \"we adopt widely used metrics\" this is jargon, please remove.\n- figure 3: only small differences visible, an uncertainty treatment would make this plot stronger\n- line 413: very general claim about universality of your approach, as the data can be consdired weak, please remove\n- table 2: no error bars or uncertainty treatment", "limitations": "", "rating": 2, "confidence": 4, "soundness": 2, "presentation": 1, "contribution": 3, "strengths": "The paper addresses an important and yet complex challenge in any classification network. The authors focus on VLMs and misclassification performance. The paper nicely combines a mathematical founded treatment and empirical experiments to validate their findings. This provides a robust backdrop for the presented scientific approach which can be followed with some investment for any reader. Moreover, the topic it addresses is of interest to practical applications of VLMs.\n\nFurther:\n- Publicly available, reproducible code.\n- Comprehensive theoretical analysis supported by empirical results.\n- Clear originality: reframing the relationship between calibration targets and separability of correct vs. incorrect predictions.\n- Consistent improvements demonstrated across datasets and settings.\n- Post-hoc design maintains the zero-shot generalization ability of VLMs while enhancing reliability.", "weaknesses": "The paper suffers from very imprecise language while presupposing a lot of knowledge from the reader. Moreover, does the style of the paper recall a blog post at many points. The mathematical treatment of the subject eliviates this to some extent as it provides a rigorous means of communication. However, especially towards the end of the report where the empirical findings are meant to support the theoretical treatment, the paper lacks scrutiny (missing uncertainty assessment of the presented results in Table 1 and 2 as well as no uncertainty estimates in figure 3 and 4). This renders the presentation of evidence weak as many of the presented performance metrics only differ by singular percentages to the competition.\n\nFurther:\n- The originality of the surrogate loss formulation is somewhat unclear. The core novelty may primarily lie in the use of a sigmoid-shaped target curve, and it would help to clarify how this extends beyond a standard activation-based reformulation.\n- The distinction between the *original calibration method* (theoretical objective) and the *post-hoc meta-network implementation* is often ambiguous, especially in the Experiments and Evaluation sections.\n- The choice of (r=0.5) in defining the new target calibration curve (Figure 2) should be theoretically or empirically motivated.\n- Some notations (especially those connecting the reliability curve to the surrogate penalty) are heavy, making it hard for readers to see how the calibration diagonal relates to the standard loss.\n- The paper does not explore implications for related uncertainty-based tasks (e.g., OOD detection) or visualize the distribution of confidence scores.\n- Key summary metrics such as **ECE** and **Accuracy** are missing from the results tables, though both are important for assessing calibration and interpreting AUROC.\n- Some conclusions slightly overstate empirical findings (e.g., \"but also empirically corroborate the theoretical analysis, validating the advantages of the normalized sigmoid reliability curve in MisD.\" - based on the calibration curve in Figure 4, which is not defined enough for me to agree that this corroborates the theoretial analysis).\n- Information about dataset size, accuracy and diversity are missing, making it hard to understand how generalizable these findings are", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4Ty0claBtv", "reviewer": "Reviewer_x3hT", "summary": "This paper addresses misclassification detection (MisD) in vision-language models like CLIP. The author first shows that the objective of confidence calibration is misaligned with MisD. Then, the authors theoretically analyze the limitations of standard calibration objectives in differentiating correct from incorrect predictions. To solve the issue, they design a differentiable surrogate loss to increase separation between correct and incorrect predictions. Extensive experiments across six datasets demonstrate that LMN can improve MisD performance compared to recent calibration or OOD detection methods.", "questions": "Questions:\n1.\tHow does LMN affect overall calibration performance?  It is unclear whether LMN compromises or preserves global calibration alignment.\n2.\tHow does DOR perform in the open-vocabulary setting? Table 2 presents open-vocabulary results for the proposed method. However, it does not compare with open-vocabulary calibration methods like DOR [4].\n\n[4] Understanding and mitigating miscalibration in prompt tuning for vision-language models.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.\tThe motivation is well-structured. The paper first clearly distinguishes the goal of confidence calibration from misclassification detection, and show that a well-calibrated model can be worse for MisD than a poorly-calibrated model.\n2.\tThe theoretical analysis is clear. The author links the area under/above the curve in the reliability diagram to the precision of detecting correct/incorrect predictions, and show that even a perfectly calibrated mode has limited MisD precision, bounded by the conditional expectation of confidence.\n3.\tThe experimental results are technologically sound and generally support the author’s claim. The author show that previous calibration can not bring significant improvement in MisD and the proposed method LMN shows consistent improvements across diverse datasets.", "weaknesses": "1.\tLack of quantitative analysis for the motivation. The paper provides a strong conceptual illustration of the failure of calibration in Figure 1. However, it lacks a quantitative experiment to directly quantify the MisD performance after calibration (e.g., TS[1] or ProximityBias [2]).\n2.\tThe connection between the proposed method and VLM is weak. The proposed LMN relies only on output logits. Consequently, the method appears applicable to any classifier rather than being inherently tied to VLMs. The author carefully discusses the role of the proposed LMN in VLM.\n3.\tMissing comparison with other failure detection methods like [3]. The experiments only compare with general calibration or OOD detection methods for VLM. \n4.\tUndefined loss notation. $L_{LDA}$ in Appendix E.2 is not defined anywhere in the paper. It looks like a typo for $L_{SUR}$.\n\n[1] On calibration of modern neural networks. ICML, 2017.\n\n[2] Proximity-informed calibration for deep neural networks. NeurlPS, 2023.\n\n[3] Rethinking confidence calibration for failure prediction. ECCV, 2022.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)