Leveraging Data to Say No: Memory Augmented Plug-and-Play Selective Prediction
Authors: Aditya Sarkar
Yi Li
Jiacheng Cheng
Shlok Kumar Mishra
Nuno Vasconcelos
Venue: ICLR 2026 Poster (2026)
Keywords: Selective Prediction
Vision Language Alignment
AI Safety
Primary area: alignment, fairness, safety, privacy, and societal considerations
TL;DR: We propose a training-free memory augmented plug-and-play selective predictor. We test it on various tasks such as classification, image-text matching and captioning.

Abstract: Selective prediction aims to endow predictors with a reject option, to avoid low confidence predictions. However, existing literature has primarily focused on closed-set tasks, such as visual question answering with predefined options or fixed-category classification. This paper considers selective  prediction for visual language foundation models, addressing a taxonomy of tasks ranging from closed to open set and from finite to unbounded vocabularies, as in image captioning. We seek training-free approaches of low-complexity, applicable to any foundation model and consider methods based on external vision-language model (VLM) embeddings, like CLIP. This is denoted as $\textit{Plug-and-Play Selective Prediction} (\textbf{\texttt{PaPSP}})$. We identify two key challenges: (1) $\textit{instability of the visual-language representations}$, leading to high variance in image-text embeddings, and (2) $\textit{poor calibration of similarity scores}$. To address these issues, we propose a $\textit{memory augmented}$ $\textbf{\texttt{PaPSP}}$ ($\textbf{\texttt{MA-PaPSP}}$) model, which augments $\textbf{\texttt{PaPSP}}$ with a retrieval dataset of image-text pairs. This is leveraged to reduce embedding variance by averaging retrieved nearest-neighbor pairs and is complemented by the use of contrastive normalization to improve score calibration. Through extensive experiments on multiple datasets, we show that $\textbf{\texttt{MA-PaPSP}}$ outperforms $\textbf{\texttt{PaPSP}}$ and other selective prediction baselines for selective captioning, image-text matching, and fine-grained classification. Source code will be made public.

Published as a conference paper at ICLR 2026
LEVERAGING DATA TO SAY No: MEMORY AUG- MENTED PLUG-AND-PLAY SELECTIVE PREDICTION
Aditya S
Li
Jiacheng Cheng2241 Shlok Mishra 1,5 Nuno Vasconcelos
Universsity of Maryland, College Park
"University of California, San Diego
Qualcoom AI
Yale University
"Meta Al
ABSTRACT
Selective prediction aims to endow predictors with reject option, avoid low confidence predictions. However, existing literature has primarily focused on closed-set tasks, such as visual question answering with predefined options or fixed-category classification. This paper considers selective prediction for vi- sual language foundation models, addressing a taxonomy of tasks ranging from closed to open set and from finite to unbounded vocabularies, in image cap- tioning. We seek training-free free approaches of low-complexity, applicable to any foundation model and consider methods based on external vision-language model embeddings, like CLIP. This is denoted as Plug-and-Plee Selective Prediction (PAPSP). We identify two key challenges: (1) instability of the visual-language representations, leading to high variance in image-text embeddings, and (2) poor calibration of similarity scores. To address these issues, we propose mem- ory augmented PAPSP (MA-PAPSP) model, which augments PAPSP with re- trieval dataset of image-text pairs. This leveraged reduce embedding vari- ance by averaging retrieved nearest-neighbor pairs and is complemented by the use of contrastive normalization to improve score calibration. Through exten- sive experiments on multiple datasets, we show that MA-PAPSP outperforms PAPSP and other selective prediction baselines for selective captioning, image- text matching, and fine- fine-grained classification. Code is publicly available at https:/ / / thub. com/ nigstnnn -ddi www.ooaaaaaaaaaaaaaaaaaaaa PMPSPP.
INTRODUCTION
The success of vision-language models (VLMs) enables wide range of promising applications. However, these models are prone to erroncous predictions for reasons that include incorrect align- ment of the modalities, image language ambiguities, samples from the tails of their train- ing distribution. This hampers their usefulness for real-world applications that require performance guarantees. As shown in Figure 1, selective prediction (SP) (Chow, 1957; 1970; EI- ganya & Wiener, 2010; Whitehead et al., 2022; Dancette et al., 2023; Geifman & BI- Yaniv, 2017; Wang & Vasconce- los, 2018; Srinivasan et al., 2024) methods address this problem by refusing make low confidence predictions, with the goal of minimizing the prediction risk on the samples that they accept. Optimal selective predictors optimize the trade-off between risk and coverage, which is the acceptance ratio throughout a dataset. While there is literature in SP, the problem has mostly been considered for closed-set prediction, usually with finite and small output vocabularies. Examples include classifi- cation with finite labels (Wang & Vasconcelos, 2018; Wu et al., 2020) or visual question answering from a finite set of choices (Wang al., 2023; Dancette et al., 2023), e.g. binary questions, questions involving a finite set of attributes, etc. However, vision-language tasks like the selective captioning problem Figure 1 require open-set prediction, where the label set is infinite. This is, in fact, the regime that most foundation models operate in. There is need for SP methods for open set tasks.
One possibility is to endow the foundation model with some prediction refusal score and train or fine-tune it to optimize the ensuing risk-coverage trade-off. However, most foundation models large and require large-scale training to avoid overfitting (Stevens et al., 2024; Gu et al., 2025). In many cases, the model may even be a proprietary black box that is impossible retrain. In this
Cooresponning author; Jiacheng Cheng.

---

Published as a conference paper at ICLR 2026
PaPSP score:
SigLIP
58.7 Accept
PaPSP
SigLIP
Proxy Emb
Contrast Score
MA-PaPSP 21.3 = Reject
LVLM
CLIP
Retrieved samples:
Captianing
Chesifiction
1. ginufit standing
Q: "Describe this image"
0: "cat" "dog" , "giraffe" A: "dog""
"A gingfic looking of fosriss.
A: "A gifet in the 200
3. "4 ginulis vailag leane..
CDeR-t: 0.15 0.15 (lw)
Accurscy: 0.om))
MA-PaPSP
external representation model and the CLIP score enable selective prediction for VLM tasks like captioning without training. MA-PAPSP augments this model with an external dataset, which is leveraged to estimate proxy embeddings of greater stability and better calibrated contrastive scores. The figure shows example where PAPSP fails but MA-PAPSI succeeds at rejecting an incorrect caption for the image shown. Also shown is the Cider-4 score between predicted and ground truth captions.
Figure I: PAPSP and MA-PAPSP overview. PAPSP
work, we investigate alternative solution, based on an external SP model that can be attached to any VLM, as illustrated in Figure 1, to compute a confidence score for the predictions of the latter. SP can then be implemented by rejecting predictions of score below a threshold. This is denoted as Plug-and-Play Selective Prediction (PAPSP) and can be implemented by using a Large Visual Language Model (LVLM), egg. Qwen (Bai al., 2023), InternVL (Zhu et al., 2025), as PAPSP model. what is called the LLM-as-judge strategy (Dong et al., 2024; Ye et al., 2025). However, LVLMs are large and have computationally expensive inference, sometimes much heavier than the VLM which makes the original predictions. Furthermore, they not calibrated for PAPSP and requires some type of training or adaptation, which can itself be expensive and induce overfitting. Ideally, the PAPSP model should be lightweight and flexible enough support any task without training.
In this work, we investigate the of visual language representation models (VLRMs), like CLIP (Radford et al., 2021) or SigLIP (Zhai et al., 2023), to implement PAPSP. This is denoted as the Selective Prediction- VLM (SP-VLM), A simple yet effective implementation, illustrated in the top branch of Figure 1, is to project the image and text produced by the foundation model into the SP-VLM embedding and reject image-text pairs of low CLIP-score (Hessel et al., 2021). We find, however that SP-VLM embeddings also suffer from calibration shortcomings, which we hypothesize to have two main causes: 1) SP-VLM representation instability, which increases the variance of image and text embeddings, compromising the reliability of similarity scores, and 2) poor calibration similarity score across the SP-VLM embedding. To overcome these, we propose the Memory Augmented PAPSP MA-PAPSP) procedure of the lower branch of Figure 1. The external SP-VLM is augmented with a retrieval dataset of image-text pairs. Methods that leverage this dataset then proposed to 1) reduce the variance of the SP-VLM representation, by using the average representation of the nearest neighbors of the query image as a praxy embedding for the latter, and 2) enhance score calibration, by using contrastive scores that normalize the similarity score with respect to alternative text predictions. MA-PAPSP can be seen as a new instantiation of memory augmented approaches recently shown successful for various applications (Nakata al., 2022; Zeng et al., 2024; Geirhos et al., 2024; Silva et al., 2024). Like these, it is training method that instead leverages an external dataset enhance SP-VLM functionality.
MA-PAPSP has several interesting properties. First, it is relatively lightweight solution, usually much lighter than an LVL.M. Second, it requires no training of either foundation model the SP- VLM. Third, it can implement SP over a taxonomy of foundation model tasks, including classifica- tion (Zhou et al., 2022a;b), image-text matching (Ma et al., 2023; Hsich et al., 2024), and captioning, that range from closed to open set and from finite to unbounded prediction vocabularies. Fourth, can be used with any foundation model. We show its effectiveness for models ranging from small VLRMs to large LVLMs. Finally, MA-PAPSP can be tailored to any application domain by simply collecting data from that domain. This usually leads to improved performance over PAPSP, as il- lustrated in Figure 1. We show this for domains ranging from COCO captioning to medical imaging
2

---

Published as a conference paper at ICLR 2026
and that good performance can be achieved even with generic datasets, like CC12M (Changpinyo et al., 2021). not directly tied to the application, for both oetn-set and closed-set SP tasks.
The contributions of this work are three-fold: 1) formulate the PAPSP problem over a taxonomy of SP tasks, ranging from closed to open set and finite to unbounded vocabularies. To our knowledge, this is the first attempt solve PAPSP for all these tasks; 2) propose the lightweight, training free MA-PAPSP method, which leverages an external dataset to improve SP- SP-VLM; and 3) we conduct an extensive evaluation demonstrating the efficacy of MA-PAPSP for selective captioning, image-text matching and classífication tasks across a range of foundation models and application domains.
RELATED WORKS
The problem of learning to reject has been studied in machine learning for over fifty years, primarily for unimodal classification (Chow, 1957; 1970; Geifman & El-Yaniv, 2017: Wang & Vasconcelos, 2018; Black et al., 2022; Srinivasan et al., 2024). It aims to design mod- els that abstain from making predictions of low confidence. Various metries and frameworks have been proposed (Geifman & El- Yaniv, 2019; Wang & Vasconcelos, 2018; De Stefano et al., 2000). While these studies focus on selective classification and multi-way visual question-anse answering, where the label set is finite, MA-PAPSP targets the more challenging setting of open-set tasks (like se- lective captioning or image-text matching) where labels are sentences and their cardinality can be unbounded.
Selective Prediction.
Image-Text Alignment Score. Selective prediction performance be improved by model training (Dancette et al., 2023). However, for foundation models, this requires large scale training and is not practical for many applications. One possibility to rely on external models, e.g. the LLM-as-judge strategy (Ye et al., 2025; Dong et al., 2024). However, these are memory and compute intensive. Recent studies have proposed using visual question-answeree ment, such as SecTRUE (Yarom et al., 2023) or VQASquare Lin et al. (2024). However, even these approaches rely on complex models, with long inference times. We show that they underperform MA-PAPSP, which has lower complexity.
to assess image-text align-
Retrieval-Augmented Scoring. Memory-augmentte pipelines, first introduced for language gen- cration (Lewis et al., 2020), have been extended to Saiionnaanguge representation learning (Iscen et al. 2023a;b; Xie et al., 2023). While they do retrieval augmentation just like MA-PAPSP, most models' pretraining is very similar to CLIPs, and as a result, they have the same limitations like CLIP unstaalll representations and uncalibrated scores. More related works discussed in D.
3 PLUG AND PLAY SELECTIVE PREDICTION
A VIM f x y maps image x X' and a text y into text sequence y Modern VLMs support a wide range tasks. In this work, we consider a taxonomy of tasks of increasing complexity, which we refer as the VLM task taxonomy in the remainder of the paper.
Level Coarse-grained Discrimination: Tasks such as classification, where f predicts a finite label set y [y
Classification-style tasks where the labels in y are very similar. For example, in image-text matching (ITM), the model chooses between two similar image captions, e.g. "A girl walking a cat." versus "A girl walking dog."
Level 2 - Fine-grained Discrimination:
Level 3 Language Production: Tasks like captioning, where unbounded, ehg set of sen-
tences in English.
3.1 SELECTIVE PREDICTION
In many applications, there is value in identifying and abstaining from making prediction errors.
Selective prediction (Chow, 1957; El-ya & Wiener, 2010; Whitchead et al., 2022) implements this by augmenting the model output with an option to abstain Ø. A selective model h. X
3

---

Published as a conference paper at ICLR 2026
Y' U {Ø} usually composes the model f :X' - ) with a selection function g gx - +0, (0,11,.00 i.e.
{ [(x)
if g(x)= =
h(x) = (f.g)(x) = @ if gx) 0 and g(x) = 1 f(x)) 2
(I)
where 1[]] is the indicator function. the selection function و decides that a prediction should be made, the model prediction S' is output, otherwise h abstains. The selection function g is imple- mented by computing score 8 : X y - for the confidence of prediction [(x) for image x, which is compared to confidence threshold R. Ideally, the score s should yield high (low) value when fx) is correct (incorrect).
Selective prediction aims to optimize the trade-off between coverage and risk (El-Yaniv & WWenne,, 21 2010). Given a set D= = as) E X' x y) of image x; and text a, (captions or class label) pairs, coverage (C) is the proportion of examples for which prediction is made while risk (R) is the average loss on the covered subset, where L : y x + IR a loss function. Mathematically, they are defined as
D Σ, ((I(x)), C(g)
R(S.9)
where C(g)
Σ g(xi), [D]
1
In this work, we use the 0-1 loss function for classification problems and (x((x),, a:) as the loss for captioning problems. It is defined as
((f())) = {
(f)(f()),))) 2 8
(3)
otherwise
where y' y + R is measure of caption similarity (e.g.CiderN (Vedantam et al., 2015), METEOR (Banerjee & Lavie, 2005)) and 3 E R a similarity threshold. We compute the Area Under the Risk-Coverage curve (AURC) (Geifman et al., 2019) for a summary of performance across different coverage levels. The lower the area, the better the model.
3.2 PLUG AND PLAY SELECTIVE PREDICTION (PAPSP)
Motivation. The tasks in the VLM task taxonomy be performed by multitude of models, including visual language representation models (VLRMs) like CLIP (Radford et al., 2021) or large visual language models (LVLLMs) like LLaVA (Liu et al., 2023a). These are frequently large and/or black-box, in which cases selective prediction training is undesirable or impossible. To address this, we investigate the design of sltg-and--lay selective prediction (PAPSP) modules that can be attached to an existing VLM to implement the score function s of (1), Idcally, the PAPSP module should support an open vocabulary, be relatively light-weight, and require no training. However, selective prediction has mostly been studied for closed-set classification problems, where can be derived from the posterior class probability estimates predicted by the classifier f. In fact, the classifier produces calibrated probability estimates (Guo et al., 2017; Cheng & Vasconcelos, 2022), the largest class probability the optimal score for the implementation of (1) (Chow, 1970).
For visual language tasks, the situation is far more complex. Because the normalization of prob- abilitics by softmax type of operations is not feasible when y is unbounded, as in captioning, is frequently impossible to obtain calibrated probability estimates for the model predictions. Un- bounded prediction also introduces ambiguities, e.g. different captions can be semantically equiv- alent or otherwise match given image, which do not appear in classification. To address this, we propose to implement PAPSP with an external CLIP-style VLRM, based on a pair of encoders of shared output space Je, An image
X' is mapped by an image encoder &ing into feature vector =0 -Oing(x) F Similarly, a caption vector w dea E Fe. The encoder pair is pre-trained with a loss function that encourages the alignment of the feature vectors and w in Fe. Note the use of the superscript to emphasize that the PAPSP model is external. The VLM making the original prediction ff can also be a Ctyl- VLM, if the task is classification. To avoid confusion, we refer to the latter as the predictive VLM (P-VLM) and the VL. used to implement PAPSP as the selective prediction VLM (LP-VLM).
y is mapped by a text encoder &ū into feature
We hypothesize that, if the text de and image dingg encoders of the VPMV differ from those of the P-VLM, erroneous prediction fxx produced by latter is unlikely induce feature vector
4

---

Published as a conference paper at ICLR 2026
Peo
Figure 2: Left: VLM problems. a) instability of representations: the representations of images (orange) and texts (blue) of the same concept can vary significantly. leading to unreliable similarity scores. b) poor calibration: distances between concepts of identical similarity (red ellipses) vary across the VLM embedding JF.. Right: PAPSP solutions: c) proxy embeddings of query q (orange) average multiple nearest neighbor representations from retricval dataset (blue) to produce a more stable representation (red), closer to the concept ground-truth (green). d) contrastive scores normalize similarity scores between image and predicted caption by those between the image and a set of hard-negatives, to ensure consistency across the space.
Variant
Query
Proxy
Type
i2tr
(xng(x)
éc (x)
cross modal
12ir
(rng())
Ping (x)
uni-modal
12tr
(xc)((
(f(x))
uni-modal
12ir
(xx(((s))
Gingg (x()))
cross-medal
Figure 3: CLIP scores hetween class lahels and im- Table 1: Variants of retrieval- based proxy embeddings ages the class.
supported by MA-PAPSP.
(fx)) close to the projection Oing (x) of the image in the feature space F of the SP-VLM.
Hence, a natural implementation of (1) is simply compute the similarity score (s) in Fe as follows,
s s(x, (x)) = cos( in (x), (x)))))
(4)
This is known as the CLIP-score (Hessel et al., 2021). 1). We denote its use in (1) as the PAPSP model and show that it has relatively weak selective prediction ability. We hypothesize that this due to two problems. The first is that the SP-VLM feature space is somewhat unstable, producing an imperfect alignment images and text. This can due either poor training or just the ambiguity of image-text alignment, since different sentences can have similar meaning and sentences with almost the same words can have very different meaning. In result, as illustrated in Figure 2 a), there can be non-trivial variability across images (orange) and texts (blue) of the same groundtruth concept and the similarity scores of two equivalent image-text pairs can be quite different. The second is that while scores computed in F effective for relative judgments of similarity. such softmax classification, they are too poorly calibrated for the absolute judgments required by the selection function g. As illustrated in Figure 2 b), the distances between an image or text and semantically equivalent examples can vary significantly depending on the location of F where they projected.
We tested these hypotheses on UCF-101 (Soomro et al., 2012). Class labels and images per class were projected into the feature space J of CLIPB/IG- The score of (4) was then computed between images and their labels. Figure shows that certain classes have high score variance, confirming the instability of the semantic representation. Some classes also exhibit much higher variance than others, confirming that the representation is poorly calibrated for certain classes or regions of F". While these problems can be ameliorated by fine-tuning the SP-VIM on the target data, this can overfit and degrade model generalization. Fine-tuning is challenging for open set problems, like captioning, where the boundaries of "application data" are not even well defined.
MEMORY AUGMENTED PAPSP (MA-PAPSP)
To improve the performance of PAPSP without training, we propose to complement the pre-trained
SP-VLM with retrieval augmentation, illustrated in Figure 4. This denoted as Memory Aug-
mented PAPSP (MA-PAPSP) and relies on a reference dataset R = {(x"". (
[R] image-text
pairs. The corresponding image-text projections (zft (eling (xf3), oh (y.) ) in _ are lever-
5

---

Published as a conference paper at ICLR 2026
Classifcation
Image Marching
Captioning model
Prony embedding
Image
Image
Encader fing
Retrieval
R
Captioner
Neighborhood
Labels y
Nx
Caption f(=)
RB
Proxy Aaaid
Encoder Pist
Repect
Contrest
Similarit
Contrasive
Figure 4: MA-PAPSP architecture (i2tr variant of Table 1). 1) VLM encoder extracts a query image embed- ding Ping (x). 2) This is used to retrieve a set NA. of image-caption pairs from retrieval set R. 3) The captions in this set are used to compute proxy embedding (f)x)) which is a weighted average the retrieved text emheddings according to (6). This serves
estimate of the ground-truth caption of (x). 4) A text emhedding (x)()) is computed for the caption predicted by (in image captioning) chosen by (in image-text matching) the VLM. 5) The MA-PAPSP score computed by computing the cosine similarity between pre- dicted, omg (x), and estimated groundtruth, (f)() captions and using (8) compute contrastive score. For image captioning, this leverages of hard-negative cantions produced hy RB approach.
aged to address the instability and calibration problems the pre-trained J using the two blocks of the figure, which are discussed next.
Proxy Embeddings. To overcome the instability of the pre-trained Fr, MA-PAPSI estimates the groundtruth embedding of a query q by a proxy embedding (q). The query can be an image text y and mod the image text modality. The proxy embedding computed in two steps. First, the K nearest neighbors of q are retrieved from R to form a neighborhood set (Nx). They are represented follows
(5) where [x) is the set of indexes of K nearest neighbors of in R. Retrieval is based on similarity measure 7(q, 2 which can be either measure of uni-modal similarity between images or between text vectors or a measure of cross-modal similarity between image and text vectors. The proxy embedding then computed by averaging neighbors according their proximity to q as follows-
(q) [(x] yff) Rli (9))
(9.2.
mod (q)
Σ
Pmod (v.
VEIN(q) (q) EjEts(a) 7(q z1t)
Note that these operations can produce text proxy (9) or an image proxy oing (q) for q. independently of whether q is an image text. Table 1 summarizes the possible variants. Proxies of the modality of the query are denoted uni-modal, otherwise they are cross-modal.
Figure (c) illustrates the computation of the proxy embedding dū... (q), where q is an image and y cross-modal similarity function that retrieves the neighborhood set Na(q) ( of neighboring texts, which are used in the averaging operation (6). This reduces the instability of the projections of the text embeddings in J and produces proxy embedding closer to the ground-truth concept than any of the retrieved texts, thereby reducing the instability problem of Figure 2 (a). Figure shows some real examples, where q is an image and captions retrieved from R per example.
Contrastive Scores. Given the greater stability of the proxy representations, the simple replace- ment of the PAPSP similarity score of (4) by a score based on either the image-based proxy di (x)
' (x, (x)) cōs (x), (a(())))))
8 (x, f(x)) (0s() de (f())))))
(7)

---

Published as a conference paper at ICLR 2026
can da prppreee ALX
thep pirpery cade
Aul seaday hox
of oven
Aomaaa tiising sedta bokk ciess bask
SiJPP score percentie e- 44.25 Mi-PatS7 scone percentile- 72.00 Cidan- score 2.8/3
SigUP score percentlle -3.83 M6-F soore pertertie Liden-i score 1.056
Figure 5: Qualitative results. Each example shows an image, predicted and ground-truth captions, retrieved captions with normalized weights, CLIP and MA- MA-PAPSP scores, and the ground-truth score (CIDEr-N), Left: CLIP accepts, MA-PAPSP rejects; Right: CLIP rejects, MA-PAPSP accepts.
or the text-based proxy oLg (x) should improve the seleetive prediction performance of the SP-VLM. This, however, does not address the poor calibration of the pretrained FF, illustrated in Figure 2 (b). The problem that due to local variations of the geometry of this space, the cosine similarities between a feature vector (orange dot) and its neighbors of fixed semantic similarity (red ellipses) vary depending on the F location. To overcome this, we propose to implement MA-PAPSP with text-based contrastive score follows-
(x
s [x), ΣyNeS(S(o) exp(sP(xx yx)/T)
(8)
where (x)( is a set of hard-negative captions produced from the P-VLM prediction f(x). In implementation, these are obtained by replacing the nouns in f(xx with different set of nouns, which significantly alter caption meaning. For this, we either used a small language model (SLM) or rule based approach (RB) ensure constructing meaningful sentence that is very different from the main caption. This operation produces set of close neighbors of fxx) with varying degrees of semantic similarity. The normalization of (8) by the scores of these neighbors results two scores in [0. 1], which have much more uniform magnitudes across the SP-VLM embedding F than the non-contrastive scores of (7), as illustrated in Figure (d). This similar to the stronger homo- geneity of softmax outputs than input logits for any classifier. Rule-based approach is discussed in supplementary section C, and the SLM prompt used in our experiments for generating negatives is discussed in E. Note that bard-negatives are not needed for the classification and ITM tasks, since these already define a set of alternative labels.
Figure 4 summarizes the proposed MA-PAPSD architecture. Given image x, a P-VLM produces caption fx). The MA-PAPSP selective prediction model starts by mapping these into a pair of feature vectors, oling (x) and text (fx(((()) using the encoder pair of the SP-VLM. Either the image or text-based feature vector can then be used as query q. This is used to retrieve set of K image text-image pairs from retrieval dataset R. which are used to generate a proxy embedding for q with (6). This can be an embedding of. based on the retrieved texts, or an embedding &ing (q) based on the retrieved images. The proxy embeddings are then used to compare a similarity score between image x and caption fx). This can be implemented with the base scores of (7) or the contrastive score of (8), which rely on a set of hard negative variants of the caption f (x) produced by RB, for captioning, or set of class labels (wx)) k= for classification.
5 EXPERIMENTS
In this section, we report on selective prediction experiments for three tasks representative of the different levels of the VLM task taxonomy: classification, image matching, and captioning.
Retrieval Set. We considered three types of retrieval sets: 1) in-domain retrieval set of training sam- ples of the evaluation dataset, e.g. training split of MS-COCO (110K); 2) out-of-domain retrieval set combining Conceptual Caption CC12M and CC3M (Changpinyo et al., 2021) and SBU-1M (Or- donez al., 2011), which have overlap with any of the evaluation datasets, captioned using BLIP-2; 3) a mixed dataset concatenating the two above. all cases, there is no overlap between evaluation and retrieval sets. Out-of-domain was used for retrieval unless otherwise specified.
7

---

Published as a conference paper at ICLR 2026
Captioning
Classification Pers
ITM
MS-COCO
30K
Methods
Flowers
UCFIO1 SugarCrepe WinoGround What's sUp VI.-Checklis
Foil
Cider-N Meteor
Cider-N Metcor
VQAScore
0.146
0.321
0.211
0.207
0.217
0.240
0.236
0.226
0.217
SecTRUE
0.158
0.316
0,251
0,411
0.214
0.213
0.171
0.153
0.242
0.236
0.234
0.223
PAPSP
0.142
0.317
0,237
0.311
0.093
0.211
0.154
0.162
0.258
0.262
0.258
0.245
MA-PAPSP-S
0.121
0.312
0.235
0.310
0.077
0.171
0.116
0.079
0.228
0.214
0.210
0.213
Gain (S)
14.78
1.57
0.84
3.25
17.20
18.95
24.67
51.23
3,00
18.32
18.60
13:06
PAPSP-L
0.136
0.311
0.229
0.307
0.074
0.169
0.113
0.078
0.208
0.202
0.204
0.197
MA-PAPSP-L
0,109
0.286
0.219
0.297
0.063
0.114
0.088
0.062
0.196
0.192
0.192
0.189
Gain (L) 19.85
8,03
4.36
3.25
14.86
32.52
22.12
20.51
5.76
4,90
5.88
4.06
Table 2: Selective prediction AURC (L). denotes the use of small SP-VI.M (Sgl.IPpnee) while I. denotes the larger SigT.IP 50-100M- 'Gain" is the % improvement of MA-PAPSP over PAPSP. "Gain (B)' shows the same for SigLIPRI6.
Predictive- VLM. For classification and ITM, we compared P-VLMs of different different sizes (base and large). We present results for SigLIP (Zhai al., 2023) in the paper and for other models in supplementary section C. For captioning, we mostly used BLIP w/ ViT-B (Li et al., 2022). but also present results for Intern VL (Zhu al., 2025) and Qwen-2.5-VL (Bai al., 2023). RB approach used in our experiments is described in C.
We compare various SP-VLMs of different sizes and present results for SigLIP (Zhai et al., 2023) in the paper and other models in supplementary section C. PAPSP (X) denotes the implementation of PAPSP with SP-VLM X. X is omitted for the default SigLIPssoaooa
Selective Prediction-VLN
Metries. Performance is evaluated with the risk and coverage measures of section 3. For captioning, the threshold 3 of (3) was set to 0.6 and 0.24 for Cider-N (N 4 for all the experiments) and METEOR respectively. These thresholds were obtained empirically. Fig. 5 shows examples of captions that are above/below these thresholds for different metrics.
Baselines. PAPSP is compared to two Image-Text Alignment (ITA) scoring methods: VQASquare (Lin et al., 2024) and SeeTRUE (Yarom et al., 2023). More comparisons can be found supple- mentary section B.
5.1
MAIN RESULTS.
We performed many experiments to evaluate the effectiveness of MA-PAPSP. Due to space limita- NNx Net and retrieval set R. proxy tions, we present most of the ablations (size of the neighborhood set
embedding variants of Table inference latency, design choices, PAPSP with finetuned CLIP) on the supplemental sections B and C. In this section, we ablate the components of the MA-PAPSP architecture and present comparisons to competing techniques. MA-PAPSP is implemented with the image-to-text retrieval (i2tr) variant of Table I and the contrastive score of (8).
Selective captioning, ITM, and Classification. Table 2 summarizes performance for tasks in the three levels of the VLM task taxonomy. Several conclusions can be drawn. First, the VQAScore and SecTRUE baselines underperformed even the implementation of MA-PAPSP with the smallest SP-VLM we tried (MA-PAPSP-S) This happens despite the fact that VQAScore and SeeTRUE rely on multiple LLM inferences to produce the selective prediction score and are thus much more computationally expensive. These results demonstrate the effectiveness of MA-PAPSP. Second, MA-PAPSP significantly outperforms PAPSP for equal SP-VLM (% gains shown in green). For example, with small SP-VLM, on 7 of the 10 datasets at least one metric improves by x 15% or more. In fact, MA-PAPSP using small SP-VIMM (SigLIP 16M parameters) outperforms PAPSP with a much larger SP-V SP-VLM (SigLIP S0-4DUM 1B parameters) on all captioning and classifi- cation tasks. Third, both PAPSP and MA-PAPSI benefit from larger SP- VLMs, which usually have a more stable representation. Note that the gains of MA-PAPSP over PAPSP are larger for the larger model (L) on most captioning and classification tasks. This suggests that representation instability and, consequently, the effectiveness of MA-PAPST hold independently of the model size. Fourth, the overall gains of the best version of MA-PAPS (L) over the best LVLM baseline (VQAScore) very significant in all tasks. Finally, comparing the performance differences across tasks, it can
8

---

Published as a conference paper at ICLR 2026
Captioning
Classification
ITM
MS-COCO
Hlcck-30KK Cider-N Metcor
Retrieval Set
Cider-N Metcor
Flowers
Pets
UCF-101
SugarCrepe Winoground What'sUp VL-checklist
Foil
Random
0.234
0.325
0.062
0.228
0.232
0.307
In-Domain 0,126
0.288
0.229
0.307
0.062
0.076
0.078
0.066
0.192
0.189
0.207
0.167
Out-of-Domain
0.109
0.286
0.219
0.297
0.063
0.114
0.088
0.062
0.196
0.192
0.192
0.189
Mixed 0.107
0.282
0.219
0.296
0.062
0.078
0,084
0.068
0.192
0.179
0.177
0.154
Table 3: Impact of retrieval set type the AURC (1) of MA-PAPSP for captioning, classification, and ITM. Random dataset: MS-COCO for captioning. Flowers for classification, and SugarCrepe for ITM. Blue denotes improved performance of out-of domain vs in-domain and boldface the best results.
CheXpert Cider N Meteor
MIMIC-CXR Cider-N Meleor
SP-VLM
MS ASS-COCO
FOK
P.V.M
Model
Cider-N Meteor
Cider-N Meteor
PAPSP
BLIP-1 (B.
PAPSP
0.138
0.320
0.233
0.313
SigL 0.264
0.313
0.2272
0.368
MA-PAPSP
0.114
0,286
0.211
0.289
BioMedCLLPP
0.164
0.238
0.175
0.229
BLIP-2 (2.78)
0.136
0.311
0.229
0.307
MA-PAPSP
MA-PAPSP
0.109
0.286
0.219
0.297
Out-of-
[SigI.PPso.oon
0.225
0.326
|0.275
0.276
V1.e (4B)
(4B)
MA-PAPSP
PAPSP
0.068
0.106
0.206
0.152
0.089
0.119
0.207
0.165
CheXpert MIMIC-CXR
SigL.P
0.137
0.276
0.155
0.326
SigI.IPso.400
0.134
0.215
0.143
0.292
(77
PAPSP
0.102
0.204
0.106
0.168
Mixed
SigLIPso-woo
0.136
0.217
0.132
0.272
MA-PAPSP
0.066
0.162
0.092
0.151
CheXpert MIMIC-CXR
BioMedCI 0.126
0.198
0.216
BioMedCLIP
0.138
0.208
0.124
0.202
Table 4:
Selective captioning AURC (4) for different P- VLMs. In all cases, the SP-VLM is SigLiPsosnmirr
Mixed
BioMedCL.P 0.1
0.204
0.138
0.268
Table 5: AURC (4) for biomedical models and datasets.
be seen that MA-PAPSP has the largest gains for classification and captioning. For ITM, the gains over PAPSP are large on SugarCrepe but less pronounced on the remaining datatsets.
Impact of the Retrieval Dataset. Table 3 shows how the type of retrieval dataset impacts MA- PAPSP performance. Beyond the in-domain, out-of-domain and mixed datasets, we also present results for a random dataset, which is one of the task datasets. Comparing the random with in- domain results shows that the training dataset of domain (e.g. Flowers classification) has good performance for that domain (e.g. Flowers classification) but weaker performance for the remain- ing domains (e.g. Pets classification). Comparing the in-domain and out-of-domain results, it can be seen that the out-of-domain dataset overcomes this problem. For captioning, out-of-domain re- trieval outperforms in-domain for all datasets. For ITM, which combines language and fine-grained discrimination, the two dataset categories have similar results. This shows that relying on a larger generic (fut-of-domain) dataset (15M samples) can outperform or achieve similar performance to smaller in-domain dataset (e.g. 10K MS-COCO samples), for most language tasks. It is only for classification, very fine-grained task that requires almost no language modeling, that the in-domain dataset is usually superior. However, the out-of-doman dataset is vastly superior to small domain specific dataset mismatched to the target domain (see random for Pets and UCF-101). Finally, the mixed dataset achieves the best of both worlds, obtaining the best results for almost all tasks and datasets. Overall, can be inferred that MA-PAPSP performance depends on both size and domain coverage of the retrieval set. A large generic dataset performs quite well, but can be outperformed by smaller retrieval sets covering the target domain, especially for classification. If the retrieval set is small and does not cover the target domain, performance degrades. We note that the choice of retrieval set is more of a problem for open-set problems like captioning and ITM. For classification, found that small number of samples (15) suffices.
Generalization across P-VLMs. Table 4 compares the selective captioning performance of PAPSP and MA-PAPSP for various P-VLMs, including BLIP and two LVLM models. Two con- clusions can be inferred. First, for both methods, performance improves with P-VLM model size. This is because the captions generated by larger models LLMs are closer to ground truth captions. Second, while MA-PAPSP outperforms PAPSP irrespective of P- P-L size, the gains increase for should be effective for even larger models.
the larger P-VL.s. This suggests that
9

---

Published as a conference paper at ICLR 2026
Captioning
Classffiation
ITM
Proxy Contrastive
MS-COCO
Flickr-30K
Flowers UCF-101
Pots
SugarCrepe WineGround
What'sUp
VL-Check
Foil
x
0.160
0.243
0.172
0.143
0.178
0.204
0.237
0.231
0.232
0.242
0.147
0.174
0.231
0.232
0.156
0.241
0.121
0,142
0.136
0.082
0.199
0.202
0.206
0.198
0.109
0.219
0.063
0.114
0.088
0.062
0.196
0.192
0.192
0.189
Table 6: AURC (+) ablation the impact of proxy embedding and contrastive scores on MA-PAPSP. all
cases, the SP-V VLM is SigLiPsoosoorr 50-400M
Generalization to Specialized Domains. To test the flexibility of MA-PAPSP, we considered the worst-case scenario of applying it to very specialized domains for which neither the P-VLM or the SP-VLM specifically trained. These experiments addressed selective captioning on the med- ical datasets CheXpert-5x42 5x200 (Irvin et al., 2019) and MIMIC-CXR-5X22 using Qwen-2.5-VL (7B) as P-VLM. We compared generic versions of PAPSP and MA-PAPSP SSggLIPso-smn and the out-of-domain dataset) to implementations of PAPSP and MA-PAPSP with a specialized SP.V BioMedCLIP (Zhang et al., 2023), and three retrieval sets: CheXpert, MIMIC-CXR, and a mixed dataset combining the above plus ROCO (Rückert et al., 2024), PadCh- est (de Castro et al., 2025), COVID (Shuja et al., 2020) and RSNA (Colak et al., 202 1). Ground truth captions were created with template: "An image of {class label)". ". The results of table support the following conclusions. First, for comparable SP-VLE SP. PAPSP was comparable to MA-PAPSP with the out-of-domain dataset but much weaker than MA-PAPSI with any of the specialized datasets. In fact, starting from generic PAPSP (SigL.IPs--ooo/ ). is more beneficial to maintain the SP-VLM and use MA-PAPSI with specialized datasets than implement PAPSP with the specialized SP- VLM (BioMedCLIP). This confirms the power and flexibility of MA-PAPSP. Second, with the specialized SP-VLM (BioMedCLIP) all specialized retrieval sets achieved results similar the best obtained on domains like MS-COCO. This illustrates how MAPPAPSP can leverage models and data that are available already to cover vast spectrum of domains without training.
(Johnson et al., 2019),
Proxy estimation and Contrastive components. Table 6 presents an ablation study of the two main components MA-PAPSP. Starting from the PAPSP model, both proxy estimation and con- trastive scores improve AURC performance and the combination of the two provides an additional gain. This evidence in support of the two hypothesis of Figure 2, namely the high variance of the SP-VLM representation and its poor calibration. The gains are larger for classification, followed by ITM, and captioning, suggesting greater benefits for finer mner-grained tasks.
LIMITATIONS AND CONCLUSION
This work extends the problem of selective prediction for foundation models to open-set problems such as captioning. We proposed MA-P -APPPP a memory- eeemmer--ugmented approach that leverages an ex- ternal visual language representation model and a dataset of text-image pairs to improve embedding stability and calibration of similarity scores. This enables a lightweight solution, that requires no training and can be applied to any foundation model. Experiments on a taxonomy of visual-language tasks demonstrated the effectiveness, flexibility, and robustness of MA-PAPSP for range of foun- dation models. However, they have also shown that performance depends on the overlap between the retrieval set and the target application domain. While large generic datasets can achieve good performance, specialized datasets and even representation models may enable non non-trivial gains for specialized domain, as we have demonstrated for medical imaging. This may require some trial and error by practitioners, achieve the very best performance on specific domain.
ACKNOWLEDGMENT
This work was partially funded by NSF awards IIS-2303153 and NAIRR- 240300, the NVIDIA Academic grant, and a gift from Qualcomm. We also acknowledge the NRP Nautilus cluster, used for some of the experiments discussed above.
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng. Yang Fan, Wenbin Ge, Han, Fei Huang, et al. Qwen technical report. arXiv preprint arXiv:2309. 16609, 2023.
Satanjeev Banerjee and Alon Lavie. Meteor: An automatic metric for mt evaluation with improved correlation with human judgments. In Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization, pp. 65-72, 2005.
Emily Black, Klas Leino, and Matt Fredrikson. Selective ensembles for consistent predictions. In ICLR, 2022.
Soravit Changpinyo, Piyush Sharma, Nan Ding, and Radu Soricut. Conceptual 12m: Pushing web- scale image-text pre-training to recognize long-tail visual concepts. In CVPR, 2021.
Jiacheng Cheng and Nuno Vasconcelos. Calibrating deep neural networks by pairwise constraints. In CVPR, 2022.
C. Chow. On optimum recognition emror and reject tradeoff. IEEE Transactions on information theory (TIT), 16(1):41-46, 1970.
Chi-Keung Chow. An optimum character recognition system using decision functions. IRE Trans- actions on Electronic Computers, 4:247-254, 1957.
Errol Colak, Felipe C Kitamura, Stephen B Hobbs, Carol C Wu, Matthew P Lungren, Luciano M Prevedello, Jayashree Kalpathy-Cramer Robyn Ball, George Shih, Anouk Stein, et al. The rsna pulmonary embolism ct dataset. Radiology: Artificial Intelligence, 3(2):e202244, 2021.
Corentin Dancette, Spencer Whitehead, Rishabh Maheshwary, Ramakrishna Vedantam, Stefan Scherer, Xinlei Chen, Matthieu Cord, and Marcus Rohrbach. Improving selective visual ques- tion answering by learning from your peers. In CVPR, 2023.
Daniel Coelho de Castro, Aurelia Bustos, Shruthi Bannur, Stephanie L. Hyland, Kenza Bouzid, Valverde, Lara Jaques-Pérez, Lourdes Pérez- Rodríguez, Kenji Takeda, et al. Padchest-gr: A bilingual chest x-ray dataset for grounded radiol- ogy report generation. NEJM AI, AIT:: 120, 2025.
Maria Teodora Wetscherek, Maria Dolores Sánchez-Valverde
Claudio De Stefano, Carlo Sansone, and Mario Vento. To reject or not to reject: that is the question- answer in case of neural classifiers. IEEE Transactions Systems, Man, and Cybernetics, Part C (Applications and Reviews), 30(1):84-8) 2000.
Xinpeng Ding, Nannan Wang, Shiwei Zhang, De Cheng, Xiaomeng Li, Ziyuan Huang, Mingqian Tang, and Xinbo Gao. Support-set based cross-supervision for video grounding. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 11573-11582, 2021.
Yijiang River Dong, Tiancheng Hu, and Nigel Collier. Can Ilm be personalized judge? arXiv preprint arXiv:2406.11663 2024.
Matthijs Douze, Alexandr Guzhva, Chengqi Deng, Jeff Johnson, Gergely Szilvasy, Pierre- Emmanuel Mazaré, Maria Lomeli, Lucas Hosseini, and Hervé Jégou. The faiss library. arXiv preprint arXiv:2401.08281 2024.
Ran El-Yaniv and Yair Wiener. On the foundations of noise-free selective classification. Journal of Machine Learning Research (JMLR), 11(53):1605-1641, 2010.
Yuxin Fang. Quan Sun, Xinggang Wang, Tiejun Huang. Xinlong Wang, and Yue Cao. Eva-02: A visual representation for neon genesis. Image and Vision Computing, 149:105171 2024.
Yonatan Geifman and Ran El- Yaniv. Selective classification for deep neural networks. NeurIPS, 2017.
Yonatan Geifman and Ran El- Yaniv. Selectivenet: A deep neural network with an integrated reject option. In ICMI,, 2019.
11

---

Published as a conference paper at ICLR 2026
Yonatan Geifman, Guy Uziel, and Ran El- Yaniv.. Bias-reduced uncertainty estimation for deep neural classifiers. ICLR, 2019.
Robert Geirhos, Priyank Jaini, Austin Stone, Sourabh Medapati, Xi Yi, George Toderici, Abhijit Ogale, and Jonathon Shlens. Towards flexible perception with visual memory. arXiv preprint UrXiv:2408.08/77 2024.
Dhruba Ghosh, Hannaneh Hajishirzi, and Ludwig Schmidt. Geneval: An object-focused framework for evaluating text-to-image alignment. NewrIPS, 2023.
Jianyang Gu, Samuel Stevens, Elizabeth G Campolongo, Matthew J Thhompson, Net Zhang, Jiaman
Wu, Andrei Kopanev, Zheda Mai, Alexander E White, James Balhoff, et al. Bioclip 2: Emergent properties from scaling hierarchical contrastive learning. arXiv preprini wrxiv::505522888 2025.
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. On calibration of modern neural networks. In ICML, 2017.
Ruiqi Guo, Philip Sun, Erik Lindgren, Quan Geng, David Simcha, Felix Chern, and Sanjiv Kumar. Accelerating large-scale inference with anisotropic vector quantization. In ICML, 2020.
Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. Clipscore: A reference-free evaluation metric for image captioning. arXiv preprint arXiv:2104.087/8, 2021.
Cheng-Yr Yu Hsieh, Jieyu Zhang, Zixian Ma, Aniruddha Kembhavi, and Ranjay Krishna. Sugarcrepe: Fixing hackable benchmarks for vision-language compositionality. NeurIPS, volume 36, 2024.
Gabriel Ilbarco, Mitchell Wortsman, Ross Wightman, Cade Gordon, Nicholas Carlini, Rohan Taori, Achal Dave, Vaishaal Shankar, Hongseok Namkoong, John Miller, Hannaneh Hajishirzi. Ali Farhadi, and Ludwig Schmidt. Openclip, July 2021. URL ptttps://eet org/ 10. 5281/ zenodo. 5143773
Jeremy Irvin, Pranay Rajpurkar, Michael Ko, Yifan Yu, Silviana Ciurea-llcus, Chris Chute, Henrik Marklund, Behzad Haghgoo, Robyn Ball, Katie Shpanskaya, et al. Chexpert: A large chest radiograph dataset with uncertainty labels and expert comparison. In Proceedings of the AAAI conference on artificial intelligence, volume 33, pp. 590 590-597, 2019.
Ahmet Iscen, Mathilde Caron, Alireza Fathi, and Cordelia Schmid. Retrieval-enhanced contrastive vision-text models. arXiv preprint arXiv:2306.00 30..0006076... 2023a.
Ahmet Iscen, Alireza Fathi, and Cordelia Schmid. Improving image recognition by retrieving from web-scale image-text data. In CVPR, 2023b.
Alistair EW Johnson, Tom J Pollard, Seth J Berkowitz, Nathaniel R Greenbaum, Matthew Lun- gren, Chih-ying Deng, Roger G Mark, and Steven Horng. Mimic-cxr, de-identified publicly available database of chest radiographs with free-text reports. Scientific data, 6(1):317, 2019.
Amita Kamath, Jack Hessel, and Kai- Kai-Wei Chang. What's" up" with vision-language models? in- vestigating their struggle with spatial reasoning. arXiv preprint arXiv:2310.19788. 2023.
Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al. aetrieval-augmente gener- ation for knowledge-intensig nlp tasks. Advances in neural information processing systems, 33: 2020.
Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-image pre- training for unified vision-language understanding and generation. In ICMI. 2022.
Tsung-Yi Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In ECCV, 2014.
Zhiqiu Lin, Deepak Pathak, Baiqi Li, Jiayao Li, Xide Xia, Graham Neubig, Pengchuan Zhang, and Deva Ramanan. Evaluating text-to-visual generation with image-to-text generation. European Conference Computer Vision, pp. 366-384. Springer, 2024.
12

---

Published as a conference paper at ICLR 2026
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. NeurIPS, 2023a.
Haotian Liu, Kilho Son, Jianwei Yang, Ce Liu, Jianfeng Gao, Yong Jae Lee, and Chunyuan Li. Learning customized visual models with 2023b.
knowledge. arXiv:330.000994,
Zixian Ma, Jerry Hong, Mustafa Omer Gul, Mona Gandhi, Irena Gao, and Ranjay Krishna. Crepe:
Can vision-language foundation models reason compositionally? In CVPR, 2023.
George A Miller. Wordnet: lexical database for english. Communications of the ACM, 38(11):
39-41, 1995.
Kengo Nakata, Youyang Ng. Daisuke Miyashita, Asuka Maki, Yu-Chieh Lin, and Jun Deguchi. Revisiting knn-based image classification system with high-capacity storage. In ECCV, 2022.
Maria-Elena Nilsback and Andrew Zisserman. 102 category flower dataset. Department of Engi- neering Science, University of Oxford. Retrieved March, 4:2024, 2008.
Vicente Ordonez, Girish Kulkarni, and Tamara Berg. Im2text: Describing images using million captioned photographs. NurIPS, 2011.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang. Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang. Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library. In NeurlPS, 2019.
Bryan A Plummer, Liwei Wang, Chris M Cervantes, Juan Caicedo, Julia Hockenmaier, and Svet- lana Lazebnik. Flickr30k entities: Collecting region-to-phras correspondences for richer image- to-sentence models. In Proceedings of the IEEE international conference on computer vision, pp. 2641-2649 2015.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and llya Sutskever. Learning transferable visual models from natural language supervision. In ICMI., 2021.
Mohammad Saleh Refahi, Gavin Hearne, Harrison Muller, Kieran Lynch, Bahrad A Sokhansanj, James Brown, and Gail Rosen. Fast and scalable gene embedding search: A comparative study of faiss and scann. arXiv preprint arXiv:2507.16972 2025.
Johannes Rückert, Louise Bloch, Raphael Brüngel, Ahmad Idrissi- Yaghir, Henning Schäfer, Cyn- thia S Schmidt, Sven Koitka, Obioma Pelka, Asma Ben Abacha, Alba G. Seco de Herrera, et al. Rocov2: Radiology objects in context version an updated multimodal image dataset. Scientific Data, 1181),688, 2024.
Ravi Shekhar, Sandro Pezzelle, Yauhen Klimovich, Aurelie Herbelot, Moin Nabi, Enver Sangineto, and Raffaella Bernardi. "foil it! find one mismatch between image and language caption". In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (ACL) (Volume Long Papers), pp. 255-265, 2017.
Junaid Shuja, Eisa Alanazi, Waleed Alasmary, and Abdulaziz Alashaikh. Covid-19 datasets: survey and future challenges. MedRxiv, pp. 2020-05, 2020.
Thalles Silva, Helio Pedrini, and Adín Ramírez Rivera. Learning from memory: Non-parametric memory augmented self-supervised learning of visual features. ICML, 2024.
Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah. Ucf101: A dataset 101 human actions classes from videos in the wild. arXiv preprint arXiv:1212.04400 2012.
Tejas Srinivasan, Jack Hessel, Tanmay Gupta, Bill Yuchen Lin, Yejin Choi, Jesse Thomason, and Khyathi Raghavi Chandu. Selective" selective prediction": Reducing unnecessary abstention in vision-language reasoning. arXiv preprint arXiv:2402 15610, 2024.
13

---

Published as a conference paper at ICLR 2026
Samuel Stevens, Jiaman Wu, Matthew J Thompson, Elizabeth G Campolongo, Chan Hee Song, David Edward Carlyn, Li Dong, Wasila M Dahdul, Charles Stewart, Tanya Lun Chao, and Yu Su. Bioclip: A vision foundation model for the tree of life. In CVPR, 2024.
Wolf, Wei-
Tristan Thrush, Ryan Jiang, Max Bartolo, Amanpreet Singh, Adina Williams, Douwe Kiela, and Candace Ross. Winoground: Probing vision and language models for visio-linguistic composi- tionality. CVPR, 2022.
Jeremias Traub, Till J Bungert, Carsten Lüth, Michael Baumgartner, Klaus H Maier-Hein, Lena Maier-Hein, and Paul F Jäger. Overcoming common flaws in the evaluation of selective classifi- cation systems. NeurIPS, 2024.
Ramakrishna Vedantam, C Lawrence Zitnick, and Devi Parikh. description evaluation. In CVPR, 2015.
Cider:
Consensus-based image
Hualiang Wang, Li,,, Huifeng Yao, and Xiaomeng Li. Clipn for zero-shot ood detection: Teaching clip to say no. In ICCV, 2023.
Lijie Wang, Xueting Wang, Toshihiko Yamasaki, and Kiyoharu Aizawa. - multi-patch image aesthetics score prediction. In CVPRW, 2019.
Liyunn Wang. Xingxing Zhang, Hang Su, and Jun Zhu. A comprehensive survey of continual learning: theory, method and application. IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), 2024.
Pei Wang and Nuno Vasconcelos. Towards realistic predictors. In ECCV, 2018.
Spencer Whitehead, Suzanne Petryk, Vedaad Shakib, Joseph Gonzalez, Trevor Darrell, Anna Rohrbach, and Marcus Rohrbach. Reliable visual question answering: Abstain rather than an- swer incorrectly. In ECCV, 2022.
Tz- Ying Wu, Pedro Morgado, Pei Wang, Chih-Hui Ho, and Nuno Vasconcelos. Solving long-tailed recognition with deep realistic taxonomic classifier. In ECCV, 2020.
Chen-Wei Xie, Siyang Sun, Xiong Xiong, Yun Zheng, Deli Zhao, and Jingren Zhou Ra-clip: Retrieval augmented contrastive language-image pre-training. CVPR, 2023.
Michal Yarom, Yonatan Bitton, Soravit Changpinyo, Roee Aharoni, Jonathan Herzig, Oran Lang, Eran Ofek, and Idan Szpektor. What you see is what you read? improving text-image alignment evaluation. Advances in Neural Information Processing Systems, 36:1601-1619, 2023.
Ziyi Ye, Xiangsheng Li, Qiuchi Li, Qingyao Ai, Yujia Zhou, Wei Shen, Dong Yan, and Yiqun Liu. Learning llm-as-a-judge for preference alignment. ICLR, 2025.
Zequn Zeng, Yan Xie, Hao Zhang, Chiyu Chen, Bo Chen, and Zhengjue Wang. Meacap: Memory- augmented zero-shot image captioning. In CVPR, 2024.
Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image prainigg In ICCV, 2023.
Sheng Zhang, Yanbo Xu, Naoto Usuyama, Hanwen Xu, Jaspreet Bagga, Robert Tinn, Sam Pre- ston, Rajesh Rao, Mu Wei, Naveen Valluri, et al. Biomedclip: multimodal biomedical foundation model pretrained from fifteen million scientific image-text pairs. arXiv preprint arXiv:2303.00915. 2023.
Tiancheng Zhao, Tiangi Zhang, Mingwei Zhu, Haozhan Shen, Kyusong Lee, Xiaopeng Lu, and Jianwei Yin. Vl-checklist: Evaluating pre-trained vision-language models with objects, attributes and relations. arXiv preprint arXiv:2207.0022). 2022.
Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Conditional prompt learning for vision-language models. CVPR, 2022a.
Kaiyang Zhou, Jingkang Yang. Chen Change Loy, and Ziwei Liu. Learning to prompt for vision- language models. International Journal of Computer Vision (ICV), 13019)22337-2448, 2022b.
14

---

Published as a conference paper at ICLR 2026
Jinguo Zhu, Weiyun Wang. Zhe Chen, Zhaoyang Liu, Shenglong Ye, Lixin Gu, Hao Tian, Yuchen Duan, Weijie Su, Jie Shao, et al. Internv13: Exploring advanced training and test-time recipes for open-source multimodal models. arXiv preprint arXwv:2504.1049, 2025.
15


Reviews:
[{"review_id": "dSlNGmSneM", "reviewer": "Reviewer_73qG", "summary": "This paper addresses the problem of selective prediction (SP) for VLMs, extending it to open-set tasks like image captioning where the output space is unbounded. The authors propose a training-free, \"plug-and-play\" method called Memory Augmented Plug-and-Play Selective Prediction (MA-PaPSP). This approach uses an external VLM as a \"judge\" to score the output of a primary predictor VLM. To overcome the inherent instability and poor calibration of the judge VLM's embeddings, MA-PaPSP is augmented with a retrieval dataset. This memory is leveraged to compute more stable \"proxy embeddings\" via nearest-neighbor averaging and to derive well-calibrated \"contrastive scores\" by comparing the prediction against a set of hard negatives. The primary contribution is a practical, lightweight framework for enabling VLMs to abstain from low-confidence predictions without requiring any model retraining.", "questions": "1. Could you provide a quantitative analysis of the inference costs (e.g., latency in ms/sample, memory usage) introduced by MA-PaPSP, especially when using a large retrieval set like CC12M? A comparison to the baseline inference time of the predictor VLM would be very helpful to contextualize the \"light-weight\" claim.\n2. The text distribution of captions from modern VLMs like Qwen-2.5-VL differs significantly from the web-scraped captions in CC12M. How robust is the proxy embedding calculation to this domain gap? Have you investigated whether this shift negatively impacts the performance for state-of-the-art predictor models?\n3. The proposed framework seems difficult to apply directly to multiple-choice Visual Question Answering (VQA) tasks, where the model's output is often a single character (e.g., 'A', 'B') whose text embedding is semantically meaningless. Could you clarify how the method could be adapted for such a common VLM task, or is this considered outside the scope of the current work?\n4. The decision to completely ignore the predictor VLM's internal confidence is a core design choice. Have you considered or experimented with scenarios where this signal is available (e.g., for non-black-box models)? Could combining the MA-PaPSP score with a measure of the predictor's internal confidence lead to further improvements?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The focus on creating a training-free, plug-and-play solution that is applicable to open-set tasks like captioning is a valuable direction for improving the safety and reliability of modern VLMs.\n- While the constituent components (selective prediction, memory augmentation, contrastive scoring) are not new in isolation, their combination to address open-set SP for VLMs in a training-free manner is a simple and well-motivated approach.\n- The proposed method is technically sound and is validated through a comprehensive set of experiments. The authors test their approach across a reasonable taxonomy of tasks and demonstrate clear improvements over baselines.", "weaknesses": "1. **Omission of Inference Cost Analysis:** The claim of being \"light-weight\" is not substantiated with empirical evidence. The method introduces a potentially computationally expensive k-NN search over a large-scale dataset (e.g., CC12M) and the storage requirements for the embeddings. The paper lacks analysis of latency, throughput, or memory footprint, which is a critical omission for a method proposed for practical application.\n2. **Methodological Limitations and Dependencies:**\n    1. **Performance is Fundamentally Capped by the Judge VLM:**  The method's ability to assess prediction quality is inherently limited by the semantic understanding of the external \"judge\" VLM (e.g., SigLIP). While the proposed techniques address the *structure* of the embedding space (instability, calibration), they cannot correct for the judge's fundamental semantic blind spots, such as failures in counting, fine-grained spatial reasoning, or attribute binding. This upper bound on performance is an important limitation that is not adequately discussed.\n\n    2. **Ignores Predictor's Internal Confidence:** The method's design completely disregards any internal confidence signals from the predictor VLM. While this enables the plug-and-play functionality, it also discards a potentially valuable source of information. It is possible that a hybrid approach, combining the external MA-PaPSP score with the predictor's internal state (when accessible), could yield superior performance.\n3. **Potential Fragility from Distribution Shift and Heuristics:**\n    1. There is a potential distribution shift between the captions generated by modern, powerful VLMs (often long and detailed) and the typically short, noisy captions found in web-scrapped datasets like CC12M. This mismatch could compromise the quality of the proxy embedding.\n    2. The effectiveness of the contrastive score relies heavily on the quality of the \"hard negatives.\" The paper states these are generated via a rule-based or small LM approach, which appears heuristic and may be a source of fragility. The sensitivity of the method to this generation process is not explored.\n4. **Disorganized Structure and Presentation:**\n    1. The structure of the experiment section (Section 4) hinders readability. The mixing of dataset descriptions, baseline details, main results, and ablation studies makes the paper's empirical contributions difficult to follow.\n    2. Minor but persistent presentation issues, such as very small font sizes in figures (especially Figure 2,3,4,5) hinders readability.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "b7rPNskdRc", "reviewer": "Reviewer_tRii", "summary": "This paper introduces MA-PaPSP, a lightweight and training-free method for selective prediction in vision-language models. It addresses the instability and poor calibration of standard similarity scores by an external memory and a calibration algorithm.", "questions": "Was any analysis conducted to quantify the distributional similarity between the memory banks and the target evaluation datasets? Such an analysis would greatly strengthen the claims about why certain memory banks outperform others and provide guidance for memory selection in practice.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Problem Formulation and Framework: The formulation of the Plug-and-Play Selective Prediction (PaPSP) problem for a taxonomy of VLM tasks, especially open-set scenarios like captioning, is timely and meaningful.\n2. Thorough Analysis: The paper provides a comprehensive analysis that successfully identifies and validates the core challenges of representation instability and score miscalibration in baseline methods.\n3. Strong Generalization: The proposed MA-PaPSP method demonstrates impressive generalization capabilities across a wide range of tasks including specialized domains, provided a suitable memory bank is available.", "weaknesses": "1. The paper's claim of being a \"lightweight\" solution is somewhat challenged by its reliance on a massive external memory bank (e.g., 15M image-text pairs). The computational overhead and storage cost of performing nearest-neighbor retrieval from such a large database during inference should be more thoroughly discussed. This overhead could impact the method's practical deployment in latency-sensitive or resource-constrained environments, and a comparison of inference time against baselines like LLM-as-Judge would strengthen this analysis.\n2. Dependence on Memory Quality and Practical Limitations: Model performance is highly dependent on the domain relevance of the memory bank, particularly in open-set settings. The necessity of procuring a memory distribution that aligns well with the target domain at test-time presents a significant practical challenge. This dependency could limit the method's applicability in scenarios where such curated data is unavailable.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8EM8A0fmIP", "reviewer": "Reviewer_WFx9", "summary": "This paper presents a unified framework for addressing the selective prediction problem across various vision-language tasks, including classification, retrieval, and captioning. The goal is to derive a more reliable confidence score than the vanilla CLIP similarity score. To achieve this, the method first leverages neighborhood representations for arbitrary modalities to enhance representation stability, and then applies contrastive normalization to improve calibration.", "questions": "Can the retrieval-based proxy proposed in this paper enhance open-set tasks in VLMs, such as captioning, beyond merely serving as a confidence score?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 1, "contribution": 2, "strengths": "The authors address the selective prediction problem, which aims to balance risk and coverage. This problem has been relatively underexplored in the context of vision-language models, particularly for open-set tasks such as captioning.\n\nThey propose a novel formulation that constructs neighborhood proxies using retrieval datasets, independent of the input or output modality. In addition, they present a unified framework that can be applied across different types of VLM tasks.\n\nThrough experiments, the authors further analyze the impact of retrieval datasets, which enhances the significance and credibility of their study.", "weaknesses": "The readability of this paper is unsatisfactory. Figure 1 is quite confusing; while it seems intended to illustrate the differences between classification and captioning in the proposed framework, the lines and arrows instead create misunderstanding. Figure 2 is difficult to interpret and contributes little to improving comprehension of the paper. Figure 4 is quite complex and lacks clear organization. The text in Figure 5 is too small, and the purpose of the figure itself is not clear enough.\n\nIn addition, the notations throughout the paper are generally hard to follow. Equation (6) is especially difficult to understand, possibly due to the ambiguous use of the symbols y and z. Overall, substantial revisions are needed before this work is ready for publication.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6nLlELj0UU", "reviewer": "Reviewer_8eZN", "summary": "This paper proposes MA-PaPSP, a memory-augmented plug-and-play method for selective prediction in VLMs. It aims to improve upon standard PaPSP (e.g., CLIP-Score) by addressing its representation instability and poor score calibration. The method retrieves $k$-nearest neighbors from a dataset to create a more stable \"proxy embedding\" and uses a contrastive score for better calibration.", "questions": "Conclusion: While the problem is well-motivated and the experiments are thorough, the two major weaknesses regarding the method's core premise (using a weak judge) and its technical novelty prevent a clear recommendation for acceptance.\n\nI am willing to reconsider my score if the authors can convincingly address these two points, especially by providing a strong benchmark against a powerful \"LLM-as-judge\" baseline.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "* The paper addresses a significant and practical problem: enabling training-free, plug-and-play selective prediction for foundation models, particularly for challenging open-set tasks like image captioning.\n\n* The experimental evaluation is comprehensive. The method is validated across a clear taxonomy of VLM tasks (classification, ITM, captioning) and tested with various predictive VLMs (P-VLMs) of different scales.", "weaknesses": "* SP-VLM as Bottleneck: I am skeptical about the core premise. The paper focuses on augmenting a relatively weak SP-VLM (like CLIP/SigLIP) to judge a much stronger P-VLM (like Qwen-2.5-VL). As P-VLMs advance, the SP-VLM (the \"judge\") inherently becomes the bottleneck. The proposed MA mechanism seems like a complex workaround for this weak judge. A critical missing baseline is the \"LLM-as-judge\" approach: using a state-of-the-art LVLM (e.g., Qwen3VL or an API-based model) via prompt-engineering to directly score the output.\n* Limited Novelty: The core technical contribution of the memory augmentation (MA) appears to be limited in novelty. The use of $k$-NN retrieval and weighted averaging (Eq. 6) to create a \"proxy embedding\" is a well-known ensemble and smoothing technique used to enhance robustness and accuracy. While its application to PaPSP is new, the underlying mechanism itself is not sufficiently novel.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)