Are vision language models robust to classic uncertainty challenges?
Authors: Xi Wang
Eric Nalisnick
Venue: Accepted by TMLR ()

Abstract: Robustness against uncertain and ambiguous inputs is a critical challenge for deep learning models. While recent advancements in large scale vision language models (VLMs, e.g. GPT-4o) might suggest that increasing model and training dataset size would mitigate this issue, our empirical evaluation shows a more complicated picture. In this work, we sanity check whether modern VLMs pass the two most ``classic'' uncertainty quantification challenges: Anomaly detection and classification under inherently ambiguous conditions, we find that newer and larger VLMs indeed exhibit improved robustness compared to earlier models, but still suffer from a tendency to strictly follow instructions, often causing them to hallucinate confident responses even when faced with unclear or anomalous inputs.
Remarkably, for natural images such as ImageNet, this limitation can be overcome without pipeline modifications: simply prompting models to abstain from uncertain predictions enables significant reliability gains, achieving near-perfect robustness in several settings.
However, for domain-specific tasks such as galaxy morphology classification, a lack of specialized knowledge prevents reliable uncertainty estimation. Finally, we propose a simple mechanism based on caption diversity to reveal a model’s internal uncertainty, enabling practitioners to predict when models will successfully abstain without relying on labeled data.

Published in Transactions on Machine Learning Research (02/2026)
Are vision language models robust to classic uncertainty challenges?
Xi Wang
zidulu@gmaii. com
Deportment of Computer Science Johns Hopkins University
Eric Nalisnick
malismick Gajaaa edu
Department of Computer Science
Johns Hopkins University
Reviewed on OpenReview: https: // openreview. net/ portffrmm?? 4
Abstract
Robustness against uncertain and ambignous inputs critical challenge for deep lcarning models. While recent advancements in large scale vision language models (VLMs, e.g. GPT- 40) might suggest that increasing model and training dataset size would mitigate this issue, our empirical evaluation shows more complicated picture. In this work, we sanity check whether modern VLMs pass the two most "classic" uncertainty quantification challenges: Anomaly detection and classification under inherently ambignous conditions. Our results based on non-rcasoning VLMs released between late 2024 and carly 2025 confirm that newer and larger VLMs tend to exhibit improved robustness compared to earlier models, but still suffer from a tendency to strictly follow instructions, often causing them to hallucinate confident responses even when faced with unclear or anomalous inputs. Remarkably, for natural images such as ImageNet, this limitation can be overcome without pipeline modifications: simply prompting models to abstain from uncertain predictions enables significant reliability gains, achieving near-perfect robustness in several settings. However, for domain-specifie tasks such as galaxy morphology classification, lack of specialized knowledge prevents reliable uncertainty estimation. Finally, we propose simple mechanism based on caption diversity to reveal a model's internal uncertainty, enabling practitioners to predict when models will successfully abstain without relying on labeled data.
1 Introduction
Uncertainty quantification of neural networks is an important problem widely studied by the deep learning community, especially in real-world, safety-critical settings such as self-driving cars (Bojarski et al., 2016) and medical imaging (Esteva et al., 2017). Traditional vision models often struggle with uncertainty quantification (Nixon et al., 2019; Guo et al., 2017; Minderer et al., 2021; Gal & Ghahramani, 2016; Ovadia et al., 2019), largely due to their training regime: These models are typically small-scale and trained on specific, highly curated datascts for single tasks. Consoquently, when faced with out-of-distribution (OOD) inputs, they frequently make incorrect yet highly confident predictions, posing significant risks in downstream decision- decision-making making systems.
To systematically evaluate the uncertainty quantification ability, two of the most widely adopted workload problems are inputs with corruption and OOD inputs from alternative datasets (see Sec. 2.1 for a more thorough discussion). Under corrupted imputs, such as those found in CIFAR-10C (Hendrycks Dietterich, 2019), models encounter covariate shift, Icading degraded accuracy, which case a model with good uncertainty quantification should lower its confidence accordingly, rather than maintaining overconfidence despite deteriorating performance, property known calibration (Guo et al., 2017). OOD inputs typically

---

Published in Transactions on Machine Learning Research (02/2026)
Model Task
ResNet trained on clean CIFAR-10
VLM prompted to classify inputs into CIFAR-10 categories
Test inputs contain covariate shifts Failure pattern: Model makes overconfident predictions despite degraded performance. Seen at training time: No
Challenge:
Challenge: Test imputs are inherently visually ambiguous Failure pattern: VLM fails to recognize ambiguity and responds with incorrect guesses, rather than abstaining Seen at training time: Unknown but very likely
CIFAR-IOC classification
Challenge: Test inputs contain concept shifts Failure pattern: Model faithfully classifies non-CIFAR-10 images (OOD) into CIFAR-10 categories.
Challenge: Inputs fall outside the semantic scope of CIFAR-10 categories.
CIFAR-IO v.s. not CIFAR-10
Failure pattern: VLM fails to detect anomaly inputs and instead generates hallucinated results.
Seen at training time: No
Seen at training time: Unknown but very likely
Figure 1: Classic uncertainty quantification tasks revisited in the VLMs era. Using CIFAR-10 ezample, we illustrate how corrupted inpuls and inputs from outside CIFAR-10 concepts expose different challenges and failure modes in small supervised models vs. large vision language models (VLMs, e.g. GPT4o) prompted to do classification, despite sharing the same evaluation data.
refer to imputs from domain different from the training dataset (e.g. test model trained on CIFAR-10 with SVHN digits), often having significant concept shift (i.c. entirely different label space). The goal of the evaluation is to test. whether a model can identily such inputs via flagging them as uncertain rather than forcing misclassifications.
Vision language models (VLMs, Liu et al., 2023; Gao et al., 2023; Liu et al., 2024), such as GPT4o, on the other hand, represent a paradigm shift in visual reasoning. These models leverage massive multimodal datasets and undergo self-supervised pre-training followed by extensive instruction tuning, enabling them to perform diverse tasks in zero-shot manner, using only the image and a natural language prompt at inference time. Importantly, the vast pretraining corpora makes it unclear if the covariate shift and concept shift challenge underlying the aforementioned two evaluation tasks would still hold for VLM, which raises the question: Are the traditional evaluation tasks of corruption robustness and OOD detection meaningful for VLMs?
We argue that these tasks still remain highly relevant to the problem of uncertainty quantification, albeit under new lens. While VLMs may no longer face clear-cut in-distribution v.s. out-of-distribution boundaries, they still encounter practical challenges in handling visually ambiguous inputs, or anomaly inputs that fall outside the semantic scope defined by a user prompt. These scenarios frequently arise in real-world applications and can reveal significant reliability gaps. In Fig. 1, we present illustrative examples of these challeuges, adapted for CIFAR-10 10 domain.
Therefore, in this paper, we empirically studied 6 non-reasoning VLMs' behavior on these two problems (Sec. 3.1). We begin by evaluating VLM performance corrupted ImageNet images, finding that although out-of-the-bax performance reveals notable vulnerabilities, larger and newer VLMs exhibit improved robustness (Fig. 2a). Furthermore, we show that prompting the models explicitly to 'reject ambiguous inputs' substantially enhances their reliability, enabling the models to abstain from making predictions when appropriate (Fig. 2b). Additionally, we study a classic anomaly detection setting using CIFAR-10 vs. non-CIFAR-10 images, where the goal is to determine whether models can correctly reject inputs that fall outside the specified label space, where we again find that simple prompting is sufficient for models to identify and reject anomalics. Lastly, building on these findings, we extend our analysis more domain-specific and specialized lasks, such as ECG signal classification and galaxy image recognition, where we find that without sufficient domain expertise, VLMs may show degraded or complete failure at providing reliable uncortainty quantification.
Additionally, we observe that VLM's uncertain level about an input image (Sec. 3.2) can be revealed by prompting the VLM to generate multiple captions for the input image under random sampling decoding, where VLMs tend to generate more diverse captions for visually ambiguous samples, which more likely be abstained, and vice versa. This insight allows us predict the model's ability successfully perform classification with rejection without relying on labeled ground truth.
2

---

Published in Transactions on Machine Learning Research (02/2026)
Classify the provided image into following ten categories Notice that if you find an image ambiguous and cannot confidently classify it, return "unknown" as the label.
Prompt
Image
Model
Llama 11B Qwen 2 7B Qwen 2B
(Sep 2024)
(Sep 2024)
(Feb 2025)
Qwen
(Feb 2025)
Qwen 2.5 72B (Feb 20255
GPT4o-mini
Dog
Person
Automobile
None of the above
None of the provided classes
None of the given classes apply unknown
Model
output
unknown
unknown
unknown
unknown
unknown
(a) VLMs are prompted to classify a noisy cat ImageNet-C image into CIFAR-10 categories. Black monospaced text shows the standard prompt and the corresponding output. Red text shows the appended explicit rejection prompt and the resulting output. Older and smaller models (e.g. Llama and Qwen 2.5 3B) hallucinate labels without the rejection prompt, whereas larger models often reject uncertain inputs even without explicit instructions.
w/o rejection (black prompt)
w/ rejection (black + red prompt)
Llama3.2 11B
Qwen2 7B
Qwen2.5 3B
Qwen2.5 7B
Qwen2.5 72B
GPT4o-mini
Accuracy
1.0
0.8
0.6
Rejection rate
0.5
1
1
2
5
1
3 4
5
2
3
5
1
2
4
5
3 4
5
1 2 3
5
Corruption level
Corruption level
Corruption level
Corruption level
Corruption level
Corruption level
(b) We prompt different VLMs (columns) to classify the same 1,000 samples subset of Gaussian noise-corrupted ImageNet into CIFAR-10 categories, under various corruption intensities (x-axis). Under standard prompt (black part top 2a only), the accuracy (among all samples the models output a proper label) decreases as corruption intensifies (solid lines). When models are explicitly prompted to reject ambiguous inputs (black and red parts), the accuracy for classified samples becomes significantly higher (dashed lines). The bottom row shows the corresponding rejection rates for each model: The models reject more inputs corruption levels increase.
Figure 2: VLMs show degraded performance under corrupted inputs, allowing rejection helps maintain reliability. Top row demonstrates VLMss outputs for selected sample with or without rejection prompt appended. Bottom row shows the classification accuracy under standard prompt without (solid line) vs. with (dashed line) rejection instruction prompt appended.
Over the last decade, much research effort has been spent characterizing the robustness properties of deep neural networks. Central to this research the development of corruption benchmarks, such as CIFAR-10C, which contaminates the original CIFAR-10 images with various kinds of noise. This was shown to be an enduring benchmark, with SOTA accuracy for a traditional classifier hovering around 85% for CIFAR- 10C (Wang et. al., 2 In this work, we aim to answer the historical question: are these corruption benchmarks like CIFAR-10C, as well as "OOD detection" benchmarks, now "solved"2 for modern VLMs? And if so, with what generation of VLM did they bocome "casy"? We find that VLMs have solved this benchmark only relatively recently: models released in 2025 rather robust, whereas earlier models, such Llama 3.2, still can easily fail without a proper prompt, despite presumably having seen these datasets during training. We believe our study demonstrates important lesson: Despite the clear superiority of modern, large-model Al systems, do not assume they solve older benchmarks.
This result for methods that do see / corruption during training, only clean images. When say "solved", we mean robust by default with minor modifications.
3

---

Published in Transactions on Machine Learning Research (02/2026)
Image
Prompt
Your task is to classify the given ECG signal into the normal case and the abnormal cases. Note that if the signal does NOT look like ECG signal or cannot be classified into either of the categories, return "unknown" the answer.
The given ECG signal appears to have a regular rhythm with no significant deviations from the expected pattern of a normal ECG. The waves (P, QRS, T) are present and consistent with each other, indicating that there are no signs of arrhythmia or other abnormalities.
Reasoning:
The given image appears to be a geometric figure with multiple lines intersecting at various points. However, it does not resemble any standard ECG (Electrocardiogram) signal pattem. An ECG signal typically shows a series of waves representing the electrical activity of the heart over time, which is not present in this image. Answer: unknown
Reasoning:
Answer: normal
Figure 3: Enabling the rejection option allows VLM to pick out anomaly inputs, preventing hallucination. When prompted to classify a random line image into normal v.s, abnormal ECG signal, certain VLM (Qwen 2.5 3B) would generate hallucinated results (left), potentially caused by its tendency towards strictly following instructions (Sharma et al., 2023). However, when the prompt explicitly permits rejecting non-ECG input (additional red texts in the prompt); the same model correctly identifies the input anomalous and responds with Unknown (right).
2
Background and related work
2.1
Improving models' robustness against uncertain inputs
Neural networks often struggle with input uncertainty due to their training methodology. When trained on carefully curated datasets with minimal ambiguity, models develop a tendency to produce high-confidence predictions for all inputs, regardless their clarity or relevance. This behavior creates significant reliability concerns when deploying these models in rcal-world scenarios where inputs may be ambiguous, corrupted, entirely outside the scope of the model's expertise. To systematically evaluate models' behavior under uncertain inputs, the deep learning community has identified two primary testing paradigms:
Distribution covariate shift occurs when model encounters inputs from different distribution than its training data, while maintaining the same label space. Examples include synthetic corruptions stylistic variations (ImageNet-C/PP Hendrycks & Dietterich, 2019), or alternative data collection pipelines IimageNet-V2, Recht et al., 2019). In these scenarios, models often show degraded accuracy, as these inputs are outside their capacity, while maintaining high confidence predictions. Such a dangerous combination will provide misleading signals for downstream decision-making whereas ideally, a model should elicit some signals indicating its "unsureness" and express confidence proportional to its likelihood of correctness (Guo al., 2017).
Concept shift (OOD/ /anomalll detection) typically considers anomaly inputs coming from entirely different datasets with non-overlapping label spaces than the training distribution. For instance, a digit classifier might encounter images of animals or vehicles. In such cases, the desired behavior for the model recognize the mismatch and refuse classification entirely, rather than confidently assigning inputs to irrelevant categories. This capability, often called OOD/ aanomaly detection, is crucial for safe deployment in world environments (Nalisnick et al., 2018; 2019b).
Standard deep neural networks without specialized training typically fail at both tasks. Researchers have developed numerous approaches to address these limitations. The central idea of many of these methods lies around twcaking the training loop in a way such that the model becomes capable of cliciting uncertainty information through the statistics (e.g. entropy) of the predictive distribution, such as Bayesian neural networks (MacKay, 1992; Neal, 2012; Graves, 2011; Blundell et al., 2015; Gal & Ghahramani, 2016; Maddox et al., 2019; Aitchison, 2020a;b; Daxberger et al., 2021a;b; Nalisnick et al., 2019a; Izmailov al., 2021; Wenzel et al., 2020), deep ensemble (Lalshminarayanaa al., 2017; Abe et al., 2022: D' A'Angelo & Fortuin, 2021), outlier exposure (Hendrycks et al., 2018), with which practitioners can be aware of when distribution
4

---

Published in Transactions on Machine Learning Research (02/2026)
shift / anomaly imputs shows up. There are also methods aiming at improving the models' generalization to certain distribution shift, such as test time adapiation (Wang et al., 2020; Wang & Aitchison, 2023; Schirmer et al., 2024; Nado et al., 2020; Schneider et al., 2020) for improving accuracy against corrupted inputs, and to anomaly inputs, such as open set classification (Geng et al., 2020) or meta learning (Hospodales et al., 2021).
Initially, one might think that such tasks pose little challenge for VLMss, based on two arguments:
1. No clear OOD boundary: Traditional models (e.g. ResNet trained on CIFAR-10) face clear distribution boundaries; in contrast, VLMs trained on extensive and diverse multimodal datasets inherently blur these distinctions.
2. Scaling improves uncertainty quantification: Prior observations suggest that simply scaling models and datasets can significantly mitigate uncertainty quantification problems. Indeed, prior studies highlight improvements in OOD detection (Fort et al., 2021) and calibration under distribution shift (Minderer et al., 2021) with larger-scale models.
However, upon closer investigation, these arguments do not fully hold:
1. Indeed the distribution shift challenges represented by some of the dataset, such as ImageNee-V22P may no longer be applicable for VLMs, there still exists certain real-world uncertainties, such as corruptions caused by sensor malfunctions or accidental user uploads, that can be simulated by datasets like ImageNet-CC and anomaly detection settings, that remain practically significant.
2. State-of-the-art large-scale VLMs may exhibit non-negligible issues due to their instruction-follovie nature (Sharma et al., 2023), when no special prompt is included, they frequently resort to incorrect or random predictions when faced with uncertain or anomalous inputs
Importantly, conventional methods for improving robustness, such as Bayesian neural networks or ensemble techniques, are impractical for VLMs due to the enormous computational overhead required for fine-tuning, Similarly, test-time interventions like test time adaptation or temperature scaling are difficult to implement given the black- box nature of many of these models.
Encouragingly, we discovered that the implicit uncertainty quantification capabilities embedded in VLMs, derived from their instruction-followinng behavior, offer a simple yet. effective solution. Directly prompting VLMs to reject uncertain inputs rather than forcing classification significantly enhances their robustness without necessitating architectural changes or specialized training. This implicit form of uncertainty handling capitalizes on the model's inherent ability to condition its behavior on input instructions, thus providing practitioners with a straightforward approach to improving reliability in practical deployments.
2.2 Uncertainty quantification in large language models
A large body of work investigates uncertainty quantification in large language models (LLMs), in the absence of multimodal inputs. One line of research explores the confidence score elicited by the model itself, where models are prompted to express their own certainty about their outputs (Xiong et al., 2023). Another class of methods leverages sampling baase approaches, such as measuring the number of semantic clusters among independently sampled completions to infer somantic consistency and model confidence (Kuhn et al., 2023; Nikitin et al., 2024). Lastly, statistics of the logits themselves also provide informative signals for uncertainty quantification (Kadavath et al., 2022). We hypothesize that such techniques are transferable to multimodal fasks, including those we examine. However, as shown in our experiments, prompting the model with an additional line of instruction that enables rejection is sufficient for the task we considered. Nevertheless, uncertainty
the aforementioned approaches could augment this with more explicit and mumerically estimates. To be more specific, compared with the aforementioned approaches, our suggested prompting approach works with pure black-box LLM access, requiring access to logits, nor does it require multiple independent samples from the model outputs. However, the cost of simplicity is the lack of controllability: Prompting does not allow us to adjust the level of conservatism for rejection, as we do not have access to a numeric uncertainty score to threshold on.
&Coruuteed inputs are typically seen distribution shift from clean training distribution, but the "noise inherent in the observations" (definition of aleatoric uncertainty, Kendall & Gal, 2017) also introduces ambiguous visual cues, making the corrupted inputs hard to recognize visually.
5

---

Published in Transactions on Machine Learning Research (02/2026)
There also exists line of work that "Bayesify" pre-trained foundation model through applying Bayesian inference on LoRA adapter during fine-tuning time to equip the fine-tuned model with better uncertainty quantification ability, via e.g., ensemble (Wang et al., 2023), Laplace approximation (Yang et al., 2023a), or variational inference (Wang et al., 2024c). Certainly, these approaches can be applied on VLMs to construct a "Bayesian VLM", however, the scope of these approaches is limited to fine-tuning on multiple-choice questions, and the applicability of free-form generation tasks on pre-trained models remains unclear, in contrast to the prompting approach.
2.3 Uncertainty quantification in vision language models
Several recent works have explored uncertainty quantification in the context of dision-language embedding models such as CLIP (Radford et al., 2021) and DINO (Caron et al., 2021), focusing on the quality and uncertainty of the learned representations (Miao et al., 2024; Fillioux et al., 2024; Cui et al., 2024). However, these embedding models are not the focus of our work, we are studying LlaVA (Liu et al., 2023) style full VLM that can follow natural language instructions.
In the domain of full VLMs, few studies examine their uncertainty estimation capabilities. For instance, Kostumov et al. (2024) and Groot & Valdenegro-Towe (2024) investigate the calibration of VLMSs standard Visual Question Answering (VQA) benchmarks, paralleling the calibration protocols established in the LLM literature discussed in the previous section. However, these works focus on standard benchmarks where there is limited inherent uncertainty in the inputs. Recent work Miyai et al. (2024a;b) introduce the concept of Unsolvable Problem Detection (UPD), where the goal is to determine whether given imago-question pair is unanswerable. Their Intrinsically Visual Question Detection (IVQD) lask is most similar to our anomaly detection setup, in which a mismatch between the question and image indicates an uncertain ambiguous case. However, our evaluation differs in construction: while they rely on manually annotated datasets, we automatically generate our test data in manner aligned with classic dut-o--disti (OOD) detection literatures, enabling more scalable evaluation. Moreover, their work focuses on uncertainty arising from the interplay between visual and textual modalities, our experiments additionally studies the ambiguity and uncertainty from images alone. Zhang et al. (2024) considers the interaction of VLMs with corrupted image, similar to our work, however their goal is to perform hallucination delection by checking VLMs output. variability under various corrupted versions of the same input, while work aims at evaluating VLMss robustness against corrupted inputs.
3 Methods
3.1
Evaluation of VLM's implicit uncertainty quantification ability
Our main contribution centers on assessing VLMs' ability to recognize and express uncertainty through rejecting problematic inputs, when performing image classification tasks through natural language prompting. Drawing inspiration from the literature on out-of-distribuutin robustness (Sec. 2.1), we design two complementary evaluation tasks:
Anomaly detection with prompting This task evaluates model's ability to identify inputs that fall outside the provided category definitions. We prompt the model to reject inputs that do not belong to any of the specified categories. For evaluation, we treat samples inside the given categories as the negative class and other samples as the positive class, employing standard binary classification metrics: precision (what fraction of rejected inputs are actual anomalies) and recall (what fraction of all anomalies are successfully rejected), to mcasure performance.
Classification with rejection against ambiguous inputs with prompting In this task, we present models with inputs that exhibit varying degrees of inherent ambiguity, making them potentially classifiable into multiple categories. We then explicitly prompt the model to reject inputs it finds difficult to classify into a single calegory. Our hypothesis is that as ambiguily increases, models without a rejection option will resort to random guessing, leading to increased error rates. Conversely, models with a rejection option can abstain from classification rather than make low-confidence predictions. To evaluate performance, we mcasure accuracy on the subset of non-rejected samples, with the ideal behavior being perfect accuracy after rejection, demonstrating the model's ability to recognize when it might make errors.
6

---

Published in Transactions on Machine Learning Research (02/2026)
Image
Caption
Image
Caption
The image shows red car that has been severely damaged, likely an accident...
The image shows spiral galaxy with distinct bar structure running through its center...
The image shows a red cur with significant front- end damage, likely after collision or accident...
The image shows spiral galaxy with prominent central bulge and a distinct spiral structure...
Clear edge on disk
Clear can
The image shows damaged red car with visible dents und a broken windshield...
The image depicts spiral galaxy with prominent, well-defined bar structure running through its center.
The image uppears show a street scene with viniage-style frum or trolley in the foreground...
The image depicts spiral galaxy with bright, diffuse central core and a visible disk structure.
The image depicts a colorfud and vibrant indoor setting with numerous people gathered what appears to be a lestive or celebratory atmosphere...
The image depicts spiral galaxy with prominent central bulge and disk stricture..
Ambiguous unbarred spiral
Ambiguous car
The image shows spiral galaxy with prominent central bulge and a spiral arm structure extending outward.
The image uppeurs to be pixelated or heavily distorted photograph of street seene...
(a) ImageNet-C
(b) Galaxy Zoo
Figure 4: VLMs generate diverse captions for ambiguous images. We prompt Qwen2.5 7B to "generate a description" given inpul image under different random seeds. Clean image from ImageNet receives consistent captions while its corrupted version having a diverse set of captions (top left v.s. bottom left). For the galaxy image where annotators show significant disagreement on whether there exist spiral arms (bottom right), VLMs fail to have diversity in the caption, indicating that the model does not understand the ambiguity, likely due to limited domain knowledge.
While these evaluation paradigms root in the study of out-of-distribution robustness, a concept not applicable to VLMss they still represent real-world challenges that VLMSs must overcome in practical applications. Additionally, unlike previous approaches that required model modifications or specialized training, our method for uncertainty quantification leverages the inherent capabilities of VLMs through just prompting. It is also worth noting that our method for quantification uncertainty is émplicit in that our model only has a binary option: Classify the inputs or reject, instead of cliciting a continuous score for uncertainty level.
3.2 Caption diversity for understanding the underlying mechanism of rejection
When adopting uncertainty quantification methods for non-black-box models, we typically have access to continuous score that reflects how wncerlain the model finds a given inpul to be. Common examples include the degree of disagreement among ensemble components (Abe et al., 2022), the typicality of a test input relative to the training distribution (Nalisnick et al., 2019b), or the distance of test inputs from the training data measured in a kernel space (Liu el al., 2020; Immer et al., 2021). In contrast, when VLMs are prompted to reject ambiguous inputs, the only available feedback is binary: whether the input was classified or rejected. This raises critical question, can we have an "ambiguity score" that tells us how ambiguous an image is for particular VLM, such that the higher score an input receives, the more likely it will be rejected by VLM.
We propose using caption diversity for diagnosing VLM's understanding of input ambiguity. Intuitively, when a model encounters an ambiguous image, one permitting multiple plausible interpretations, it produces a more diverse set of captions across independent generations (under random sampling decoding). Conversely, clear and unambiguous images yield consistent descriptions. To quantify this, we compute a caption diversity score by first embedding all generated captions with a sentence transformer model (all-mpnet-base- from Reimers & Gurevych, 2019), then calculating one minus the averaged pairwise cosine similarity among the embeddings. Our experiments indeed support this hypothesis (Fig. 4a and Fig. 5): As input images become more ambiguous due to higher corruption level, the overall caption diversity increases. Additionally, the images model chooses to reject consistently exhibits higher caption diversity than the classified ones.
Beyond serving as an analytical tool, caption diversity also offers a practical mechanism for predicting rejection behavior without labeled data. By examining the relationship between diversity scores and input ambiguity, practitioners can assess whether a model is likely to abstain from unreliable predictions. Notably, our experiments reveal that for specialized domains requiring expert knowledge, such as galaxy morphology,
7

---

Published in Transactions on Machine Learning Research (02/2020)
Classified
Rejected
Caption diversity
Llama3.2 11B
Qwen2 7B
Qwen2.5 3B
Qwen2.5 7B
Qwen2.5 72B
GPT4o-mini
0.25
0.26
0.25
0.20
0.100
0.20
0.30
0.20
0.15
0.15
0.25
0.24
0.15
0.075
2
5
1 Corruption level
1
3
3 Corruption level
1
2
4
2
4
2
4
Corruption level
Corruption level
Corruption level
Corruption level
Density
2.5
0.00 0.25 Diversity score
0.250.50 Diversity score
0.0
0.5
0.5 Diversity score
0.0
0.5 Diversity score
0.0
0.0
0.2 Diversity score
Diversity score
Figure 5: Caption diversity reflects model uncertainty under ambiguous inputs. We empirically verified the hypothesis from Sec. 3.2. Top: As corruption increases, caption diversity rises across all models, indicating greater uncertainly. Bollom: Rejected samples exhibit higher caption diversity than classified ones, suggesting that diversity of independently generated captions correlates with models' internal uncertainty level for an input and the tendency for rejecting it when prompted.
models generate low-diversity caplions even for ambiguous inputs, failing to recognize their own uncertainty (Fig. 4b) and leading to ineffective rejection (Fig. 6).
Scope and limitation of caption diversity Lastly, is worth noting that we do not view caption diversity as generic uncertainty estimator in that only captures the uncertainty in the inputs, as it not conditioned on the query. Therefore, it does not reflect query-dependent uncertainty, e.g, the query involves non-existent components in the image. Additionally, visually ambiguous images not the only type of input that could trigger diverse captions. When the inputs contain significant amount of information, e.g. image of a scene with hundreds of people, the model may also output diverse captions under random decoding where different random samples capture different perspectives/ details of the inputs (Chan et al., 2023).
4 Experiments
In this section, we empirically evaluate the two tasks proposed in Sec. 3.1 on the following families of VLMs: GPT4o-mint (Achiam et al., 2023), Llama 3.2 (Dubey et al., 2024), and Qwen 2/2.5 (Wang et al., 2024a; Bai et al., 2025). GPT4o-mini has a version of 2024-07-18, Llama 3.2 is released in September 2024, Qwen 2 and 2.5 are released in September 2024 and February 2025 respectively. For the Qwen2.5 72B, we use the official released AWQ quantized version.
For all experiments, we treat all models as black boxes, where we compute the metrics by looking at the output string without using the information from the logits. Unless explicitly stated, we use deterministic sampling for querying the VLMs. For experiments that use multiple generations from random decoding, we use a temperature of 0.6, a top-P of 0.95, and top-K of 50 for all models. We conduct all experiments on an internal cluster of Nvidia H100s and A100s.
The complete prompt used are shown in Appendix. A. We conduct ablation study over the prompting style (Appendix. E): Broadly, we find that the ability of rejection is sensitive to prompting style for Llama and Qwen2, but the rest models are less sensitive to the prompting mode.
4.1 Anomaly detection
We begin by examining anomaly detection tasks across two datasets:
We selected images from ImageNet (Deng et al., 2009) with concepts overlapping with CIFAR-10 10 categories as the target classification samples (detailed class mapping shown in Appendix. B), then considered images outside these concepts anomalies. We prompted the models
CIFAR-10 v.s. Not CIFAR-10:
8

---

Published in Transactions on Machine Learning Research (02/2026)
CIFAR-10 v.8. Not C-FAR-10
ECG v.s.
Not ECG
Model
Precision f
Recall f
F1 f
Precision f
Recall t
F1
Llama 3.2 11B
0.991
0.718
0.833
0.698
0.308
0.426
Qwen 2 7B
0.964
0.757
0.848
0.998
0.994
0.996
Qwen 2.5 3B
0.993
0.782
0.875
0.598
1.000
0.749
Qwen 2.5 7B
0.982
0.967
0.974
0.907
0.972
0.939
Qwen 2.5 72B
0.976
0.986
0.981
0.398
1.000
0.570
GPT4o-mini
0.964
0.974
0.969
0.360
1.000
0.529
Table I: VLMs can perform anomaly detection. Results are evaluated using precision, recall, and Fi-score across two anomaly detection tasks where anomaly inputs are considered positive cases. In both tasks, VLMS achieve high recall, successfully identifying most anomalous inputs. However, for the ECG task, models exhibit lower overall performance (lower F1), primarily due to low precision caused by over-rejection by frequently abstaining even on valid inputs.
to perform classification on target samples, i.c. those classifiable into CIFAR-10 categories, and to identify and reject the anomalous samples outside CIFAR-10 concepts. This setup is known to be challenging for OOD/anoman detection methods (Yang et al., 2023b) as the anomalous samples are visually very similar classifiable samples, since they come from the same dataset. We construct the evaluation dataset with 3,000 images in total, composed of 1,800 classifiable images and 1,200 anomaly ones, where random classifier would give precision of 0.4 and recall of 0.5.
ECG v.8. Not ECG: We used ECG signals from the PTB database (Wagner et al., 2020), treat the signals as images, prompting the VLM to classify signals as normal or abnormal, as well as identifying and rejecting anomaly inputs, which we construct with randomly generated line patterns (example shown in the top left panel of Figure 2, detailed in Appendix. C). Our evaluation dataset consists of 500 images each from the normal, abnormal, and anomaly categories, therefore, random guess baseline should have precision and recall value of 1/3
Fig. 3 illustrates representative example from the ECG vs. not ECG experiment. Without explicit instructions to reject anomaly inputs, certain VLMs (e.g., Qwen 2.5 3B) hallucinate answers when presented with random lines, attempting to force classification despite the input clearly not being an ECG. Howevor, when the prompt is augmented with a single line allowing rejection of non-ECG inputs (highlighted red in the prompt), the same model correctly identifies the input as anomalous and responds with Unknown.
The quantitative results shown in Table. 1, where evaluate the performance using precision, recall, and Fl value as discussed in Sec. 3.1. For the CIFAR-10 vs. Not CIFAR-10 lask, all models achieve high precision larger than 0.96, indicating that models are not showing "over refusal", but with varying recall rates, indicating some models (c.g. Qwen2.5 v.s. Qwen 2) can more reliably identify anomalous inputs when instructed to do so than others. For ECG vs. Not ECG task, overall performance drops significantly as indicated by the lower F1 value. Llama 3.2 11B shows limited ability to identify anomalous FCG inputs (0.308 recall), while Qwen 2 and Qwen 2.5 models demonstrate remarkable precision and recall. Interestingly, GPT4o-mini shows perfect recall but lower precision (0.360), suggesting a tendency toward over-rejecting valid ECG signals anomalous.
4.2 Classification with rejection under ambiguous inputs
For classification with rejection, consider the following two problems: Classifying corrupled ImageNet images into CIFAR-10 categories and morphological classification of Galaxy Zoo images. The reason behind choosing these two datasots is that they have ground truth ambiguity levels provided, such that we could better understand how VLM behaves as the level of uncertainty varies.
Classify ImageNet-C into CIFAR-10 categories Similar to the anomaly detection setting, we again selected some classes from ImageNet that overlaps with CIFAR-10 categories (dctailed in Appendix. B), then we used their corrupted version to introduce ambiguity in the input, we considered types of corruption: Gaussian noise, defocus blur, pixelated, all from ImageNet-C (Hendrycks & Dietterich, 2019),
9

---

Published in Transactions on Machine Learning Research (02/2026)
w/o rejection w/ rejection
Qwen2.5
Qwen2.5 7B
Qwen2.5 72B
GPT4o -mini
Qwen2.5 3B
Qwen2.5 7B
Qwen2.5 72B
GPT4o
3B
-mini
Confidence Accuracy
0.75
Rejection rate
0.25
0.50
0.00
0.9
Caption diversity
0.15
0.8
0.7
0.10
1
1
2
3
3
3
Disagree level
Disagree level
Disagree level
Disagree
Disagree level
Disagree level
Disagree level
Disagree level
level
Figure 6: Classification with rejection fails Galaxy Zoo. As annotator disagrecment level increases (higher input ambiguity), VLMss accuracy degrades, but do not elfectively utilize rejection to improve performance (top left). Caption diversity and confidence remain flat (bottom row), indicating that models fail to recognize the uncertainty in this domain, likely due to insufficient domain knowledge.
and pixmix from Hendrycks et al. (2022). For all corruption types and levels, we use the same 1,000 randomly selected images.
Classification of Galaxy Zoo images We considered Galaxy Zoo 2, a crowdsourced dataset for galaxy morphological classification. Importantly, each image comes with an annotator agroement score (leaf_prof in the metadata table), where low agreement score implies that an image ambiguous among classes due to, e.g. unclear visual features. We select 5,000 sample subset from the dataset and prompt the model to classify them into the four categories provided from Galaxy MNIST (Walmsley, 2022).
We present more details on dataset construction in Appendix. D.
ImageNett-C results We begin by looking at the VLM's behavior on images corrupted by Gaussian noise, shown in Fig. 2. Before explicit prompting VLMs to reject ambiguous samples, first look at VLM's behavior when prompted naively just lo classily the image, where we find that some models can already reject highly noised imputs (Fig. 2a) instead of hallucinating an answer, however if we look at their accuracy, we can see that it still drops significantly as corruption level increases (solid linc, Fig. 2b). Now when they are explicilly prompted to reject ambiguous samples, the accuracy among classified samples becomes much higher, indicating that the models are internally aware of the uncertainty in the input, but it needs be "activated" via prompting. We observe similar results for other corruption types shown Appendix. F. Additionally, studied caption diversity (Sec. 3.2), for each image, prompt the model to independently generate 20 captions. Broadly, we find that as the corruption level increases, the overall caption diversity increases (Fig. 5 top); additionally, rejected samples are those that receive higher caption diversity (Fig. 5 bottom, red v.s. blue lines), indicating that indced caption divorsity reflects VLMs' internal uncertainty level on input images, aligning with our hypothesis.
GalaxyZoo results However, this is not the case for Galaxy Zoo, the results are shown in Fig. 6, where see that the accuracy significantly drops ambiguity level increases, but rejection option only provides marginal improvement, indicating that the VLMs do not understand the ambiguity, such result is also predicted if we look at the caption diversity, which stays constant as annotator diagreement level increases, in contrast to the pattern for ImageNet-C images, an illustrative comparison is provided in Fig. 4. Notice that we also study just the output confidence, i.e. we prompt the model to randomly generate multiple answers and look at the maximum softmax probability of the averaged prediction vector, and again, we did not see any sign that the model aware of the ambiguity. We hypothesize that such task would roquire more domain specific knowledge with customized image embedding model (Walmsley et al., 2022; Parker al., 2024), indeed as verified by Wang et al. (2024b), general purpose CLIP model significantly underperforms
10

---

Published in Transactions on Machine Learning Research (02/2026)
fine-tuned CLIP on galaxy classification task, however building VLM with domain knowledge goes beyond the scope of this work.
Hypothesis and potential solution for GalaxyZoo's failure We hypothesize that the failure of rejection prompting on GalaxyZoo arises from the existence of high epistemic uncertainty. In particular, for natural images, there is low epistemic uncertainty; the model can be prompted to detect samples with high alcatoric uncertainty (corrupted images) and reject them. For Galaxy Zoo images, there is high epistemic uncertainty (due to lack of domain knowledge), and the model cannot. distinguish between samples of high low alcatoric uncertainty (the rejection rate stays constantly low human annotator disagreement level increases). Intuitively, this suggests that the model needs domain knowledge (low epistemic uncertainty) to understand the ambiguity in the inputs (level of aleatoric uncertainty). This also indicates that if we manage to reduce epistemic uncertainty, e.g., via prompting or few-shot examples, the model may become capable of rejecting samples of high aleatoric uncertainty. Our additional experiment results (Appendix G) verify the Teasibility: Instead of directly asking the VLM to perform galaxy classification, we can prompt. the model to answer questions in Galaxy Zoo's decision tree which docomposes galaxy classification into a sequence of simple visual questions with no domain knowledge of astrophysics needed. This can be viewed augmenting the model verbally with the knowledge of how to recognize galaxy images. As a proof of concept, we considered the first question in the decision tree (Smooth v.s. Featured galaxy), which decides whether the image falls into (smooth_round, smooth_ ciirrr) v.s. (edge_on on_disk, unbarred_ _strrel___raaa categories. Broadly, given carefully designed prompts, where we explain the concepts of the categories and rejection conditions in plain language. On instances where human annotators show high disagreement, the model rejects the instance by returning "Unknown" (Figure 10). On instances where human annotators reach a consensus, the model's answer aligns with the annotators' consensus (Figure 11). Note that we do find that the model's rejection behavior is more sensitive to the prompting design in this case compared with the ImageNet-C cases. We believe the prompt sensitivity can be reduced through fine-tuning to internalize the knowledge and conditions for rejection.
teeee//eereees...eeeeeeeeeeeeeee aadaaatgglaayzooooooooooo orrg///.ooo.o//z...... wtg/gattrees//zz.......... trees.ree.... . .htm),,
5 Conclusion
To summarize, our work revisits two classic uncertainty quantification evaluation settings: handling corrupted inputs and anomaly detection, which are challenging for small-scale models trained from scratch due to the inputs' OOD nature. While the boundary of OOD may be less clear for VLMs, we argue that these tasks still represent worldw challenges involving inherend data ambiguity and anomalousness, issues that cannot be resolved through model scaling alone. We evaluate VLMs on these two problems and find that, for standard benchmarks (c.g. ImageNet-CC and CIFAR-10 vs. Not CIFAR-10), models generally perform well through providing explicit rejection option in prompt.
However, in problems requiring specialized expert knowledge, such as galaxy classification, VLMs consistently exhibit suboptimal performance, which highlights the importance of domain-specific foundation models: These models are not only essential for achieving strong task performance but are also critical for ensuring reliability. Without a proper understanding of the input domain, models struggle to recognize and quantify uncertainty, since a good understanding of the input could be necessary prerequisite for understanding the uncertainty associated with it.
6 Limitation and future work
Evaluation on reasoning VLMs The current evaluation is limited to non-reasoning VLMs. Future work may consider how long chain of thoughts may alter the VLM's robustness, mimicking studies in the pure language domain (Kirichenko et al., 2025).
Extended experiments on Galaxy Zoo The current evaluation (Appendix G) on integrating domain knowledge via Galaxy Zoo's decision tree is limited to the proof-of-concept stage. Future work can consider further extending this setup, as well as exploring the idea of decomposing difficult tasks into tree of simpler tasks in broader range of AT4Sci problems.
11

---

Published in Transactions on Machine Learning Research (02/2026)
References
Taiga Abe, Estefany Kelly Buchanan, Geoff Pleiss, Richard Zemel, and John P Cunningham. Deep ensembles work, but are they necessary? Advances in Neural Information Processing Systems, 35:33646-33660 2022.
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Aliman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprind arXiv:2303.08774, 2023.
Laurence Aitchison. Bayesian filtering unifies adaptive and non-adaptive neural network optimization methods Advances in Neural Information Processing Systems, 33:18173 18182, 2020a.
Laurence Aitchison. A statistical theory of cold posteriors in deep neural networks. arXiv preprint arXiv:200s.0500 05911, 2020b.
Shuai Bai, Kegin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al. Qwen2. 5-vl technical report. arXiv preprint Xixu:2502..1993 2025.
Charles Blundell, Julien Cornebise, Koray Kavukcuoglu, and Daan Wierstra. Weight uncertainty in neural network. In International conference on machine learning, pp. 1613 1622. PMLR, 2015.
Mariusz Bojarski, Davide Del Testa, Daniel Dworakowski, Bernhard Firner, Beat Flepp, Prasoon Goyal, Lawrence D Jackel, Mathew Monfort, Urs Muller, Jiakai Zhang, et al. End to end learning for self-driving cars. arXiv preprint or
07316, 2016.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-superviee vision transformers. In Proceedings of the IEFE/CV international conference on computer vision, pp. 9650-9660, 2021.
David Chan, Austin Myers, Sudheendra Vijayanarasimhan, David Ross, and John Canny. Ic3: Image captioning by committee consensus. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 8975-9003, 2023.
Peng Cui, Guande He, Dan Zhang, Zhijie Deng, Yinpeng Dong, and Jun Zhu. Exploring aleatoric uncertainty in object detection via vision foundation models. arXiv preprint 2rXiv:2411.17777, 2024.
Francesco D' D'Angolo and Vincent Fortuin. Repulsive deep ensembles are bayesian. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, 2021. URL
Erik Daxberger, Agustinus Kristiadi, Alexander Immer, Runa Eschenhagen, Matthias Bauer, and Philipp Hennig. Laplace redux-effortless bayesian deep learning. Advances in Neural Information Processing Systems, 34, 2021a.
Erik Daxberger, Eric Nalisnick, James U Allingham, Javier Antorán, and José Miguel Hernández-Lobate Bayesian deep learning via subnetwork inference. In International Conference on Muchine Learning, pp- 2510-2521. PMLR, 2021b.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image databasc. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248-255. Ieco, 2009.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llaaa 3 herd of models. arXiv preprint arXiv:2407.21188, 2024.
Andre Esteva, Brett Kuprel, Roberto A Novoa, Justin Ko, Susan M Swetter, Helen M Blau, and Scbastian Thrun Dermatologist-leva classification of skin cancer with deep neural networks. nature, 542(7639): 115-118, 2017.
12

---

Published in Transactions on Machine Learning Research (02/2026)
Leo Fillioux, Julio Silva-Rodriguez, Ismail Ben Ayed, Paul-Henry Cournède, Maria Vakalopoulou, Stergios Christodoulidis, and Jose Dolz. Are foundation models for computer vision good conformal predictors? arXiv preprint arXic:2412.0000 06082, 2024.
Stanislav Fort, Jie Ren, and Balaji Lakshminarayanan. Exploring the limits of out-of-distribution detection. Advances in neural information processing systems, 34:7068-708$3 7081, 2021.
Yarin Gal and Zoubin Ghahramani. Dropout a bayesian approximation: Representing model uncertainty in deep learning. In international conference on machine learning, pp. 1050-1059. PMLR, 2016.
Peng Gao, Jiaming Han, Renrui Zhang, Ziyi Lin, Shijie Geng, Aojun Zhou, Wei Zhang, Pan Lu, Conghui He, Xiangyu Yue, et al. Llama-adapter v2: Parametar-officint visual instruction model. arXiv preprint arXiv:2304.15010, 2023.
Chnanxing Geng, Sheng-jun Huang, and Songcan Chen. Recent advances in open set recognition: A survey. IEEE transactions on pattern analysis and machine intelligence, 33(10):3614-3666 2020.
Alex Graves. Practical variational inference for neural networks. Advances in newral information processing systems, 24, 2011.
Tobias Groot and Matias Valdenegro-Tora Toro. Overconfidence is key: Verbalized uncertainty evaluation in large language and vision-language models. arXiv preprint arXiv:2405.02D0 2024.
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. On calibration modern neural networks. International conference on machine learning, pp. 1321-1330. PMLR, 2017.
Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common corruptions and perturbations. arXie preprint arXiv:1903. 12261, 2019.
Dan Hendrycks, Mantas Mazeika, and Thomas Dietterich. Deep anomaly detection with outlier exposure.
Xixiv preprint arXiv: 04606, 2018.
Dan Hendryeks, Andy Zou, Mantas Mazeika, Leonard Tang, Bo Li, Dawn Song, and Jacob Steinhardt. Pixmix: Dreamlike pictures comprehensively improve safety measures. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pottern Recognition, pp. 16783-16799 2022.
Timothy Hospedales, Antreas Antoniou, Paul Micaelli, and Amos Storkey. Meta-learning in neural networks: A survey. JEEE transactions on pattern analysis and machine intelligence, 44(9):5149-5169, 2021.
Alexander Immer, Maciej Korzepa, and Matthias Bauer. Improving predictions of bayesian neural nets via local lincarization. In International conference on artificial intelligence and statistics, pp. 703-711. PMLR, 2021.
Pavel Izmailov, Sharad Vikram, Matthew D Hoffman, and Andrew Gordon Gordon Wilson. What are bayesian neural network posteriors really like? In International Conference on Machine Learning, pp. 4629-4640. PMLR, 2021.
Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Percz, Nicholas Schiefer, Zac Halfield-Dodds, Nova DasSarma, Eli Tran-Johnson, et al. Language models (mostly) ) know what they know. arXiv preprint arXiv:2207.05221, 2022.
Alex Kendall and Yarin Gal. What uncertainties do we need in bayesian deep learning for computer vision? Advances in neurol information processing systems, 30, 2017.
Polina Kirichenko, Mark Ibrahim, Kamalika Chaudhuri, and Samuel J Bell. Abstentionbench: Reasoning Ilms fail on unanswerable questions. arXiv preprint Xrxi2:2506. 09038,, 2025.
Vasily Kostumov, Bulat Nutfullin, Oleg Pilipenko, and Eugene Ilyushin. Uncertainty- aware evaluation for vision-language models. Xi preprint arXim:2402.144. 2024.
13

---

Published in Transactions on Machine Learning Research (02/2026)
Lorenz Kuhn, Yarin Gal, and Sebastian Farquhar. Semantic uncertainty: Linguistic invariances for uncertainty estimation in natural language generation. oiv preprint. or arXiv:2302.09664 2023.
Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty ostimation using deep ensembles. Advances in neural information processing systems, 30, 2017.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. In NewrIPS, 2023.
Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jac Lec. Improved basclines with visual instruction tuning. In Proceedings of the IEEE/CVF Conference on Compuler Vision and Pollern Recognition, pp- 26296 26306, 2024.
Jeremiah Liu, Zi Lin, Shreyas Padhy, Dustin Tran, Tania Bedrax Weiss, and Balaji Lakshminarayanan. Simple and principled uncertainty estimation with deterministic deep learning via distance awareness. Advances in neural information processing systems, 1177498-5512. 2020.
David JC MacKay. A practical bayesian framework for backpropagation networks. Neural computation, 4(3): 448 472, 1992.
Wesley J Maddox, Pavel Izmailov, Timur Garipov, Dmitry P Vetrov, and Andrew Gordon Wilson. A simple baseline for bayesian uncertainty in deep learning. Advances in Neural Information Processing Systems, 32, 2019.
Yibo Miao, Yu Lei, Feng Zhou, and Zhijie Deng, Bayesian exploration of pre-trained models for low-shot image classification. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 23849 23859, 2024.
Matthias Minderer, Josip Djolonga, Rob Romijnders, Frances Hubis, Xiaohua Zhai, Neil Houlsby, Dustin Tran, and Mario Lucic. Revisiting the calibration of modern neural networks Advances in Neural Information Processing Systems, 34:15682 15694, 2021.
Atsuyuki Miyai, Jingkang Yang, Jingyang Zhang, Yifei Ming, Yueqian Lin, Qing Yu, Go Irie, Shafiq Joty, Yixuan Li, Hai Li, et al. Generalized out-of-distribution detection and beyond in vision language model era: A survey. arXiv preprint arXiv:2407.21199 2024a.
Atsuyuki Miyai, Jingkang Yang, Jingyang Zhang, Yifei Ming, Qing Yu, Go Irie, Yixuan Li, Hai Li, Ziwei Liu, and Kiyoharu Aizawa. Unsolvable problem detection: Evaluating trustworthiness of vision language models. Xixiv preprinl arXiv:2403.2033 20331, 2024b.
Zachary Nado, Shreyas Padhy, D Sculley, Alexander D' D'Amorr Balaji Lakshminarayanan and Jasper Snoek. Evaluating prediction-time batch normalization for robustness under covariate shift. arXiv preprint arXiv:2006.1000 2020.
Eric Nalisnick, Akihiro Matsukawa, Yee Whye Teh, Dilan Gorur, and Balaji Lakshminarayanan. Do deep generative models know what they don't know? arXiv preprint Xrx:::1810.000 0900996 2018
Eric Nalisnick, José Miguel Hernández- and Padhraic Smyth. Dropout as a structured shrinkage prior. In International Conference on Machine Learning, pp. 4712-4722 PMLR, 2019a.
Eric Nalisnick, Akihiro Matsukawa, Yee Whye Teh, and Balaji Lakshminarayanan. Detecting out-of- distribution inpuls to deep generative models using typicality. arXiv preprint Xrx: 990,0.299999 2019b.
Radford M. Neal. Buyesian learning for neural networks, volume 118. Springer Science & Business Media, 2012.
Alexander Nikitin, Jannik Kossen, Yarin Gal, and Pekka Marttinen, Kernel language entropy: Fino-grained uncertainty quantification for 1lms from semantic similarities. Advances in Neural Information Processing Systems, 37:8901-8922 2024.
14

---

Published in Transactions on Machine Learning Research (02/2026)
Jeremy Nixon, Michael W Dusenberry, Linchuan Zhang, Ghassen Jerfel, and Dustin Tran. Measuring calibration in deep learning. In CVPR Workshops, 2019.
Yaniv Ovadia, Emily Fertig, Jie Ren, Zachary Nado, David Sculley, Sebastian Nowozin, Joshua Dillon, Balaji Takshminarayanan, and Jasper Snoek. Can you trust your model's uncertainty? evaluating predictive uncertainty under dataset shift. Advances in neural information processing systems, 32, 2019.
Liam Parker, Francois Lanusse, Siavash Golkar, Leopoldo Sarra, Miles Cranmer, Alberto Bictti, Michael Eickenberg, Geraud Krawezik, Michael McCabe, Rudy Morel, et al. Astroclip: a cross-modal foundation model for galaxies. Monthly Notices of the Royal Astronomical Society, 531(4):4990-50) 5011,, 2024.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 8748-8763. PmLR, 2021.
Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do imagenet classifiers generalize imagenet? In International Conference on Machine Learning, pp. 5389 5400. PMLR, 2019.
Nils Reimers and Tryna Gurevych. Sentence-bert: Sentence embeddings using siamese bert-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, 11 2019. URL http:///aaxxvv.o/abbbbb/// 190b. 10084.
Mona Schirmer, Dan Zhang, and Eric Nalisnick. Test-time adaptation with stace models. In ICML 2024 Workshop on Structured Probabilistic Inference {188 Generalive Modeling, 2024.
Steffen Schneider, Evgenia Rusak, Luisa Eck, Oliver Bringmann, Wieland Brendel, and Matthias Bethge. Improving robustness against. common corruptions by covariate shift adaptation. Advances in Newral Information Processing Systems, 33:11539 11551, 2020.
Mrinank Sharma, Meg Tong, Tomasz Korbak, David Duvenaud, Amanda Askell, Samuel R Bowman, Newton Cheng, Esin Durmus, Zac Halfield-Dodds, Scott R Johnston, et al. Towards understanding sycophancy in language models. arXiv preprint arXiv:2310. 13548, 2023.
Patrick Wagner, Nils Strodthoff, Ralf-Dieter Bousseljot, Dieter Kreiseler, Fatima Lunze, Wojciech Samek, and Tobias Schaeffter. Ptb-xl, a large publicly available electrocardiograply dataset. Scientific data, 7(1): 1-15, 2020.
Martin Walmsley, Galaxymnist: Galaxy images labelled by morphology (shape). awaimmseer/galaxxxymiis 2022. Accessed: 2025-05-14.
Mike Walmsley, Inigo Val Slijepcevic, Micah Bowles, and Anna MM Scaife. Towards galaxy foundation models with hybrid contrastive learning. arXiv preprint orXiv:2206. 11927, 2022.
Dequan Wang, Evan Shelhamer, Shaoteng Liu, Bruno Olshausen, and Trevor Darrell. Tent: Fully test-time adaptation by entropy minimization. In International Conference on Learning Representations, 2020.
Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. Qwen2-vl: Enhancing vision-language model's perception of the world at any resolution. Xiv preprint 00x: 12191, 2024a.
Ruoqi Wang, Haitao Wang, and Qiong Luo. Effective fine-tuning of vision-languuge models for accurate galaxy morphology analysis. arXiv preprint 111.v:2411.19/77 2024b.
Xi Wang and Laurence Aitchison. Robustness to corruption in pre-trained bayesian neural networks. In The Eleventh International Conference on Learning Representations, 2023. URL https://openreriev.nee net/ Dermmiid=UI44nnYehi
Xi Wang, Laurence Aitchison, and Maja Rudolph. Lora ensembles for large language model fine-tuning. Xix preprint arXi:223100000 2023.
15

---

Published in Transactions on Machine Learning Research (02/2026)
Yibin Wang, Haizhou Shi, Ligong Han, Dimitris Metaxas, and Hao Wang. Blob: Bayesian low-rank adaptation by backpropagation for large language models. Advonces in neural informalion processing systems, 37: 67758 67794, 2024c.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting clicits reasoning in large language models. Advances in neural information processing systems, 35:24824 24837, 2022.
Florian Wenzel, Kevin Roth, Bastiaan S Veeling, Jakub Swiatkowski, Linh Tran, Stephan Mandt, Jasper Snoek, Tim Salimans, Rodolphe Jenatton, and Sebastian Nowozin. How good is the bayes posterior in deep neural networks really? arXiv preprint 20i:20002002455, 2020.
Miao Xiong, Zhiyuan Hu, Xinyang Lu, Yifei Li, Jie Fu, Junxian He, and Bryan Hooi. Can Ilms express their uncortainty? an empirical evaluation of confidence clicitation in llms. arXiv preprint Xiw:2306611303 2023.
Adam X Yang, Maxime Robeyns, Xi Wang, and Laurence Aitchison. Bayesian low-rank adaptation for large language models. arXiv preprint Xix: 13111, 2023a.
William Yang, Byron Zhang, and Olga Russakovsky. Imagenet-ood: Deciphering modern out-of-distribution detection algorithms. arXiw preprint 3rXiv:2310.01755 2023b.
Ruiyang Zhang, Hu Zhang, and Zhedong Zheng. Vl-uncertainty; Detecting hallucination in large vision- language model via uncertainty estimation. Xi preprint arXi:2241.11191 2024.
16

---

Published in Transactions on Machine Learning Research (02/2026)
A Prompt
A.1 Anomaly detection
CIFAR-10 v.8. not CIFAR-10
You are given image, check if the image is from of the following 10 classes
airplane, automobile, bird, cat, deer, dog, frog, horse, ship, and truck, if so, classify the image into one of 10 classes, if not, label them as "unknown".
Think step by step, and give an answer in the folloving format: Reasoning: kyour_reasoning? Answer: <your answer>
ECG v.s. not ECG
Your task is to classify the given ECG signal into the normal case and the abnormal
Note that if the signal does NOT look like ECC signal cannot classified into either of the categories, return "unknown' as the answer
Give answer and stspep reasoning in the format:
Reasoning: yrour_reasoning
Answer: normal/abnoralluukee
A.2 Classification with rejection under ambiguous inputs
ImageNet--C
You are given an image, the image belongs to one of the following 10 classes: airplane, automobile, bird, cat, deer, dog, frog, horse, ship, and truck.
Please assign each image label from the 10 classes, think step by step, and give an ansver in the following format: Reasoning: your_reasoning
Label: class_name
Notice that if you find an image very ambiguous and cannot confidently classify it return "unknown" as the label.
Galaxy Zoo classification
You expert astronozer apecializing galaxy morphology. You will be shown images of galaxies and need to classify them into one of the following categories by analyzing their visual characteristics:
smooth_round: Galaxies that appear completely or nearly circular, with a smooth decrease in brightness from center to edge.
2. smooth_cigar: Galaries that appear elongated and smooth, with an elliptical shape resembling cigar, showing gradual decrease in brightness from center to edge.
thin line disk with a bright central bulge, similar to vieving a dinner plate from its edge
3. adge_on_disk. Galaxies viewed from the side, appearing
4. imbarred_spiral:: Spiral galaries without central bar structure, showing distinct spiral arms emanating directly from the galactic center.
17

---

Published in Transactions on Machine Learning Research (02/2026)
Classify the galaxy into exactly of the four categories listed above with step by step reasoning. Notice that if you find image very ambiguous and cannot confidently classify
"unknown" the label.
Your response should be structured as: Reasoning: [Brief explanation of the key visual features that support this classification]
Answer (category_name, unknown
A.3 Caption generation
ImageNet-C
You are given an image, please generate short description of the image.
Galaxy Zoo
You are an expert astronozer specializing in galaxy morphology. Please generate a concise description of the image that describes its key visual characteristics.
18

---

Published in Transactions on Machine Learning Research (02/2026)
B Mapping between CIFAR-10 and ImageNet classes
In the main text, we extensively consider the setting where we pick images from ImageNet (or its corrupted version) that overlap with CIFAR-10 categories and prompt VLMs to classify the images into CIFAR-10 categories. The exact mapping is presented in Table. 2. Notice that here we ignored the deer and horse categories from CIFAR-10 as we cannot find exact mapping ImageNet categories.
Table 2: Mapping from CIFAR-10 categories to corresponding ImageNet classes (ImageNet class indices in parentheses).
CIFAR-10 Category
ImageNet Classes (Indices)
Airplane
Airliner (404), Warplane (895)
Automobile
Beach Wagon (436), Couvertible (511), Model T (661), Sports Car (817)
Bird
Jay (10), Magpie (11), Eagle (12), Vulture (13),
Additional Birds (92, 93, 94, 95, 96)
Cat
Tabby Cat (281), Tiger Cat (283), Persian Cat (284), Siamese Cat (285)
Dog
Chihuahua (151), [Every 20th dog class up toj Mexican Hairless (268)
Frog
Bullfrog (30), Tree Frog (31)
Ship
Container Ship (510), Liner (628), Pirate Ship (724),
Schooner (780), Submarine (833)
Truck
Fire Truck (555), Garbage Truck (569), Moving Van (675), Pickup Trruck (717),
Police Van (731), Tow Truck (864), Trailer Truck (867)
19

---

Published in Transactions on Machine Learning Research (02/2026)
C Anomaly ECG inputs construction
To construct anomaly inputs that are clearly not ECG signal, we use a two step routine: We begin by randomly choose a random number generator
def get_random pot_rradom_mumberrGe www.nrummmaaaae (size=128);
rng_ list =
np. random.om.omm arrandm..ormal. np. app.raddmm.amae np ap.ddnoom. exponential, up.random, uniform, np ap.rrnnom.. chisquare, np. wrandom.geometee.
ronddmmmmniiform
]
rng = np.rondom. choice (rng_ list) return r0.60.6, 6, size=size)
Then we generate random line figure, with 128 points, the anomaly inputs via
plt. plot (get_random rwwdaaannnnnerrrrreee number (128), get_randor number (128), 1inevidth=0.86 ., c='black')
20

---

Published in Transactions on Machine Learning Research (02/2026)
D Galaxy Zoo explanation
Galaxy Zoo is citizen science dataset, for each galaxy image, annotators are asked to follow the decision tree (https://data .awpetaa.g.axyzzo.oooooo wwt.yyyoo..oggg_ttteete Beteee/gg_teees___eeeeee srreeshtml) and perform classification for the galaxy image based on the visual features. Importantly, the metadata of the dataset provides a property called leaf_ leaf_prob that describes the agreement level among annotators for the mostly agreed category. We later use leaf_prob a score that denotes the inherent ambiguity level of given galaxy image.
To actually construct the dataset for evaluation VL.M, first select all images whose majority vote label falls into one of the following categories:
smooth_round, smooth_ cigar, edge_on_ _isk,, unbarred_spiral
which the four representative galaxy categories studied in Galaxy MNIST (Walmsley, 2022). Then randomly selected 5,000 images from the pool.
Then, based on the leaf _prbb, we create three levels of disagree level
Leaf prob in (0. 75, 1.0): Disagree level 1
Leaf prob in (0.5, 0.75): Disagree level 2
Leaf prob in (0.0, 0.5) Disagree level 3
The distribution of categories and leaf _prbb from the 5,000 samples are plotted in Fig. 4b
2.0
1.5
2000
Density
Count
1.0
1000
0.5
0.0
0
0.0
0.2
0.4
0.8
1.0
smooth_roude round
unbarred spiral
-hbarred_spiralll
smooth_ cigar edge_on _one_oo___ disk _n_disk
Leaf Prob
Class
Figure 7: Statistics of the 5,000 sample galaxy zoo datasets used in the experiments.
21

---

Published in Transactions on Machine Learning Research (02/2026)
E
Ablation study: Effect of instruction prompt
Prompt method
Allow rejection
Simple (default)
Direct (visual only)
Caption answer (visual text)
No Yes
Llama3.2 11B
Qwen2 7B
Qwen2.5 3B
Qwen2.5 7B
Qwen2.5 72B
GPT4o-mini
Accuracy
1.00
0.75
Accuracy diff
0.2
0.0
Rejection rate
0.5
0.0
2
1 2
4
2 3
2
Corruption level Corruption level Corruption level Corruption level Corruption level Corruption level
Figure 8: Ablation study on the prompting strategy. Similar to the setting studied in Fig. 2b, here we consider different prompting strategies (line color, detailed in Appendix. E), and models (columns). The second row illustrates the accuracy improvement, can that for certain old models, the prompting strategy matters, but for other, especially the newer ones, it matters little.
Simple
Direct
Caption Answer
Model
Precision
Recall f
Precision t Recall
Precision f Recall
Llama 3.2
0.991
0.718
0.994
0.492
0.994
0.897
Qwen 2
0.964
0.757
0.994
0.215
0.992
0.917
Qwen 2.5 3B
0.993
0.782
0.994
0.549
0.993
0.725
Qwen 2.5 7B
0.982
0.967
0.992
0.837
0.999
0.972
Qwen 2.5 72B
0.976
0,986
0.969
0.992
0.987
0.987
GPT4o-mini
0.964
0.974
0.974
0.975
0.988
0.974
Table 3: Similar to the setting in Table. 1, but here considered different prompting strategy, discussed in Appendix. E. Overall, smaller and older models such as Llama 3.2 and Qwen 2 are sensitive to the prompting strategy, more powerful models are less sensitive (e.g. GPT4o-mini and Qwen 2.5 72B).
We perform an ablation study where we vary the prompt that instructs the VLM to answer the query in a certain way. To more specific, we considered the following three regimes:
Simple Simple and standard way: we prompt the model to provide step-by-step reasoning (Wei al., 2022) and then provide classification answer.
Direct We prompt the model to only oulpul the classification unswer, which prohibits the model from generating any explicit intermediate verbal reasoning steps, including image caption.
Caption answer We explicitly prompt the model to always first caption the image, then answer the question using both the caption and the image.
The goal is to understand whether:
The model perform the uncertainty quantification task through only reading the image (Direct prompting)
The model cannot understand the uncertainty by just reading the image, but would need first caption the image and then utilize the textual description to determine the level of ambiguity.
22

---

Published in Transactions on Machine Learning Research (02/2026)
In Fig. 8, present accuracy with and without the rejection option (solid and dashed lines) in the first row v.8. corruption level, the improvement of allowing the rejection option (second row) and the rejection rate (third row) under different models (columms) and prompts (line colors). Particularly, for Llama 3.2 and Qwen2, the direct prompt (blue lincs) shows almost no improvement when rejection option is enabled and a low rejection rale, and the models only start to reject ambiguous inputs under Caption & answer (green line), whereas for other models, the VLMs demonstrate the ability of rejection under all types of prompts. This implies that these two models (Llama 3.2 and Qwen2) lack the ability to "silently" reason about uncertainty, instead, they need to to first (be asked to) spell the image out, then use the textual feature to determine uncertainty. The observation is again confirmed in the OOD detection task (Table. 3), the metrics under the direct prompt for these two models are outperformed by the Caption & answer prompt.
Regardless, the ability rejection from the latest models does not seem to be affected by the prompting strategy. It also worth noting that, the fact that these models can "silently" quantify uncertainty does not rule out the possibility that they are verbal thinker internally: They could be "implicitly" using verbal reasoning inside.
In the blocks below, provide example outputs from Llama 3.2 and Qwen2.5 7B given Gaussian noise corrupted ship image, where Llama 3.2's output varies significantly with prompts whereas Qwen2.5 7B shows much higher stability
Llama3.2
Direct:
Label: cat cat<leot_ien
Simple:
**Problem Statement:**
are given an image and asked to classify it into one of the following 10 classes: airplane, automobile, bird, cat, deer, dog, frog, horse, ship, and truck.
**Aproach:**
To solve this problem, we will analyze the image step by step and make an educated guess based the visual features present in the image.
**Step Observe the Image**
The image appears to be low-resolution, pixelated photograph of vehicle or an animal. The image too blurry to make out any distinct features.
**Step 2: Look for Clues**
Upon closer inspection, notice that the image has dark body with long, curved shape protruding from it. This could be tail or a limb.
**Step 3: Eliminate Impossible Classes**
We can eliminate the classes that do not match the visual features observed in the image. For example, we can eliminate "airplane" and "ship" because they do not have curved shape
dark body
Step Make an Educated Guess** Based on the remaining classes, we can make an educated guess. The image resembles cat, which has dark body and a long, curved tail.
**
Reasoning: The image is too blurry to make out any distinct features, but it resembles a cat.
Label: cat<llottidd>>
Caption
The image appears to be pixelated representation of a building, possibly skyacraper high-rise structure. The dominant colors shades of gray, with
23

---

Published in Transactions on Machine Learning Research (02/2026)
red and green hues visible in the upper right corner. The overall effect is one of highly distorted or resoluttn image, making it difficult to discern any specific details.
Claassification:*
Based the description, I would assign the image the label "unknown" as it does not clearly resemble any specified classes (airplane, automobile, bird, cat deer, dog, frog, horse, ship, or truck) <leot_id|>>
Qwen 2.5 7B
Direct:
unknown
Simple:
Reasoning: The image appears to be close-up of a textured surface with various colors patterns. It does resemble any the typical shapes features associated with the 10 classes provided (airplane, automobile, bird, cat, deer, dog, frog, horse, ship, and truck). The image characteristics that would allow for confident classification into one of these categories.
abstract and lacks distinct
Label: unknown
Caption Answer:
Description: The image appears to show close-up view of colorful, patterned surface with geometric shapes and vibrant colors. The shapes seem to be arranged in grid-like pattern, possibly resembling tiles mosaic.
Label: unknown
24

---

Published in Transactions on Machine Learning Research (02/2026)
F
Full results across all corruptions
Allow rejection No Yes
Llama3.2 11B
Qwen2.5 3B
Qwen2.5 7B
Qwen2.5 72B
GPT4o-mini
Rejection Accuracy
1.0
0.5
0.0
Caption diversity
0.3
0.2
0.1
3
3
2 3
5
2
5
2
5
Corruption level
Corruption level
Corruption level
Corruption level
Corruption level
(a) Defocus blur
Accuracy
1.00
0.95
Rejection
rate
0.2
Caption diversity
0.2
0.1
2
Corruption level
Corruption level
Corruption level
Corruption level
Corruption level
(b)
Pixelate
Accuracy
1.0
0.8
Rejection
rate
0.25
Caption diversity
0.3
0.2
0.1
3
Corruption level
Corruption level
Corruption level
Corruption level
Corruption level
(c) Pixmix
Figure 9: Similar to the setting in Fig. 2b, but under different corruption types. The observation still holds: Enabling rejection allows VLM to abstain ambiguous samples, providing nearly perfect accuracy for classified samples. Additionally, caption diversily, as predicted in Sec. 3.2, steadily increases as corruption level intensifies.
25

---

Published in Transactions on Machine Learning Research (02/2026)
G Extended result on GalaxyZoo
We prompt the VLM (Qwen2.5 7B) to answer the first question in the decision tree using the following prompt, where the red part denotes the prompt for rejection
Select one these categories that best describes the image:
features disk: The image has any features at all, such spiral arms, bars, dust Lanes. smooth: The image gradually fades in all directions from the center (it can have tight bright core).
**Notice that** if you see features but cannot interpret the structure with confident, return "Unknown",
**Notice that** if you unsure between the options, return "Unknown".
Think step by step, answer the question in the following format: Explanation: <your explanation here> Answer: <your answer here>
In Figure. 10 and Figure. 11, we present example images with low and high consensus between annotators, respectivcly, for the first question in the decision tree (https://data. galaxyzoo org/gz_tar trees/gz gaaes/gz_tut trees. html). Broadly, we find that, unlike the observations in the main text, where the model fails to reject high disagreement images, now the VLM becomes capable of rejecting inputs for which no consensus is reached thanks to the simplicifcation of tasks and domain knowledge provided in the prompt.
P(Smooth): 0.40
P(Featured): 0.60
Explanation: The image appears show galaxy with central bright region and some fainter structures extending outward. There are hints of spiral arms other features, but they not clearly defined or easily interpretable.
Answer: Unknown
P(Smooth): 0.59 P(Featured): 0.41
Explanation: The image shows galaxy with a distinct elongated shape and a bright central region, which is characteristic of an elliptical galaxy. There are no visible spiral arms, bars, or dust lanes that would suggest it is a spiral galaxy. The brightness gradient is not smooth; instead, there is a concentration of Light towards the center, which is typical for elliptical galaxies. Answer: Unknown
P(Smooth): 0.67 P(Featured): 0.33
Explanation: The image shows galaxy with a distinct elongated shape and faint glow extending along its length. There are no clear spiral arms, bars, dust lanes visible in this image. The brightness is relatively uniform across the galaxy, without any noticeable bright core fading pattern. Answer: Unknown
Figure 10: On images where annotators show disogreement (annotator voting probability shown on the top of cach image), VLM also rejects these samples by answering "Unknown".
26

---

Published in Transactions on Machine Learning Research (02/2026)
P(Smooth): 0.95
P(Featured): 0.05
Explanation: The image appears to show a galaxy with a smooth, elongated shape and a central bright region. There are visible spiral arms, bars, or dust lanes that would suggest the presence features typical of galaxies with active star formation or recent interactions. Answer: smooth
P(Smooth): 0.89
P(Featured): 0.11
Explanation: The image appears to show a galaxy with a central bright region and fainter halo extending outward. There are distinct spiral arms, bars, or dust lanes visible in the image. The brightness is more uniform across the galaxy, suggesting it might be smooth distribution rather than having specific structural features. Answer: Smooth
P(Smooth): 0.92 P(Featured): 0.08
Explanation: The image appears to show galaxy with central bright region and a fainter halo extending outward. There are no distinct spiral arms, bars, or dust Lanes visible in the image. The brightness is concentrated in the center, which suggests smooth distribution of light.
Answer: Smooth
P(Smooth): 0.19
P(Featured): 0.81
Explanation: The image shows galaxy with distinct elongated shape and a central bright region, which is characteristic of a spiral galaxy. The structure appears to have spiral arms extending outward from the central bulge. Answer: features or disk
P(Smooth): 0.12
P(Featured): 0.88
Explanation: The image shows a galaxy with distinct spiral structure. There are visible spiral arms extending outward from the central region, which is characteristic of spiral galaxies. The presence of these spiral arms indicates that the galaxy has features. Answer: features disk
P(Smooth): 0.03
P(Featured): 0.97
Explanation: The image shows a galaxy with a distinct spiral structure, which includes spiral arms extending outward from the central region. There also a bar-like feature running through the center the galaxy, which is common structural element in many galaxies.
Answer: features or disk
Figure 11: On images where annotators show consensus (annotator voting probability shown on the top of each image), VLM returns answers aligned with the majority vote results.
27

---

Published in Transactions on Machine Learning Research (02/2026)
H
Licenses for existing assets
Datascts:
ImageNet
ImageNet--C
License: Custom ImageNet Terms of Use (Non-commercial research only)
URL: https://ww.inage. net.org/dowwload
CIFAR-10
License: MIT License
URL: https://twwt cs. torooto. forronto.edl//tarii/ccafaa...aa.
Galaxy Zoo / Galaxy MNIST
License: Creative Commons AtrbbttionkouCCmmmerraiiSaaaAiia (CC BY-NC-SA 4.0)
URL: https://datee waaaraalglaaxxzooooooo org/
PTB-XL (ECG Dataset)
License: PhysioNet Credentialed Health Data License (Restricted; requires credentialed access)
URL: https: hatp:s::/hhhsinettte httg///oten//tbb///////
Models:
GPT-4o-mini
License: Proprietary (OpenAl Terms of Use)
URL: https: https://openai.coom
Llama 3.2 (Meta)
License: Meta Llama 3 Community License (Non-commerciaal research use only)
URL: https: facebook. comm/ressuurceesnooellss antarddedaaa---i-rrrree///aaaa
Qwen 2 / Qwen 2.5 (Alibaba)
License: Qwen License Agreement (Permits research and certain commercial uses)
URL: attps://qwen ali aliyun. com/
28


Reviews:
[{"review_id": "XW0Pt8arX0", "reviewer": "Reviewer_V15z", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper evaluates the robustness of modern vision-language models (VLMs) against two classic uncertainty quantification challenges: anomaly detection and classification under ambiguous inputs. Its core contributions include: (1) Empirically demonstrating that newer/larger VLMs outperform earlier models in handling corrupted/OOD inputs but still tend to hallucinate confident responses without proper prompting; (2) Proposing a simple yet effective solution—explicitly prompting models to abstain from uncertain predictions— which achieves near-perfect reliability on standard benchmarks (e.g., ImageNet-C, CIFAR-10 vs. Non-CIFAR-10) without architectural modifications; (3) Introducing caption diversity as a label-free metric to reveal VLMs’ internal uncertainty, enabling prediction of model rejection behavior.\n\nKey strengths: Rigorous experimental design with diverse datasets (natural images, ECG signals, galaxy morphology) and model families; practical, low-cost intervention (prompting) with clear real-world utility; novel insight linking caption diversity to uncertainty.\n\nKey weaknesses: Limited solution for domain-specific tasks (e.g., galaxy classification) where VLMs fail due to insufficient expertise; insufficient analysis of why certain models (e.g., Llama 3.2) are more sensitive to prompting styles; lack of comparison with state-of-the-art uncertainty quantification methods for VLMs.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The paper’s claims are strongly supported by comprehensive empirical evidence. First, the authors design well-controlled experiments across multiple tasks (anomaly detection, ambiguous classification) and datasets, covering both general and domain-specific scenarios. Second, they benchmark 6 VLMs (spanning different sizes and release dates) and report quantitative metrics (precision, recall, F1-score, accuracy, rejection rate) alongside qualitative examples (e.g., Figure 2a’s noisy image classification). Third, visualizations (e.g., Figure 5 on caption diversity vs. corruption level) and ablation studies (Appendix E on prompting styles) validate causal links between prompting, model behavior, and uncertainty. All experiments are described in detail (e.g., dataset construction, sampling parameters) to ensure reproducibility, aligning with TMLR’s evaluation criteria for accurate and convincing evidence.", "audience_yn": "Yes", "audience_explanation": "TMLR’s audience includes researchers focused on machine learning, deep learning robustness, and multimodal models—groups for whom this work is highly relevant. Uncertainty quantification is a critical topic for safe deployment of AI systems, and VLMs are increasingly adopted in real-world applications (e.g., medical imaging, autonomous systems). The paper addresses a pressing gap: while VLMs are scaled to massive sizes, their ability to handle ambiguous/anomalous inputs remains understudied. The findings (e.g., prompting as a quick fix, caption diversity for uncertainty estimation) provide actionable insights for practitioners and open new directions for research on VLMs’ reliability. Additionally, the contrast between general and domain-specific task performance informs future work on domain-adapted VLMs, making the paper of interest to both fundamental and applied researchers.", "requested_changes": "- Address the limitation in domain-specific tasks: Provide concrete proposals (e.g., integrating domain knowledge via prompt engineering, few-shot fine-tuning, or hybrid models) to improve VLMs’ uncertainty quantification in specialized fields like galaxy morphology. This gap currently undermines the paper’s generalizability.\n- Expand analysis of prompting style sensitivity: Explain why older models (e.g., Llama 3.2) rely on “caption & answer” prompts while newer models (e.g., Qwen 2.5 72B) are robust to prompt variations. This analysis will strengthen the understanding of how VLMs process uncertainty instructions.\n- Compare with existing uncertainty methods for VLMs: Include a brief comparison with state-of-the-art techniques (e.g., Bayesian VLMs, ensemble-based methods) to contextualize the proposed prompting approach’s advantages (e.g., efficiency, simplicity) and limitations.\nExtend experiments to additional VLMs (e.g., Gemini Pro, Claude 3) to confirm the generalizability of the prompting and caption diversity findings.\n- Supplement Appendix E’s ablation study with qualitative examples of how different prompts alter model reasoning (e.g., verbatim outputs for “direct” vs. “caption & answer” prompts).\n- Discuss practical trade-offs of the rejection prompt (e.g., false rejection rates in high-stakes scenarios) and provide guidelines for choosing prompt styles based on model capabilities.\n- Clarify the mechanism behind caption diversity: Include a short discussion on why ambiguous inputs lead to more diverse captions (e.g., conflicting visual features, lack of domain knowledge) to deepen the theoretical grounding.", "broader_impact_concerns": "N/A.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "F7i0uzBprz", "reviewer": "Reviewer_GM8A", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "**Summary**\n\nThe paper investigates if contemporary large VLMs have effectively solved the classic challenges of anomaly detection (OOD/label space mismatch) and classification under ambiguity (corruption/covariate shift). The authors find that prompting the VLMs to explicitly reject ambiguous inputs leads to better correctness, while a standard QA prompt leads to guessing & hallucinations.\n\nThe paper evaluates closed-source model GPT-4o-mini, and open weight VLM families Llama-3.2, Qwen2VL & Qwen2.5VL on two tasks i) Anomaly detection on CIFAR-10 vs not CIFAR-10 & ECG vs  not ECG, where the model is evaluated on ability to reject images that don't belong to any GT category ii) Classification on ambiguous inputs with rejection on ImageNet-C & Galaxy Zoo 2 datasets, where accuracy is computed on samples that are not rejected.  For domains requiring expert knowledge, eg, Galaxy Zoo, reject options doesn't show performance gains, indicating that abstention does not work for domains where the model's internal uncertainty estimates may be poorly calibrated. Finally, they propose to use caption diversity as a proxy for the VLM's uncertainty on an input. Many text descriptions of the i/p image are sampled from the VLM and one minus avg. pairwise cos similarity of all description embeddings gives the caption diversity score. For ImageNet-C images, more corrupt images have higher caption diversity. However this diversity is not observed for the ambiguous samples from the galaxy classification task.\n\n  \n**Strengths**  \n\\+ The ability to handle ambiguous queries is critical for real-world use of VLMs. The paper shows that a simple system prompt change can enable VLMs of various strengths to handle ambiguous inputs. This simple solution makes use of recent advancements in foundation models, where relatively smaller models too have strong instruction-following capabilities and robust world knowledge.  \n\\+ The paper is well written and motivated.  \n\\+ The proposed caption diversity metric makes intuitive sense for images with occlusions, corruptions and other sources of visual ambiguity.    \n  \n  \n**Weaknesses**  \n1. How do VLMs handle more nuanced types of ambiguity? For instance question-conditional ambiguity, where the image is clear but the question is underspecified (color of the bottle when the image has many bottles), or when the object of interest is occluded? In such cases perhaps clarification can help the VLM answer the question.  \n2. For question based semantic ambiguity, it is expected that caption diversity remains low, as a general image description would be relatively unchanged. A discussion on this would better highlight the limitations.  \n3. For open weight models caption diversity is an expensive metric to compute. A comparison with common logit-based baselines such as answer token entropy (or first token entropy) would strengthen the metric.  \n4. Although the paper shows rejection prompting to be a good solution, it provides less control on when a sample should be rejected (as observed in the ECG task). It would be interesting to see if different prompting strategies could offer a better tradeoff between accuracy and rejection rates.  \n5. The failure of rejection prompting on Galaxy Zoo can be seen as due to epistemic uncertainty, where the model lacks sufficient domain knowledge. It is unclear if rejection prompting inherently performs poorly under high epistemic uncertainty. Exploration of simple solutions such as in-context examples in the prompt can help understand this better.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The claims are well supported by the experimental results. Rejection prompting improves performance on both tasks. The paper highlights the limitation of this approach in domains that require expert knowledge.", "audience_yn": "Yes", "audience_explanation": "The paper addresses a practical problem on VLM hallucinations under ambiguous inputs. The paper shows that rejection prompting works well for general images but fails in cases of specialized domains, which is an actionable insight in practice.", "requested_changes": "\\- [required] Clarify limitations of rejection prompting w.r.t epistemic uncertainty. As mentioned in weakness 5, it remains unclear if rejection prompting only works for aleatoric uncertainty. This clarification is important to understand the limitations of the method.  \n\\- [optional] Expanding discussion to additional, realistic forms of ambiguity seen in VQA would strengthen the paper. A discussion of applicability of caption diversity under such ambiguity types can highlight its applicability.  \n\\- [optional] Compare caption diversity with simpler logit-based uncertainty baselines (as mentioned in weakness point 3).  \n\\- [optional] Can few-shot in-context examples improve performance in Galaxy Zoo (weakness point 5)?", "broader_impact_concerns": "There are no major concerns related to the work's broader impact.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "WYyzGHDcEf", "reviewer": "Reviewer_egu1", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "**Summary**\n\nThis paper claims that today's VLMs, despite scaling in both data and parameters compared to prior approaches, still suffer from weakness to both covariate and concept shift, and cannot explicitly tag such shifts as uncertain, rather than providing a misclassification. Explicitly:\n\n> While VLMs may no longer face clear-cut in-distribution v.s. out-of-distribution boundaries, they still encounter practical challenges in handling visually ambiguous inputs, or anomaly inputs that fall outside the semantic scope defined by a user prompt.\n\nThe paper studies how VLMs behave through several case studies: (1) corrupted ImageNet images, (2) CIFAR-10 vs. Non CIFAR-10 images, and (3) two specialized tasks, ECG signal classification and Galaxy image recognition. They come to the conclusion that modern models are robust under many situations, but still struggle in others. \n\nThe paper shows the following findings:\n* In anomaly detection, the paper asks VLMs to predict semantic mismatches, first by taking CIFAR-10 vs. non CIFAR-10 classes, which the models achieve with quite high F1 scores, and then ECG vs. not ECG signals (as images), on which the models are asked to predict which signals are, and are not, valid ECG. In general, the paper shows that VLMs are quite capable of anomaly detection when prompted to do so.\n* The paper also asks VLMs to perform classification under ambiguous inputs (ImageNet-C into CIFAR-10, GalaxyZoo). The paper shows  in F.2 that as corruption level increases, accuracy generally falls, and rejection rate rises. In all cases, it appears that explicit prompting for uncertainty rejection appears to help. In F.6, using annotator disagreement as a proxy for ambiguity, we see that VLM accuracy degreades, and VLMs do not naturally use rejection to improve performance rates (likely due to insufficient domain knowledge).\n* The paper finds that generated caption diversity is correlated with corruption, and input ambiguity.", "claims_and_evidence_yn": "No", "claims_explanation": "The primary technical contribution is based on the  the observation that VLMs uncertainty level about an input image can be revealed by prompting the VLM to generate multiple captions under random sample decoding, where VLMs tend to generate more diverse captions for visually ambiguous samples. This claim, while factually supported, in my opinion is somewhat misguided. VLMs will produce more diverse captions under at least two scenarios: the first, which the paper correctly identifies, is when the image is OOD (grounded in Figure 4a and 5.  The second, which the paper ignores, is when the captions **require** uncertainty. In many cases, there are multiple ways to correctly describe an image, which will lead to higher caption diversity naturally under sampling. In these cases, we are penalizing a model for flexibility that is actually desired, rather than uncertainty which should be expressed, see [1]. \n\nSome other claims are also overly broad:\n\n* \"Neural networks often struggle with input uncertainty due to their training methodology. When trained on carefully curated datasets with minimal ambiguity, models develop a tendency to produce high-confidence predictions for all inputs, regardless of their clarity or relevance.\" (Sec. 2.1) I don't know of any work which has clearly attributed input uncertainty to training methodology - in general, I would instead expect this to be a consequence instead of the data, where there isn't sufficient examples of uncertainty in the data to avoid biasing the model to produce a confident answer. This, to me, seems like an unnecessary claim that could be removed.\n* The reasons at the end of page 3/the arguments on page three are actually quite poor arguments for a lack of uncertainty classification. In fact, since many times uncertainty is data-centric, to me at least, there's no expectation that model scaling will improve uncertainty estimation.\n\nFinally, there are several statements in the introduction that appear to be motivation, but are not resolved in the paper:\n\n* \"We believe our study demonstrates an important lesson: Despite the clear superiority of modern, large-model AI systems, do not assume they solve older benchmarks.\" From what I can tell, the performance on older benchmarks is competitive with, if not exceeds performance of older methods? What is the intent here? \n* \"Are the traditional evaluation tasks of corruption robustness and OOD detection meaningful for VLMs?\" I don't feel like this paper makes a stance on this claim, or provides sufficient evidence to determine either way if this is the case. \n* The paper makes claims about all VLMs, but tests a very minimal range of older VLMs (Llama 3.2, Qwen 2, GPT-4o-mini), and no reasoning models, which are (in my personal observation), much better at understanding and reasoning over uncertainty. \n\n[1] Chan, David, et al. \"Ic3: Image captioning by committee consensus.\" Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. 2023.", "audience_yn": "Yes", "audience_explanation": "This experiment, while limited, does not have any clear structural concerns. The experiments themselves are novel and structured. The clearest finding is:\n\n> prompting the models explicitly to \"reject ambiguous inputs\" substantially enhances their reliability\n\nwhich is an unambiguous, and well-supported finding that some people may find interesting. If the introduction were scoped correctly, the findings are at least interesting as a building block to larger uncertainty.", "requested_changes": "Some clear changes are, in my opinion, necessary:\n* Narrow/Reframe the caption diversity claim: Caption diversity should probably be explicitly reframed as a diagnostic proxy, rather than a general uncertainty estimator (and the difference between sources of diversity should be specified). The claim should also probably be restricted to classification-oriented prompts\n* Remove/weaken the overly broad training-methodology claims: one of the things that needs to be discussed is the connection between uncertainty and data. \n* Clarify the Paper’s Position on “Are Classic Benchmarks Solved?”: I think that this paragraph needs a second look in terms of the overall claims, and what is meant by \"solved\" - perhaps we mean \"robust by default?\"\n* The paper makes claims about “VLMs” broadly but evaluates a narrow subset, excluding reasoning-focused models, the set should be expanded, or the claims rephrased. \n* Human baselines/baselines in the tables themselves would be appreciated, to help ground overall performance. \n\nAdditionally, the paper would benefit from clearer alignment with recent work framing uncertainty as primarily data- and task-dependent rather than a consequence of insufficient model scale, as well as a more explicit interpretation of the Galaxy Zoo failure as evidence that uncertainty awareness depends on semantic grounding. Reordering the contributions to foreground prompted abstention as the main result, with caption diversity treated as a secondary diagnostic, would also improve clarity and better reflect the strength of the empirical findings.", "broader_impact_concerns": "I have no ethical concerns with the implications of this work - and in my opinion a broader impact section is not required.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)