Reflexive Guidance: Improving OoDD in Vision-Language Models via Self-Guided Image-Adaptive Concept Generation
Authors: Jihyo Kim
Seulbi Lee
Sangheum Hwang
Venue: ICLR 2025 Poster (2025)
Keywords: Out-of-distribution detection
Trustworthiness
Vision-language models
Foundation models
Primary area: foundation or frontier models, including LLMs
TL;DR: Large vision-language models perform well on out-of-distribution detection, and this capability can be significantly improved through image-adaptive concept generation.

Abstract: With the recent emergence of foundation models trained on internet-scale data and demonstrating remarkable generalization capabilities, such foundation models have become more widely adopted, leading to an expanding range of application domains. Despite this rapid proliferation, the trustworthiness of foundation models remains underexplored. Specifically, the out-of-distribution detection (OoDD) capabilities of large vision-language models (LVLMs), such as GPT-4o, which are trained on massive multi-modal data, have not been sufficiently addressed. The disparity between their demonstrated potential and practical reliability raises concerns regarding the safe and trustworthy deployment of foundation models. To address this gap, we evaluate and analyze the OoDD capabilities of various proprietary and open-source LVLMs. Our investigation contributes to a better understanding of how these foundation models represent confidence scores through their generated natural language responses. Furthermore, we propose a self-guided prompting approach, termed Reflexive Guidance (ReGuide), aimed at enhancing the OoDD capability of LVLMs by leveraging self-generated image-adaptive concept suggestions. Experimental results demonstrate that our ReGuide enhances the performance of current LVLMs in both image classification and OoDD tasks. The lists of sampled images, along with the prompts and responses for each sample are available at https://github.com/daintlab/ReGuide.

Published as a conference paper at ICLR 2025
REFLEXIVE GUIDANCE: IMPROVING OODD IN VISION-LANGUAGE MODELS VIA SELF-GUIDED CONCEPT GENERATION
Jihyo Kim", Sculbi Lee", Sangheum Hwang Department of Data Science, Scoul National University of Science and Technology {jibyo. kim, seulbi, chwang) seoultech. ac kr
ABSTRACT
With the recent emergence of foundation models trained on internet-scale data and demonstrating remarkable generalization capabilities, such foundation models have become more widely adopted, leading an expanding range of application domains. Despite this rapid proliferation, the trustworthiness of foundation mod- els remains underexplored. Specifically, the out-of-distribution detection (OoDD) capabilities of large vision-language models (LVLMss) such as GPT-4o, which are trained on massive multi-modal data, have not been sufficiently addressed. The disparity between their demonstrated potential and practical reliability raises concerns regarding the safe and trustworthy deployment of foundation models. To address this gap, evaluate and analyze the OoDD capabilities of various pro- prietary and open-souree LVLMs. Our investigation contributes to a better under- standing of how these foundation models represent confidence scores through their generated natural language responses. Furthermore, we propose a self-guided prompting approach, termed Reflexive Guidance (ReGuide), aimed enhancing the OoDD capability of LVLMs by leveraging self- seff-generated image-adaptive con- cept suggestions. Experimental results demonstrate that our ReGuide enhances the performance of current LVLMs in both image classification and OoDD tasks
INTRODUCTION
Thanks to substantial advancements in hardware and the availability of large-scale datasets, foundation models have achieved remarkable performance across wide range of tasks, demon- strating exceptional generalization. This evolution has shifted toward leveraging multiple data modalities, particularly vi- sion and natural language. As a sult, vision-language foundation mod- els have demonstrated their capabilities across diverse domains, from general tasks like text-to-image generation and vision-question answering, to special- ized fields such as medical diagnosis
3H-4x-42
AeGuide
HowrL
tho
Marviz
Hia
SCB hard
SSB Hars
MMCO
MHKD
TesStar
Tuhum
Natard
Ope-trage-5
Openimage
Comparison of LVLMs
(b) Effectiveness of ReGuide
Performance of proprietary and open-source LVLMs on the ImageNet200 benchmark: (a) state-of-the art comparison and (b) the boosting effect of our ReGuide
Figure 1:
and robotics (Esser et al.] 2024 Li et al.) 2023 Wake et al. 2023 Majumdar et al. 2024 Despite
the widespread adoption these large langan-language models (LVLMs)) including GPT (OpenAT, 2024), Claude (Anthropic) 2024), and Gemini (Reid al. 2024). their trustworthiness and relia- bility have not been adequately investigated. Ensuring the robustness of deep neural networks has been a key research area guarantee their safe application in practice. With the rapid populariza- tion of LVLMs, practical concerns such as harmful content filtering and domain generalization have
"Equal contribution
"Corresponding author
https:: /i thub. com/da a ReGui

---

Published as a conference paper at ICLR 2025
been actively studied (Zhang et al., 2024b Han al. 2024 However, fundamental aspects, such as the quality of confidence estimates for model predictions extensively studied in single-modal models remain underexplored. Some works have studied OoDD in CLIP (Ming et al. 2022 Diang
et al. 2024| Cao et al. 2024) and Stable Diffusion (Zhu et al., 2024), but few have examined OoDI in L.V.Ms particularly with respect confidence scores expressed through their natural language responses. This gap between their demonstrated capabilities and real-world reliability raises con- cerns about ensuring the safe and dependable deployment of LVLMs.
To bridge the gap, we first evaluate and compare the OoDD capabilities of LVLMs. The OoDD task allows us to investigate how LVLMs behave when required to generate responses beyond the categories defined within the user aser-provided prompt. Due to the lack of prior experimental config- urations, we develop framework for evaluating the OoDD capabilities of LVLMs. Our focus is on detecting Ool image inputs based on the in-distribution (ID) space specified by text inputs. To this end, we design a prompt that defines the II) space and guides the LVLM to provide confidence estimates through its generated responses. Using this prompt, we evaluate the OoDD capabilities of both proprietary and open-source LVLMs and analyze their behavior in expressing confidence from various perspectives. Fig. I(a) summarizes the comparison of the evaluated LVLMs. In general, proprietary models outperform open-source models, and open-source models show performance improvements as model size increases. However, some open-source models including VLvv (Mistral-7B) (Li et al. 2024a) and GL-M-v-9B (GLM et al. 2024) exhibit low OoDD performance despite achieving decent results on popular VLM benchmarks We found that one of the reasons for this discrepancy is the insufficient image interpretation capabilities of these models, which limits their ability accurately distinguish between ID and OoD inputs.
To enhance the OoD detectability of LVLMs, we propose a two-stage self-guided prompting ap- proach called Reflexive Guidance (ReGuide). In the first stage, ReGuide prompts an LVLM to suggest two groups of concepts based on the given image: semantically similar concepts (i.e., near- OoD) and semantically dissimilar concepts (i.e., far-OoD). In the second stage, these suggested concepts are employed as auxiliary Ool) classes. To the best of our knowledge, this is the first study to leverage image inputs to generate informative texts for OoDD. By utilizing the visual interpreta- tion capabilities of LSS ReGuide remains simple and model-agnostic, allowing the same prompt to be applied to different LVLMs. Fig. [(b) shows the effectiveness of ReGuide. Notably, ReGuide significantly boosts the overall performance of open-source models, making them comparable to proprietary models. GPT-4o, which is the top performer prior to applying ReGuide, also benefits from ReGuide, especially on near-OoD datasets (e.g.. NINCO, SSB Hard). Our results highlight the effectiveness of guiding LVLMs through self-generated, image-adaptive concept suggestions.
From the results of our study, ReGuide: Despite the strong visual interpretation capabilities of LVLMs, which enable them to pre- dict fine-grained classes of objects effectively, these models tend to avoid generating responses that fall outside the given prompt categories. Our findings suggest that the models may have developed a form of positive bias due to their training on positive image-text pairs, or that their ability to ex- plore areas beyond the information embedded in the prompt has been diminished, possibly due to human alignment processes. Addressing these limitations can lead to more reliable and versatile applications of LVL Ms in various domains.
draw the following insights regarding the effectiveness of
2 RELATED WORK
In single-modal vision models, OoDD has been actively studied. Starting with the baseline method
of using the maximum softmax value as an OoD score (Hendrycks & Gimpel| (017), methods have
evolved to improve confidence modeling (Moon et al. 2020 Liu et al. 2020 Bibas et al.)
2021
post-hoc techniques (Liang et al. 20188] Lee et al. 20188] Sun et al. 2021 Djurisic et al.
2023
Sun al. [2022] Zhang et 2023 Liu & Qin 2024). or a combination of both (Xu et al.
2024)
Another approach involves leveraging auxiliary OoD samples to better distinguish between 1D and
OoD inputs. While real OoD samples show strong detection performance (Hendrycks et al. 2019
Chen et al. 202T, Lu et al. 2023] Katz-Samuels et al. (022), they require access to real OoD data.
Synthetic samples offer an alternative, providing the benehts of auxiliary OoD data without the need
for collecting real samples (Lee et al. 2018a Du et al. 2022 Tao et al. 2023 Zheng et al. (023).
https::/huggingffiae co/ httpaa//co..ccec..ooapa/////aaoa 1_eee_1eeedrrrrd
2

---

Published as a conference paper at ICLR 2025
Xyp
Xip
8:11
Imoog Eroecer
Cassner
FouD
ITAge
Lenguge
XouD
Apheto of
Encoder
Pestution tiaee
Prettton chude
7: Mu (deg..r.co) cot)
(a) Image classifier
(b) CLIP
(c) LVLM
Figure 2: Comparison of the OoDD framework for single-mode modal classifiers, CLIP, and LVLMs
With the growing interest multi-moda models, particularly VLMs, OoDD methods have advanced toward leveraging both vision and language modalities. A pioneering LVLM (non- gnon-generative is CLIP (Radford et al., 202T). Ming al. (2022) established a baseline for OoDD on CLIP by us- ing the maximum cosine similarity between image and text features as an OoD score. One main approach in CL.IP-based OoDD focuses on using auxiliary text for OoD concepts, as text data is rel- atively easy to collect. Wang et al. (2023) simply added 'no' prefix to ID classes, while Fort et al. (202T) used labels from OoD datasets, which requires prior knowledge of OoD inputs. Jiang et al. 2024 Esmaeilpour et al. 2 y Cao al. (2024) derived auxiliary texts using only ID data, with Cao et al. (2024) obtaining OoD texts from large language models (LLMs) like GPT. Previous work such NegLabel (Jiang et al. 2024) and EOE (Cao al., 2024), as well as our proposed method, share the idea of using negative concepts for OoDD. However, we focus on generative LVLMs and negative concepts generated by LVLMs.
our approach is the first to utilize
Diffusion models, a type of generative model, have also been employed for OoDD. One approach is to compare the reconstruction quality between ID and OoD images (Gao et al. 2023 Tong & Dai 2024). or to generate synthetic images for OoDD in single-modal vision models (Du et al. 2023 Girella et al. 2024). However, generative VLMs, particularly foundation models (i.e., LVLMs), remain understudied in the context of OoDD. While several works have explored confidence esti-
Groot Valdenegro-Torg 2024). they do not focus introduced Unsolvable Problem Detection to evalu- ate VLMs' ability to reject unanswerable inputs. However, their focus is on rejection ability, while our study focuses on distinguishing between answerable and unanswerable inputs and proposes method to enhance this ability. In LLMss, several works use model-generated answers to guide final responses more accurately (Yao et al., 2023 Wei et al. 2022 Shanahan et al. 2023). Our work is the first to propose a similar approach in LVLMs, leveraging the model's 's own generated answers.
mates in model predictions (Han et al.
2024
on detecting OoD inputs. MMyai et al.
20224
3 OoD DETECTION ON VISION-LANGUACA FOUNDATION MODELS
3.1 PROBLEM DEFINITION
OoD is conventionally defined as distributions outside the training distribution. However, given the vast amount and broad domain coverage of data used to train LVLMs, this conventional definition faces challenges in its direct application to LVLMs. To address this, we extend the zero-shot OoDD framework of CLIP RRafford et al. 202I) to generative LVLMs.
Let X' be the image space and y = { the set of class-representing words, where C' is the number of classes, The OoDD problem for CLIP in zero-shot image recognition is defined as the scenario where y' does not contain the ground-truth label of an input image X. Given x, CLIP yields a prediction for x based on (f)( fo (prooppt (y.)) where sim(u, v) is the cosine similarity, prompt is text template designed to reflect y/ (e.g., a photo of y!), and Jr, JT are the image and text encoders, respectively. The cosine similarities between x and each yi E y are used to determine whether belongs to ID or OoD. If the ground-truie truth label of is not in ), those similarities for all y/ in y will be relatively low, leading to the classification of x as OoD.
We frame the OoDD problem for LVLMs based on the zero-shot OoDD scenario defined for CLIP. Similarly, ) is set to a fixed word set containing only ID class words. Given both x and y as inputs, an LVLM is instructed to produce prediction results in structured format. Then, class predictions and confidence estimates for x are extracted from the natural language responses by the LVLM. If the ground-truth label of is not in y, the LVLM should provide low confidence scores for all y; in
3

---

Published as a conference paper at ICLR 2025
Your
clessity the image into one clenses [ID classes none these classes) and asign contideree Task Description
each
dlserlyy image
you
image into you Rejection
mage belongs
you you
should provido your cont dence each class
following are guideelies for your repaoee Plaas resporei accenting guiceines
100 100.00.. Sercily the guidelines above.
Guidelines
Here is eiample of your respo noe Please respond with the following examp format Prediction car Confidence: Inipllnne: 34, car: 2307, bird 12 72, cal 54, 94, deer: 20103 ..., none of these dasses: 1,29
Examples
Prediction Confidence: jainlann: 32.05, car: 5.22 truck: 3.22, none of these dassess 0.00)
LVLM
Figure 3: Framework for OoDD evaluation on LVLMs
Y in its answers. Fig. summarizes the OoDD frameworks for Lingl-modal image classifiers, CLIP, and LVLMs. Single-mone modal classifiers detect OoD images based solely on outputs from image inputs. In contrast, CLIP and LVLMs leverage both image and text inputs, with CLIP combining them via similarity measures and LVLMs integrating them through cross-modal interactions. LVLMs also generate natural language outputs, controllable through prompt instructions.
3.2 PROMPT DESIGN
Fig. illustrates the framework for OoDD evaluation LVLMs with simplified prompt. Our prompt consists of four components: a task description, an explanation of the rejection class, guide- lines, and examples for the response format. Unlike previous work LVLMs (Han et al. 2024 Hwang et al. 2024| Groot & Valdenegro- Toro 2024), we encounter a significant number of failure cases when using simple prompt consisting of the task statement including )' and formatted out- put structure. We attribute this inconsistency to differences in how confidence scores are assigned in our framework. In our prompt design, the LVLM is expected provide confidence scores for each class in V, whereas prior work required only a single confidence score for the predicted class. To mitigate these failures, we enhance the prompt by adding the following components, resulting the final prompt used in our experiments. The effectiveness of providing guidelines and examples and the complete prompt can be found Appendix B.5 and B.6 respectively.
Rejection class. When we provide only y, we observe that predictions for Ool samples often do not correspond to any class in y. Due its strong zero-shot visual recognition capabilities, the LVLM can either provide the ground-truth label for an OoD input image or reject classification into any of the classes in y. To address this unintended behavior, we introduce a rejection class named none of these classes and provide a detailed explanation of its role to the LVLM.
Guidelines addressing failure cases. In addition to the case where the prediction does not fall within V, we identify several other issues, such mismatches between the class assigned the high- est confidence score and the predicted class, assigning a confidence score of 0.0 to all classes in- cluding the rejection class, failing provide confidence scores for all classes in y. To reduce the frequency of these failures, we include guidelines that address the most common cases.
Response examples. In-context learning (ICL) is commonly used improve response quality by providing models with examples that illustrate the desired format during inference, allowing them to generate outputs that better align with these examples (Brown al., 2020). We leverage ICL to reduce failure caused by formatting issues. However, we observe that when only single example provided, the LVLM tends to mimic the example, regardless of whether the input image ID or OoD. To mitigate this tendency, we provide examples for both ID and OoD input images.
3.3 OoD SCORE DESIGN
Since we provide the rejection class for OoD inputs, the ideal behavior of LVLMS for confidence estimates to assign high confidence scores to one of the classes in y for ID inputs, and to the rejection class (i.e., none of these classes) for OoD inputs. Therefore, we use the maximum confi- dence score among the classes in y (i.e., ID classes) as the OoD score. Note that we do not constrain the sum of confidence scores. We apply the softmax function to all confidence values to normalize them, including that of the rejection class. Based on this OoD score design, an input image is likely to be ID if the score is high, and likely to be OoD if the score is low.
4

---

Published as a conference paper at ICLR 2025
Comparison on the ImageNet200 benchmark. Full model names are in the footnotes. 'Valid' indicates the ratio of valid responses out of a total of 23, 031 image-promp prompt queries, with counts in brackets. Bold highlights the best performance among generative LVLMs.
Table 1:
IN200
NINCO
Near-OD
SSB-Hard
INaramaist
Fan-OoD
Testures
Openinage-I
All OnD
Models
Valid
ACC (t)
FPR of 95%TTR (JV (UROC
SCALE"
86.37
93.98
Angh
87.01
55.83 85.74
71.22/ 80.00
19.14/95.81
21,00 95.67
31.06/9.55
OpenCLIP
100.00 (23,031)
87.41
62.27 85.31
71.48 78.36
42.76/ 92.49
47,83/ 89.62
17,47/90.08
61 83.54
GPT-4o
85:49 689)
89.78
22,30/ 92.00
38.95 /81.41
2.06 /97.58
7,45 95.85
3,78/97.17
23.50 88.50
Claude
(18.911
72.18
78.41 55%.00
10.17/94.20
18.31/
Geminl Pro
(2.92.221,170)
21.55 89.03
55.24 77.40
997.73
(9644
32.97/ 81.74
LLaVA-v1.6
71.63 (16.4%)
2.45
100.00 / 50.85
100.00 / 48.95
100.000 50.05
1000/00 / 26
100.00/ 49.23
/ 50.11
GLM--y
89, 00 (20.498)
69.41
100,000 79.23
100.00/ / 74.35
100 001 83.01
100.000 /83///33888 .45
100.00/ 83.11
100000 77,96
62.68 (14,436)
90.22
$2.59 / 58.32
94,21 52:51
36.69/
28.08 /
50.89/ / 774.6
75.95/ 61.63
VL2-76B
97.36 (22.444))
88.30
100,00 72.27
62
100.00/ 95.57
100.00/ 91.62
1 90.12
100.00/ 74.14
OOpenCLIP-ViT (-P--I----33 32, GPT-4o (2024 08 06),
LLaVA=<1.6 6 6-MMittal--BB, GLM-4v.9
JB, InteraVL2 Intem1 Chat 26B, IntemVLL2-LL LLL2-LLLM33760
Resulls based 100% of the bunchmaark from the OpenOOD v1.5 laaderboan
Only the resultx available from the lenderbunrr shown.
3.4 EXPERIMENTAL SETTINGS
Comparison models. To compare LVLMs from diverse perspectives, we consider both propri- etary and open-source models. For proprictary models, we employ three statc-of-the-art models: GPT-4o (2024-08-06)
(OpenAl 2024). Gemini Pro 1.5 (Reid et al. 2024). and Claude 3.5 Son-
net (Anthropic 2024 For open-source models, we use four models: LLaVA-v1.6-Miistee
(LLaVA-v1.6) .6) (i al. 2024a), GLM-4v-9B (GLM-4v) (GLM et al. 2024). QWEN- VL-Chat (QWEN) (ai et al. 2023).
VL2-26VL2-26BB) and Intern VL2- LLaMA3-76B (InternVL2276 VL2-76B) (Chen et al. 2024). Additionally,
include OpenCLIP (ViT- B-32 pretrained on LAION 2B-s34b-b79k) (Cherti et al. 2023) as a non generative LVLM. To further facilitate comparison between single- and multi-modal models, we also include three single-
modal SOTA vision OoDD models, SCALE (Xu et al. 2024), fDBD (Liu & Qin 2024). Aug-
Mix+ASH (Djuiisic et al. 2023: Hendryeks et al.) 2020).
Benchmark datasets. We evaluate the comparison models on the CIFAR10 and ImageNer200 benchmarks proposed in OpenOOD v1.5 (Zhang al., 2024a). We focus on the standard OoD setting in OpenOOD v1.5, which includes two types of datasets, near- and far-OoD, categorized based on the semantic distance between ID and OoD datasets. Since LVLMs are trained on high- resolution images, main experiments conducted on the ImageNet200 benchmark. We also evaluate the models on the CIFARIO benchmark to assess their scalability with respect to input image resolution. For each benchmark, as y. Due to cost, time, and API rate limits, we use 25% subsets of the benchmarks. A detailed explanation of the benchmark datasets can be found in Appendix B.1
consider the of class names from the ID dataset
Evaluation metrics. We measure the OoDD performance using two commonly used metrics in OoDD: the area under the receiver operating characteristic curve (AUROC) and the false positive rate at the 95% true positive rate (FPR @ @9R%TPR; FPR). We also include the ratio and number of valid responses-thowe those that adhere to the provided guidelines and example format as part of the evaluation metrics. We consider the ability to understand given instructions to be one of the model's key capabilities. Only valid responses are included when measuring performance. We additionally evaluate models on the shared valid responses across all models for rigorous comparison. This result can be found in Appendix B.2
3.5 RESULTS
Tab. D presents the OoDD capabilities of the compared models on the ImageNet200 benchmark. The near- and far-OoD results for SCALE and fDBD represent the average performance across their respective categories. 'All OoD' refers to the performance in distinguishing all OoD inputs, including near- and far-OoD inputs, from ID inputs (i.e., ID vs. all OoD). The results of QWEN-VL- Chat are ommitted due to its exceptionally low ability to follow instructions, with a valid response rate of less than 1%.
http:::/iilytteeenngithtbbbb
5

---

Published as a conference paper at ICLR 2025
100
100
100
80
Frequency
100
InterMLL-77BB
90
90
FPA
60
75
GPT-4o
FPR
AUROC
40
20
33
50
75
85
95
25
75
100
48
255
760
0sD score
size (inbernVL))
Accuracy
(h)
(d)
Figure 4: Further analysis on the ImageNet200 benchmark. (a) FPR across different TPR thresholds, (b) OoD score distribution, (c) OoD detectability based on model sizes, and (d) ID classification
accuracy and OoDD performance of Intern VL2-26B according to the given class order
On both image recognition and OoDD tasks, proprietary models outperform the open-source models in most cases, with reasonable valid response rates. All compared models have more difficulty in detecting near-OoD than far-OoD. The overall performance ranking of the evaluated LVL.Ms gener- ally aligns with the VLM leaderboard" except for Gemini Pro 1.5. In our results, Gemini Pro 1.5 shows better ID accuracy and OoDD performance than Claude 3.5 Sonnet and Intern VL-76B. Claude 3.5 Sonnet frequently generates invalid responses compared to other proprietary models and struggles to detect near-OoD, resulting in worse performance on NINCO and SSB-Hard compared the open- open-source model GLM-4v terms of AUROC, despite achieving 16.65% higher ID accu- racy. InternVL2-266 achieves the best ID accuracy, but has difficulty with OoDD and shows the lowest valid response rate. While there no clear-cut relationship between image recognition and OoDD capabilities, models with better image recognition performance generally exhibit stronger OoDD performance, consistent with the findings of Vaze et al. (2022).
Notably, the proprictary models generally perform on par with or better than the single-modal SOTA OoDD models. In addition, GPT-4o and Gemini Pro 1.5 outperform OpenCLIP both ID classifi- cation and OoDD, showing significantly better performance in OoDD. For more rigorous compar- ison, we evaluate OpenCLIP on images where GPT-4o generates valid responses, as GPT-4o has a relatively lower valid response rate. On the GPT-4o valid query set, GPT-4o still yields better results on both tasks. This demonstrates its superior visual recognition capability and ability to express con- fidence scores through its generated responses. This observation consistent with the evaluation on the shared valid set across all models compared. Detailed results can be found in Appendix B.2
Among the open-source models, one notable observation their FPR of 100%. Our analysis indi- cates that this is caused by extremely biased confidence values, which will be further discussed in Sec. 3c. GLM-4v exhibits the best OoDD performance, while VL2-777 shows the second- best zero-shot image recognition performance among the open-source models. Both models also demonstrate a better understanding of instructions than Claude 3.5 Sonnet. Overall, V1.2 76B shows superior performance in all aspects, including valid response rate, ID accuracy, and achieves higher ID accuracy and comparable OoDD performance to the single-modal SOTA models, alongside the highest valid response rates among all compared models. LLaVA-v1.6 struggles with image recognition, achieving an accuracy of just 2.45%. Most responses from LLa 1.6 incorrectly identify ID images OoD, resulting in poor classification accuracy and OoDD performance.
OoDD performance.
3.6 FURTHER ANALYSIS
Scalability with image resolution. We assess the input resolution scalability of LVLMs using the CIFAR10 benchmark. Most models demonstrate higher performance on the CIFAR10 benchmark than on the ImageNet200 benchmark, with a higher ratio of valid responses. Although the smaller number of class candidates and coarser class contribute to the improved performance, it indicates that the LVLMs handle lower-resolution images effectively. As with the ImageNet200 benchmark, proprietary models consistently outperform open-source models across all datasets, demonstrating superior performance even on fow-resolution images. However, in terms of valid response rate, only GPT-4o shows decrease, excluding GLM-4y which is significantly less accurate overall. This sug- gests that GPT-4o has more difficulty handling low-resolution images compared to high-resolution images. Detailed results can be found in Appendix B.2

---

Published as a conference paper at ICLR 2025
Table 2: Reasoning behind the predictions of GPT-4o and LntemVL2-226 on the ImageNet200 benchmark. Bold indicates the predicted class and its confidence score before applying softmax. Bold italic highlights notable points. Classes with confidence score of 0.0 are omitted.
Prediction
10
OuD
GGourd-rutth Certipode
Cecand-trult trch: capple eagle The
locorppdd:: 40000, seall: 45.00, these classes
GP7.46
6,25) wotth baid casognizahle bo distintive white Avad, yollre baak, and Aanor wingspnn Henee, a ligh confidence svsgted this class
+ The bur more of the given 200 classes directly makih this otser- vation. This the poodiction "muse of fazse classe with high confidence.
revemblle millipede dik stactture
Iateoo 1.3-44
Grazzd-ratt: Buaiet
Gecund frog
fnone of these clases 10000)
The image shows e person ie melesaalt kending ovr e which appoas be is creuzy arez. image prodiction
The imags shows foor with yollow which choracteriate of moe flog The frog perched os leaf, and Its coloratios and ehe pattern are contident with nee frags.
diarly depict thno clesss' confideaco of
100000
Gocund muth NiA (Cyeninuss-O) CJ
ppolificck 5.67, jance 2.00, bald caple 1.22, bottheet 96.72 mann
0.8 gollon namooor 0.14 om- quass closcls aeseubles lonlect, Arand grace plamagy and colorful fuou markings Ofher bird bie norran and bulgbaechr aho shore souw rosesblanee bat the burkkeet clurs has the highest confirence dae the specfle combitation of colors and features
Goverdenat:: N/A (Dpenimens-))) (sune shows purple flwwrs (Allion) which de nol motch gion cleas.
GPT40
OlD
Crouns--ndhh Anmes gamns (NINDO)
Goverdenatt: Crnesl (NINCT))
[some of these clasies 99,80)
InbraVL.5200
(qjider web. 99,39)
* The csinal the inuge is corneat whis is mediues- styal satinn as Africo and part of Middle distiectte blectaufted
iraage slows spidter a green plor with visible yoider's
tackgpoend. Ale, and intricale patien is enidant The confidercs the prediction of "gkkee wel' is high doe so the clear surbiity the web and the spidtr i pasttion wilkin it.
lage
haown
apili and aflen found graslandi and unvanna. The insage shotes an atimal due mactes thes sharacecssiis, making ita "here these classe" with high contideso..
Highly biased confidence scores. One common observation across both benchmarks is an FPR of 100%. As shown in Fig. 4(a) LLaVA-v 1.6 consistently yields 100% FPR across different TPR thresholds. This phenomenon is observed in most open-source models, but not in proprietary mod- els. To understand the cause of this issue, we examine the distribution of OoD scores. Fig. 4bb) depicts the OoD score distribution of VL2-77 and GPT-4o on the ImageNet200 benchmark. We find that the compared models tend to produce highly biased OoD scores, with most OoD scores being either 0.0 or 100.00. A detailed explanation can be found in Appendix B.2 This highlights the need for calibration of confidence scores in the responses of open- models.
Reasoning. We analyze the contribution of the model's visual feature interpretability its OoDD capability by examining the rationale behind its predictions. The prompt used for reasoning can be found in Appendix [B.6] Tab. presents the reasoning results from GPT-4o and Intern V1.2-26B for four different cases on the Image benchmark. The results indicate that the high inter- pretability of visual features contributes to stronger OoDD capabilities. GPT-40 provides detailed descriptions of images, leading to predictions in fine-grained categories. Intern VL2-26B also de- scribes objects in a given image effectively, but not with the same level of detail as GPT-4o. A detailed explanation including the reasoning results for LLaVA-v1.6 can be found in Appendix [B.3]
Confidence scores on ID. We assess confidence scores for ID rigorously explore the expressive- ness of LVLMs in generating confidence estimates. To measure the quality of confidence scores, we employ ECE (Pakdaman Nacini et al. 2015) and AURC (Geif- man et al.] 2019). As shown in Tab. 3 LVLMs exhibit relatively low ECE despite their highly biased con- fidence scores. GPT-4o and OpenCLIP demonstrate good quality in predictive confidence for ID, show- ing lower ECE and AURC compared to other models. Although Intern VL2-76B achieves classification accu- racy comparable to GPT-4o, its ECE and AURC are significantly worse, indicating that Intern VL2--6B as- signs high confidence scores to misclassified inputs.
Table 3:
Comparison on the quality of con- fidence scores in ID inputs. Bold indicates the best performance excluding OpenCLIP.
ImageNet200
CIFARIO
Models
ECE
AURC
FCB
AURC:
OpenCIIP
2.39
21.95
2.39
11.82
GPT-40
2.42
18.64
1.18
10,30
L.aVA-v1.6
4.40
887.86
5.58
48.03
GLM-4
3.24
87.57
0.59
283.69
QWEN
76.82
703.73
InternVL2-76B
5.52
75.25
2.36
37.34
"CB and AURC values are multiplied by 10 and 10 respectively. Lower values are better.
Scaling law in terms of model size. We examine the relationship between OoDD capabilities and model size in terms of the number of parameters. Fig. [ho]sssows AUROC and FPR for Intern VL2. 4B, 26B, and 76B in distinguishing all OoD from IDD on the ImageNet200 benchmark. We observe
7

---

Published as a conference paper at ICLR 2025
Mear-OoD Class
Far-OoD Class
rocket, turboprop, toin-engine eircralt, aiflnnet commuter plane, regioral jet, Dash fuselage, winglet, propellr piane, nose cone
bicydle, skateboard lawnnower washing machine. vacuum deeine laptog, tennis racket, electric guist plano, telescope
STAGE1
Your task is to classify the given image: soipllnne--.ccket.--- bicydee.. these classes) and assign confidence value for your prediction each class.
STAGE2
Your task is to provide classes that are visually similar dissimillar based on the given image...
LVLM
Prediction: airplane
Confidence: (airplane: 95.07, m", rocket: 12.22, birycle: 2.09, none of these classes: 0.00)
Figure 5: Framework of the proposed Reflexive Guidance for OoDD
that OoD detectability improves as model size increases. This observation aligns with our results and the Open VLM leaderboard, where larger models generally demonstrate superior performance.
Class order the prompt. To examine the influence of input class order in the prompt, we evalu- ate Intern V122-26B with different class orders. Detailed experimental settings for class order be found in Appendix B.4 Fig. [(d)]shows the ID classification accuracy and AUROC (ID vs. all OoD). "Random", "Similar First". and "Similar Last" refer random ordering, placing similar classes to input images first, and placing similar classes last, respectively. "Similar First" improves both accu- racy and OoDD performance, while "Random" and "Similar Last" result in a decrease compared to the baseline. Based on these results, we can infer that the model tends to prioritize imputs received earlier over those received later. We assume that this tendency arises from the structure of the data on which Intern V1.2-26B is trained. It is observed that in the text of the image-text pair dataset used for the Intern VI.2-26B training, words referring to objects in the image are predominantly positioned at the beginning of the text (Li al. 2024c Wang al. 2024: Shahroudy et al. (0166). This may have given InterVV22226 an implicit bias regarding word order in the given prompts.
Response failure. Despite providing guidelines to mitigate failures, we still observe failures across all models. A common issue is the mismatch between the predicted class and the class with the highest confidence score. For most proprietary models, the inability to identify the given image is the most frequent failure case. This suggests that proprictary models are more conservative, often declining provide answers when outcomes are uncertain. One contributing factor this is their refusal to respond to harmful or inappropriate inputs. All three proprietary models decline to answer questions about these inputs, while none of the open-source models do. Thus, this characteristic of proprietary models may contribute to their low valid response rates. Failures are not confined to any specific dataset, indicating that they stem from both datasets and the underlying mechanisms of models. Detailed results for failure cases and reject responses can be found in Appendix B.5 (.5)
4 REFLEXIVE GUIDANCE
We introduce a simple and model-agnostic prompting strategy, Reflexive Guidance (ReGuide), to en- hance the OoD detectability of LVLMs. The LVLM'S strong generalization ability has been demon- strated through its performance across various downstream tasks. Therefore, we leverage the LVLM itself to obtain guidance for OoDD from its powerful zero-shot visual recognition capabilities. Fig. illustrates the overall framework of ReGuide, which is implemented in a two-stage process. Details on the prompts for each stage can be found in Appendix C.5
Stage 1: Image-adaptive class suggestions. In the first stage, the LVLM is asked to suggest 2N class names Asux derived from the given image. Specifically, we request two groups of class names: 1) N classes that are visually similar to the image denoted as Apear and 2) N classes that are visually dissimilar or belong to different domains denoted as After In context of ID, A and A can provide classes conceptually corresponding to near-OoD and far-OoD, respectively. If the input image is OoD, Aaux = Aneir U Afar can offer potential ground-truth label or class names closely related to the ground-truth label.
Stage 2: OoDD with suggested classes. Stage 2 follows a similar procedure to the original OoDD evaluation presented in Fig. 3 The major difference is that Agux is employed as auxiliary OoD classes. NegLabel (Jiang et al. 2024) and EOE (Cao et al. 2024) also use auxiliary OoD classes for improving OoDD performance, but they rely texts to obtain negative concepts. In contrast,
8

---

Published as a conference paper at ICLR 2025
Table 4: ReGuide effects on the ImageNet200 benchmark. 'Valid' indicates the valid response ratio from 4, 170 queries. Bold highlights the best performance among the results from each LVLM.
IN/00
NINCO
Now OLD
INatoraiie
Fai OLD Trasrrss
OptiimagoO
All OxD
Vili
(UROC(()
InteraVi.. (a.avaLJ--00 26B
61,01
(2,584)
91.25
62.75
982-75 (58.51
94, 14.38/10
24 52.51
38.00 38.13 50165
28,86/ 25 8G $5, 25
47.01 (479) 75.72
75/12/7212762
+ GeEEtss
(2.914)
$9.58
09.44
02:44 162.17
$5.65/1
25573/55355 53.55
2632 28 (X) 54.72
29.20/
29 20 85.51
39,39 39 39/ 76.10
62 41 /62 (62.64) 65:45
RoGGadd
86.14 (3,592)
18653
1,30 98,12
3.33/303/ 99786
2.04 (004/2.0411
10.24 10.27
1.2eraVE22-7BB
97.26 (4.156)
73189
2,20/
96.43
18.76
14.27
44.46
75:30
+ReeGoidd
95.30 (3.905)
90.33
KJ66 16, 99.35
14.58 (6.A8 187.8
ILIB 99:75 99515
4.06 500 98.38
2/12/ 6612/65.46.0 99396
8.22 /64.16 98.60
GPT-4e
87.58 (3,632)
50.64
1577/147::93.0
20:25
03.5033.50//88 82.25
/ 98.11
5.601
/.21/363 /97.82
15.627 19.34) 89.85
ReGaaide
(9.37)
91.89
0.49/118 199.76
33.97/9.77/// /92.9
(UB9/1700599770
1,32//76 831 95.96
0.18/ /9.66/99966
4J12 25.66 99681
(3124-08-000 1e 208, Ineen\L2LLLMM32200
approach utilizes the LVLM, allowing images to be utilized to obtain negative concepts for auxiliary OoD classes. Given the strong zero-shot visual recognition capabilities of LVLMs, it is expected that OoD input images can be assigned higher confidence scores for Asux than for V, since Anux is derived from the input image itself. The rejection class none of these classes is retained as a fallback in case Aaux does not adequately function as auxiliary OoD classes.
OoD score design. To evaluate the OoDD performance of ReGuide, we employ the same OoD score as in Sec. 3 Since ReGuide leverages auxiliary OoD classes Aaux. compare different OoD scores considered in Diang et al. (2024) and Cao et al. (2024). However, we observe that they yield similar outcomes. The comparative results of the different OoD scores can be found in Appendix C.4
4.1 RESULTS
We evaluate ReGuide with GPT-4o and Intern V1 -122-266///766 on a 5% subset of the ImageNer200 benchmark due to computational and API costs Since ReGuide benefits from the strong image recognition capabilities of LVLMs, we exclude LLa VA-v1.6 and GLVM-vv from the comparison. For this experiment, we set the number of negative class suggestions for each group N to 20. Note that the only difference between the ReGuide prompt and the prompt used in Sec. 3.5 is the addition of classes suggested by Stage 1. All other prompts remain unchanged to ensure controlled evaluation.
As shown in Tab. ReGuide significantly improves various aspects of the LVLM's performance. Surprisingly, ID classification accuracy improves even though Asux includes class names that are visually similar to ID, i.e., Auarr We speculate that considering similar classes collectively assists in classifying images with otherwise ambiguous predictions, rather than hindering classification ability. Similar to fine-grained classification training, considering similar classes together may offer an opportunity compare features between them and reason about their relevance. This analytical process enables the model to categorize harl-so-y images more accurately. We investigate Aaux for dog breed class images where ReGuide correctly changes predictions of GPT-4o. The Aaux for samples with changed predictions include highly fine-gres tne-grained classes. For example, Anux for a toy poodle image includes spoo, cockapoo, and cavapoe, all of which are mixed breeds involving poodle. e. While this is not sufficient to fully support the hypothesis, the inclusion of highly relevant classes may have contributed to improved classification accuracy.
In addition, with ReGuide, InternVL--26B/00 demonstrate performance comparable or superior to the SOTA OoDD single-modat models listed in Tab. 1 We find that the model successfully clas- sifies OoD inputs into one of Aaux. resulting in clearer separation between ID and OoD inputs. As shown in Tab. 5 the ratio of OoD images predicted as non-ID classes (i.c., Aaux + the rejection class for ReGuide and only the rejection class for the baseline) increases with ReGuide, notably by 62.25% on Intern VL.-2 els, with Intern VL2-26B/-76666 and GPT-4o showing average improvements of 33.06%, 22.55% and 6.54% in AUROC, respectively. We further examine how ReGuide alters the predictions of OoD im- ages previously classified as one of the ID classes. ReGuide correctly classifies these misclassified images into Aaux closely related to the ground-truth label. For example, an image of donut pre- viously misclassified as the ID class bagel is now correctly classified as chocolate dipped
Table 5: Comparison of the ratio of OoD in- puts predicted to non-ID classes
Models
Baseline
RReGuide
Intern 21-2666
26 880 2,236)
19.13% (2.651 2,974)
InternVL22-661
(2.089) (,6771)
9147%% (3.297/3.66 3.620)
GPT-4o
83.92% (2,751 / 3,278)
95.97% (2,885/ 3,006)
"GPT-4 (2024 08-06), Miamm/LL2-Intem.... M2.Chhr-ChhBBB... IntennL22
LLaMA3-76B
Near-OoD detection benefits the most from ReGuide across all compared mod-
'The detailed analysis of the inference cost for ReGuide can he found in Appendix C.6

---

Published as a conference paper at ICLR 2025
donut in Asux with ReGuide. This demonstrates the effectiveness of leveraging the image under- standing capabilities of LVLMs. Detailed results of ReGuide can be found in Appendix C.2
We observe that while Intem shows improvements in both FPR @ TPR and AU- ROC, larger models like InternVL2-76B and GPT-4o exhibit improved AUROC but degraded FPR@95%TPR. Upon examining the OoD score distributions and misclassified inputs (i.e., ID in- puts predicted as one of the suggested classes or vice versa; see Appendix CC3) we find that this is due to a small subset of ID images with ground-truth labels that either lack a strong correspondence to the object in the image or correspond to only one of multiple objects present in the image. For these ID imputs, models with powerful visual recognition capabilities, such as IntemVVL2-700 and GPT-4o, suggest more appropriate classes than their given labels. These more suitable suggested classes lead Intern VI.2-76B and GPT-4o to predict ID input images as Ool) classes with high confi- dence, resulting in an increase in FPR @95%TPP It is important to note that this phenomenon is not a negative effect of ReGuide but result of small subset of ID samples with ambiguous or inac- curate labels. This explanation is further supported by the improvement in FPR @ 1090TTTP TPR, which indicates that ReGuide generally enhances the model's ability to handle other ID and OoD inputs effectively. At TPR threshold of 90.0, ReGuide significantly reduces FPR both Intern VLB-76B and GPT-4o across all OoD datasets as the influence of this small subset of images is diminished. A detailed analysis, including the ROC curve, OoD score distribution, and examples of ID inputs with ambiguous or inaccurate labels can be found in Appendix C.3
On the other hand, the suggested classes that are very close to given image helps detect near-OoD as discussed above. By including classes derived from near-OoD images, these images are more likely to be classified into one of the suggested classes rather than ID classes, leading to improved near-OoD detection performance. The significant improvement in near-OoD detection compared to far-OoD detection with GPT-4o+ReGuide highlights this effect.
Image-adaptive vs. text-adaptive. We compare ReGuide with an approach based on EOE (Cao al. 2024). EOE leverages Auux suggested by LLMs: we ask GPT-4o to provide Aneea and A ref- erencing the ID) class names. We denote this approach as GPTT-tett, as shown in Tab. 4 The prompt used for GPT-text can be found in Appendix C.5 The key difference between GPT--ext and ReGuide is that GPT-text obtains class suggestions via ID class names in text form, whereas ReGuide utilizes visual information. As shown in Tab. 4 although GPT-text also enhances InternVL2-26B, the im- provement not significant that achieved by ReGuide. We infer that this difference arises from the aforementioned key distinction. When images are used, the model can provide diverse class concepts based the context of the given image, each image serves unique input. In contrast, when text is used, providing diverse concepts becomes challenging because the ID class set is static. This highlights the effectiveness of image-adaptive OoD class suggestions.
5 CONCLUSION AND LIMITATIONS
In this paper, we address the lack of rigorous evaluation and comparison of the OoDD performance of LVLMs. To tackle this, we establish a framework to evaluate and compare various proprictary and open-source LVLMs. Our comparative analysis provides interesting takeaways into how L.LMSs represent confidence scores through their generated natural language responses. Overall, proprietary LVLMs outperform open-source LVLMs in both image classification and OoDD tasks, demonstrat- ing comparable or even superior OoDD performance relative to SOTA single-modal OoDD mod- els. Additionally, open-source LVLMs tend be overconfident in their response, highlighting the need for confidence calibration. Analyzing the rationale behind LVLM predictions reveals that their visual interpretation capabilities impact their OoDD performance. Based on our findings, we pro- pose ReGuide, a self-guided prompting approach that enhances the OoDD capabilities of L.V.Ms by leveraging self- generaene image-adaptive concepts. Experimental results demonstrate that ReGuide significantly boosts the OoDD performance of both proprictary and open-source LVLMs. We hope our findings contribute to enhancing the reliability of vision-language foundation models for practi- cal deployment.
Limitations of this study include the challenges of exerting precise control over LVLM behavior and the insufficient effectiveness of guidelines to mitigate unintended outputs. Additionally, the image- adaptive nature of ReGuide may lead to suboptimal class suggestions based on image context. A detailed discussion of these limitations can be found in Appendix C.7
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
This work was supported by the National Research Foundation of Korea (NRF) under Grant [RS- 2024-00352184] and (RS-2024-003546755 funded by the Ministry of Science and ICT (MSIT).
REFERENCES
Anthropic. Claude 3.5 sonnet system card, 2024.
Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A frontier large vision-languee model with versatile abilities. arXiv preprint arXiv:2308. 0tt:::238..296000 12966, 2023.
Koby Bibas, Meir Feder, and Tal Hassner. Single layer predictive normalized maximum likeli- hood for distiibbuittiiio detection. In Advances in Neural Information Processing Systems, volume 34, pp. 11 1179-1191 191, 2021.
Julian Bitterwolf, Maximilian Müller, and Matthias Hein. In or out? Fixing ImageNet out-of- distribution detection evaluation. In Proceedings of the International Conference on Machine Learning. volume 202, pp. 2471-2508 2023.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Berbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In Advances in Neural Information Processing Systems, volume 33, pp. 1877-1901, 2020.
Chentao Cao, Zhun Zhong, Zhanke Zhou, Yang Liu, Tongliang Liu, and Bo Han. Envisioning outlier exposure by large language models for -istrr-iiittiibttion detection. In Proceedings of the International Conference on Machine Learning, 2024.
Jiefeng Chen, Yixuan Li, Xi Wa, Yingyu Liang, and Somesh Jha. Atom: Robustifying out-of- distribution detection using outlier mining. Proceedings of the Machine Learning and Knowl- edge Discovery in Databases, pp. 430-445, 2021.
Zhe Chen, Weiyun Wang, Hao Tian, Shenglong Ye, Zhangwei Gao, Erfei Cui, Wenwen Tong, Kongzhi Hu, Jiapeng Luo, Zheng Ma, et al. How far are we gpt-4v? closing the gap to commercial multimodal models with open-source suites. arXiv preprint arXiv:2404 16821, 2024.
Mehdi Cherti, Romain Beaumont, Ross Wightman, Mitchell Wortsman, Gabriel Ilharco, Cade Gor- don, Christoph Schuhmann, Ludwig Schmidt, and Jenia Jitsev. Reproducible scaling laws for contrastive language-image learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2818-2829, 2023.
Mircea Cimpoi, Subhransu Maji, Iasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. De- scribing textures in the wild. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2014.
Andrija Djurisic, Nebojsa Bozanic, Arjun Ashok, and Rosanne Liu. Extremely simple activation shaping for out-of-distribution detection. In Proceedings of the International Conference on Learning Representations, 2023.
Xuefeng Du, Zhaoning Wang, Mu Cai, and Sharon Li. Towards unknown-aware learning with virtual outlier synthesis. In Proceedings of the International Conference Learning Representattonss 2022.
Xuefeng Du, Yiyou Sun, Jemry Zhu, and Yixuan Li. Dream the impossible: Outlier imagination with diffusion models. In Advances in Neural Information Processing Systems, volume 36, pp. 00878-60901. 2023.
11

---

Published as a conference paper at ICLR 2025
Sepideh Esmacilpour, Bing Liu, Eric Robertson, and Lei Shu. Zero-shot out-of- de- tection based on the pre-trained model clip. Proceedings of the AAAI Conference on Artificial Intelligence, 36(6):6568-66) 657766666676,,,, 6576, 2022.
Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, Dustin Podell, Tim Dockhorn, Zion English, and Robin Rombach. Scaling rectified flow transformers for high-resolution image synthesis. In Proceedings of the International Conference on Machine Learning, 2024.
Stanislav Fort, Jie Ren, and Balaji Lakshminarayanan. Exploring the limits of -utto--distribution detection. In Advances in Neural Information Processing Systems, 2021.
Ruiyuan Gao, Chenchen Zhao, Lanqing Hong. and Qiang Xu. Diffguard: Semantic mismatch- guided out-of-distribution detection using pre-trained diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 1579-1589, 2023.
Yonatan Geifman, Guy Uziel, and Ran El-Yaniv. Bias-reduced uncertainty estimation for deep neural classifiers. In Proceedings of the International Conference on Learning Representations, 2019.
Federico Girella, Ziyue Liu, Franco Fummi, Francesco Setti, Marco Cristani, and Luigi Capogrosso. Leveraging latent diffusion models for training-free in-distribution data augmentation for surface defect detection. In Proceedings of the International Conference on Content-Based Multimedia Indexing, 2024.
Team GLM, Aohan Zeng, Bin Xu, Bowen Wang, Chenhui Zhang, Da Yin, Diego Rojas, Guanyu Feng, Hanlin Zhao, Hanyu Lai, et al. ChatGLM: A family of large language models from GLM- 130b to GLM-4 all tools. arXiv preprint arXiv:2406.122 12793, 2024.
Tobias Groot and Matias Valdenegro-Tore Toro. Overconfidence key: Verbalized uncertainty evaluation in large language and vision-language models. arXiv preprint arXiv:2405.02917 2024.
Zhongyi Han, Guanglin Zhou, Rundong He, Jindong Wang. Tailin Wu, Yilong Yin, Salman Khan, Lina Yao, Tongliang Liu, and Kun Zhang. How well does GPT-4v(ision) adapt to distribution shifts? a preliminary investigation. In Proceedings of the International Conference on Learning Representations Workshop on Mathematical and Empirical Understanding of Foundation Models, 2024.
Dan Hendrycks and Kevin Gimpel. A baseline for detecting misclassified and out-of-distribution examples in neural networks. In Proceedings of the International Conference on Learning Rep- resentations, 2017.
Dan Hendrycks, Mantas Mazeika, and Thomas Dietterich. Deep anomaly detection with outlier exposure. Proceedings of the International Conference on Learning Representations, 2019.
Dan Hendrycks, Norman Mu, Ekin Dogus Cubuk, Barret Zoph, Justin Gilmer, and Balaji Lakshmi- narayanan. Augmix: A simple method improve robustness and uncertainty under data shift. In Proceedings of the International Conference on Learning Representations, 2020.
Hochul Hwang, Sunjae Kwon, Yekyung Kim, and Donghyun Kim. Is it safe to cross? interpretable risk assessment with gpt-4v for safety-aware street crossing. arXiv preprint 44X64:2402.06794, 2024.
Xue Jiang, Feng Liu, Zhen Fang. Hong Chen, Tongliang Liu, Feng Zheng, and Bo Han. Nega- tive label guided OOD detection with pretrained vasion-language models. In Proceedings of the International Conference Learning Representations, 2024.
Julian Ratz-Samuels, Julia B Nakhleh, Robert Nowak, and Yixuan Li. Training OOD detectors in their natural habitats. In Proceedings of the International Conference on Machine Learning, volume 162, pp. 10848-10865, 2022.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. Technical report, University of Toronto, 2009.
12

---

Published as a conference paper at ICLR 2025
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng. Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS Symposium Operating Sys- tems Principles, 2023.
Ya Le and Xuan Yang. Tiny imagenet visual recognition challenge. CS 231M, 7(7):3, 2015.
Kimin Lee, Honglak Lee, Kibok Lee, and Jinwoo Shin. Training confidence-calibrated classifiers for detecting duttif--istributiin samples. In Proceedings of the International Conference on Learning Representations, 201 8a.
Kimin Lec, Kibok Lee, Honglak Lec, and Jinwoo Shin. simple unified framework for detecting out-of-distribution samples and adversarial attacks. In Advances in Neural Information Process- ing Systems, volume 31, 2018b.
Bo Li, Kaichen Zhang, Hao Zhang, Dong Guo, Renrui Zhang, Feng Li, Yuanhan Zhang, Ziwei Liu, and Chunyuan Li. Llava-next: Stronger Ilms supercharge multimodal capabilities in the wild, 2024a.
Chunyuan Li, Cliff Wong, Sheng Zhang, Naoto Usuyama, Haotian Liu, Jianwei Yang, Tristan Nau- mann, Hoifung Poon, and Jianfeng Gao. LLaVA-Med: Training large language-and-visst assistant for biomedicine in one day. In Advances in Neural Information Processing Systems, volume 36, pp. 28541 28541-28564, 2023.
Kevin Y Li, Sachin Goyal, Joao D Semedo, and J Zico Kolter. Inference optimal vlms need only one visual token but larger models. arXiv preprini arXi::241 2024b.
Qingyun Li, Zhe Chen, Weiyun Wang, Wenhai Wang. Shenglong Ye, Zhenjiang Jin, Guanzhou Chen. Yinan He, Zhangwei Gao. Erfei Cui, Jiashuo Yu, Hao Tian, Jiasheng Zhou, Chao Xu, Bin Wang, Xingjian Wei, Wei Li, Wenjian Zhang, Bo Zhang, Pinlong Cai, Licheng Wen, Xiangchao Yan, Zhenxiang Li, Pei Chu, Yi Wang, Min Dou, Changyao Tian, Xizhou Zhu, Lewei Lu, Yushi Chen, Junjun He, Zhongying Tu, Tong Lu, Yali Wang, Limin Wang, Dahua Lin, Yu Qiao, Botian Shi. Conghui He, and Jifeng Dai. Omnicorpus: A unified multimodal corpus of 10 billion-level images interleaved with text. arXiv preprint arXiv:2406. 08418, 2024c
Shiyu Liang Yixuan Li, and R. Srikant. Enhancing the reliability of citto--distribution image detection neural networks. In Proceedings of the International Conference on Learning Repre- sentations, 2018.
Litian Liu and Yao Qin. Fast decision boundary based out-of-distribution detector. In Proceedings of the International Conference on Machine Learning, 2024.
Weitang Liu, Xiaoyun Wang, John Owens, and Yixuan Li. Energy- out-of-distributee de- tection. Advances Neural Information Processing Systems, volume 33, pp. 21464-21475, 2020.
Fan Lu, Kai Zhu, Wei Zhai, Kecheng Zheng, and Yang Cao. Uncertainty-awarn optimal transport for semantically coherent -iittri-bitiiitiinnn detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3282-3291, 2023.
Arjun Majumdar, Anurag Ajay. Xiaohan Zhang, Pranav Putta, Sriram Yenamandra, Mikael Henafff Sneha Silwal, Paul Mevay, Oleksandr Maksymets, Sergio Arnaud, Karmesh Yadav, Qiyang Li, Ben Newman, Mohit Sharma, Vincent Berges, Shiqi Zhang, Pulkit Agrawal, Yonatan Bisk, Dhruv Batra, Mrinal Kalakrishnan, Franziska Meier, Chris Paxton, Alexander Sax, and Aravind Ra- jeswaran. Openeqa: Embodied question answering the era of foundation models. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 16488- 16498, 2024.
Yifei Ming, Ziyang Cai, Jiuxiang Gu, Yiyou Sun, Wei Li, and Yixuan Li. Delving into out-of- distribution detection with lision-language representations. In Advances in Neural Information Processing Systems, 2022.
13

---

Published as a conference paper at ICLR 2025
Atsuyuki Miyai, Jingkang Yang. Jingyang Zhang, Yifei Ming. Qing Yu, Go Irie, Yixuan Li, Hai Li, Ziwei Liu, and Kiyoharu Aizawa. Unsolvable problem detection: Evaluating trustworthiness of vision language models. Proceedings of the International Conference Learning Representa- fions Workshop on Reliable and Responsible Foundation Models, 2024.
Jooyoung Moon, Jihyo Kim, Younghak Shin, and Sangheum Hwang. Confidence-await learning for deep neural networks. In Proceedings of the International Conference on Machine Learning, volume 1 119. pp. 7034-7044 2020.
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y. Ng. Reading digits in natural images with unsupervised feature learning. In Advances in Neural Information Processing Systems Workshop Deep Learning and Unsupervised Feature Learning. 2011.
OpenAl. GPT-4o system card, 2024.
Mahdi Pakdaman Naeini, Gregory Cooper, and Milos Hauskrecht. Obtaining well calibrated prob- abilities using bayesian binning, Proceedings of the AAAI Conference Artificial Intelligence, 29(1), 2015.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. Proceed- ings of the International Conference on Machine Learning, volume 139, pp. 8748-8763, 2021.
Machel Reid, Nikolay Savinov, Denis Teplyashin, Dmitry Lepikhin, Timothy Lillicrap, Jean- baptiste Alayrac, Radu Soricut, Angeliki Lazaridou, Orhan Firat, Julian Schrittwieser, et al. Gem- ini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint arXiv:2403.05530. 2024.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei- Fei. Imagenet large scale visual recognition challenge. International Journal of Computer Vision, 115(3):211-252 2015.
Amir Shahroudy, Jun Liu, Tian-Tsong Ng. and Gang Wang. Ntu rgb+d: A large scale dataset for 3d human activity analysis. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2016.
Murray Shanahan. Kyle McDonell, and Laria Reynolds. Role play with large language models. Nature, 623(7987):493-4999 498, 2023.
Yiyou Sun, Chuan Guo, and Yixuan Li. ReAct: Out-of-distr detection with rectified activa- tions. In Advances in Neural Information Processing Systems, volume 34, pp. 144-157, 2021.
Yiyou Sun, Yifei Ming, Xiaojin Zhu, and Yixuan Li. Out-of-distribution detection with deep nearest neighbors. In Proceedings of the International Conference on Machine Learning, volume 162, pp. 20827-20840, 2022.
Leitian Tao, Xuefeng Du, Jerry Zhu, and Yixuan Li. Non-parametric outlier synthesis. In Proceed- ings of the International Conference on Learning Representations, 2023.
Jinglin Tong and Longquan Dai. Out-of-distribution with text-to-image diffusion models. In Pro- ceedings of the Pattern Recognition and Computer Vision, pp. 276-288, 2024.
Grant Van Hor, Oisin Mae Aodha, Yang Song, Yin Cui, Chen Sun, Alex Shepard, Hartwig Adam, Pietro Perona, and Serge Belongie. The inaturalist species classification and detection dataset. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition, 2018.
Sagar Vaze, Kai Han, Andrea Vedaldi, and Andrew Zisserman. Open-set recognition: A good closed-set classifier is all you need. In Proceedings of the International Conference on Learning Representations, 2022.
14

---

Published as a conference paper at ICLR 2025
Naoki Wake, Atsushi Kanehira, Kazuhiro Sasabuchi, Jun Takamatsu, and Katsushi Ikeuchi. Gpt- 4v (ision) for robotics: Multimodal task planning from human demonstration. arXiv preprint arXiv:2311.12015, 2023.
Haoqi Wang, Zhizhong Li, Litong Feng, and Wayne Zhang. Vim: Out-of-distribution with virtual- logit matching. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4921 -921-, 2022.
Hualiang Wang, Li, Huifeng Yao, and Xiaomeng Li. Clipn for zero-shot ood detection: Teaching clip to say no. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 1802-1812, 2023.
Xiyao Wang, Yuhang Zhou, Xiaoyu Liu, Hongjin Lu, Yuancheng Xu, Feihong He, Jaehong Yoon, Taixi Lu, Gedas Bertasius, Mohit Bansal, Huaxiu Yao, and Furong Huang. Mementos: A compre- hensive benchmark for multimodal large language model reasoning over image sequences. arXiv preprint arXiv:2401.10529 2024.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed Chi, Quoc V Le, and Denny Zhou. Chain-of of- thought prompting elicits reasoning in large language models. In Advances in Neural Information Processing Systems, volume 35, pp. 24824-24837, 2022.
Kai Xu, Rongyu Chen, Gianni Franchi, and Angela Yao. Scaling for training time and post-hoc out-of-distribution detection enhancement. In Proceedings of the International Conference Learning Representations, 2024.
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R Narasimhan, and Yuan Cao.
React: Synergizing reasoning and acting language models. Proceedings of the International Conference on Learning Representations, 2023.
Jingyang Zhang, Jingkang Yang. Pengyun Wang, Haoqi Wang. Yueqian Lin, Haoran Zhang, Yiyou Sun, Xuefeng Du, Kaiyang Zhou, Wayne Zhang, Yixuan Li, Ziwei Liu, Yiran Chen, and Hai Li. OpenOOD v1.5: Enhanced benchmark for out-of-distribution detection. In Advances in Neural Information Processing Systems Workshop on Distribution Shifts: New Frontiers with Foundation Models, 2024a.
Jinsong Zhang, Qiang Fu, Xu Chen, Lun Du Zelin Li, Gang Wang, xiaoguang Liu, Shi Han, and Dongmei Zhang. Out-of-distribution detection based on in-distribution data patterns memoriza- tion with modern hopfield energy. In Proceedings of the International Conference on Learning Representations, 2023.
Yimeng Zhang, Xin Chen, Jinghan Jia, Yihua Zhang, Chongyu Fan, Jiancheng Liu, Mingyi Hong, Ke Ding, and Sijia Liu, Defensive unlearning with adversarial training for robust concept erasure in diffusion models. arXiv preprint vrXiv:2405.15234 2024b.
Haotian Zheng, Qizhou Wang. Zhen Fang, Xiaobo Xia, Feng Liu, Tongliang Liu, and Bo Han. Out-of-distribatioo detection learning with unreliable out-of-distributtin sources. In Advances in Neural Information Processing Systems, volume 36, pp. 12110-7212, 2023.
Bolei Zhou, Agata Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralba. Places: A 10 mil- lion image database for scene recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 40(6): 1000::152211000 1464, 2018.
Armando Zhu, Jiabei Liu, Keqin Li, Shuying Dai, Bo Hong, Peng Zhao, and Changsong Wei. Ex- ploiting diffusion prior for oft-of- distrrbutttnnnnnnnnnn detection. Irish Interdisciplinary Journal of Science & Research, 08(02): 2024.
15

---

Published as a conference paper at ICLR 2025
A IMPLEMENTATION DETAILS
All experiments are implemented with Python 3.9 and PyTorch 1.9, using NVIDIA A100 80GB GPUs. For the Intern V62- model, due its significant computational requirements, employ 4 A 100 GPUs with the VLLM (Kwon et al. (202)) library to reduce time overhead. For the remaining models, a single A100 GPU is sufficient to rum the experiments.
B EXPERIMENTAL DETAILS FOR SEC. 3
B. DATASET
We adopt the CIFARIO benchmark and the ImageNet200 benchmark of the standard OoD setting in OpenOOD v1.5. The detailed composition each benchmark as follows:
The CIFAR10 benchmark consists of CIFAR10 (Krizhevsky et al., 2009) as the ID) dataset, CIFARI00 as the near-OoD dataset, and five datasets MNIST, SVHN (Netzer et al.) 2011). Places365 (Zhou et al. 2018), Textures (Cimpoi al. 2014). and Tiny ImageNet (Le & Yang 2013 the far-OoD datasets.
The ImageNer200 benchmark consists of ImageNet200 (Russakovsky et al. 2015) as the ID dataset,
two datasets- -aICO [Bitterwolf et al., 2023) and SSB Hard Vaze et al., 2022) as the near-OoD datasets, and three datasets iNNturalistt Van Horn al. 2018). Textures, and Openimage-O (Wang
et al. 2022 as the far-OoD datasets.
We sample 25% of each dataset, ensuring that the proportion of datasets in each benchmark maintained. During sampling, we maintain the ratio the number of samples for each label from the original dataset. Tables B.I.I and [..122 present the number of images in each dataset for the ImageNer200 and CIFARII 10 benchmarks, respectively. Fig. B...I provides visual representations of this information. We additionally provide the class distributions of datasets for which label informa- tion is available. To improve clarity in visualization, we sampled 200 classes from SSB- which contains large number of labels that would otherwise make the plot difficult to interpret. Fig. 1.222 represents the class distributions for ImageNet200, NINCO, SSB-Hard, and Textures within the Im- ageNet200 benchmark, while Fig. B.1.3 represent the class distributions for CIFARIO, CIFAR 100, MNIST, SVHN, Textures, and Places365 within the CIFARIO 10 benchmark. The lists of sampled images, along with the prompts responses for each sample used in the main experiments are available at https::///// / ht////ttt.b///../ com/ daint Maintlab//Reiie ReGuide
16

---

Published as a conference paper at ICLR 2025
The number of images in each dataset in the ImageNet200 benchmark used in experiments
Table B.1.1:
ID ImageNet200
Near-OoD
Far-OoD Textures
Total
Subset %
NINCO
SSB-Hard
iNaturalist
Openimage-O
100% %
9,000
5,879
49,000
10,000
15,869
5,160
94,908
25% 5 %
2,200
1,304
11,770
2,500 500
3,967 793
1,290 258
23,031
400
249
1,970
4,170
Table B.1.2: The number of images each dataset the CIFARIO benchmark used in our experi-
ments
ID CIFARIO
Near-OoD CIFAR100
Far-OoD
Total
Subset %
MNIST
SVHN
Textures
Places365
Tiny ImageNet
100 %
9,000
9,000
70,000
26,032
5,640
35,195
7,793
162,660
2,250
2,214
17,497
6,504
1,410
8,745
1,948
40,568
The Number of Images (x10')
The Number of images (x100)
Number images
ImagetaC1o
MICCO
Waturaltt
Tbaure
Openteage-O
NINCO
5GB Hard
Mnturelft
Romtrr
Openinage--
Gireeoo
M8R37
SWW
Ratures
Macee33d5
Tiy imagettd
5G0
Class
Class name
Class
25% ImageNer200 benchmark (b) 5% ImageNet200 benchmark
(c) 25% CIFARIO benchmark
Figure B. 1.1.1: The number of images in each dataset for each benchmark. Blue bars represent the full datasets, while red bars indicate the sampled datasets.
17

---

Published as a conference paper at ICLR 2025
The No. moerrerr of imaigns
4
Class name
(a) ImageNer200
The Number of imagges
150
120
50
amphis mears
aphaniarmannn flavcus
araniuu gonma
aetorephat getapgnrims
batrackieep alter uatus
callee
chicken quesediin
pttcher
Carollptumaa orgicaudaaa
cup ca
eattngga califaniaa
condroingg lagoss fumhorti coruts
dodf
eepply mater disperes
spithela inccommn's
erysmum mncsromuun
(fonst paih
reno caccacuus pmecaccssploo plaza
fre acinguu
firowotd
french fites
heemulo scurc high heess
happor-n heppoten
logton carr rectvopsm
lallyy colooaat repuntt su bus
lapt aetogiise phy taposs
moro
microoylie wishe burerbrrr meree tamar marrutatee
ecepu tu
salcciiin halus furus
colitoo domidun
Eonoooun cartiderd
sky
Seaeen sll
ser sin
fildd
cataral
inturox
Class
(b) NINCO
The Number of imaigss
20
Class name
(c) HSrd
The Namber of imagges
50
bandud
biarcht
braveed
bumoy
choquered
cratked
croshhateeee
cryzalline
duttacd
librous
miy
qauzy
unid
gropoeed
Interaaced
bacelike
marb to
mattoo
meshee
pablloy
perforaled
pitted
preaterd
dotted
porous
pothokkd
scalyy
varre
(d) Textures
Figure 1.1.2: The class distribution in each dataset for the ImageNet200 benchmark. Blue bars represent the full datasets, while red bars indicate the 25% sampled datasets.
18

---

Published as a conference paper at ICLR 2025
The Number of Images
The Number of images
8000
The Number of images
5000
800
6000
4000
600
3000
4000
400
2000
200
1000
Class name
Class name
Class name
(a) CIFARIO
(b) MNIST
(c) SVHN
The Number of images
20
Class name (d) CIFARTO0
The Number of imiggns
2
ges
Class name (c) Places365
The Number of iimages
120
50
craccs
bub
chocuered
creshhachees
pod
P3
serfecal
Textures
Figure B.1.3: The class distribution in each dataset for the CIFARIO benchmark. Blue bars represent the full datasets, while red bars indicate the 25% sampled datasets.
19

---

Published as a conference paper at ICLR 2025
B.2 COMPREHENSIVE EXPERIMENTAL RESULTS
Tab. B.2.1 and B.2.3 provide comprehensive results for experimental results on the ImageNer200
and CIFARIO 10 benchmark, respectively. For the CIFARIO benchmark, we employ only GPT-4o as the proprietary model.
CLIP on GPT-40 valid query set. We evaluate OpenCLIP on the inputs where GPT-4o generates valid responses to analyze whether GPT-4o's better results are influenced by input bias, such as rejecting difficult inputs. As shown in Tab. [.2.T] OpenCLIP (GPT-40 valid) shows slightly better overall performance compared to its performance on the entire 25% subset. Although there is an improvement, GPT-4o still outperforms OpenCLIP on both tasks. This demonstrates GPT-4o's o's su- perior visual recognition capability and its ability to produce better confidence scores through its generated responses.
Comprehensive results on the ImageNer200 benchmark. For clarity, the full names of the compared models are provided in the footnotes below the table." + and 1 mean that lower and higher values are better, respectively. The AURC values are multiplied by 1 and the other values are percentages. 'Valid' indicates the ratio of valid responses out of total of 23, 031 image-prompt queries. The number in brackets represents the number of valid responses. Bold represents the best performance among generative LVLMs.
Table 1.2.1:
ID
Net-OOD
Redtan Testure
InuxeNer200
NINCO
NNataralii
Openinage
Modul
Valid
ADC
BCB
AURC
AUROCC
Open IP
10000)(23.041
R7.41
2:59
11.50 04
62.29/88.5
71.4k. 76 16
42: 02:96/900 90
43.8% 80-62
47.47/ 90 m8
61/42/81
Op (GPT-6o
88 (19.689)
58.05
2.65
19.51
59.55
70.311
99:34
45.65/ 90.10
44.80/90
$4.112
GPT-4
$5.40
18.64
22.30
81.41
2,06 99..58
7.45 96.85
3.76
23,50/
Clande 35 Semmet
01.94 (18 515)
86.05
7.30
$2 72
78-411 58 00
9:23 19695
10.37 94
18.11 89
.6.01 /71
Genini Pro 1.5
91.92 (21.170)
88.84
3,09
30.56
21.55 / 59.00
55.24/ 77.40
97.73
95.61
5:25 196.45
32.97/ 85.77
T1 (16, 6980
2.45
4.40
86
100000
100/00//11115
100000 49:23
1001000 90.1
10000
100/00/835.01
10010)
100000
OWEN
V1.24
91 (16, 584)
63.00
12.08
90.8% $7.49
67.0760000
44.99 76-42
79 19 59 14
38. M6.90
Intera VL2 300
$2.68 (34.436)
90.23
9.01
91.98
$2.59 558.22
/4.2155.551
36.09181 18.25
85.56
00.99/1 74L
75,95/6163
Intera VL2-768
97.36 (22.424)
88.30
5.32
15.25
100000 (2.22
100,00 00000/62..30 52..99
100000995.59
100,00991.0 91,42
100,00
/9010/99012
100,00 74414
OpenCCIP- V17---2-- GPT-4o (2004-88-6). 11.VAA-1.6.Miaaa 4LM-4-08, OWEN-VL--Cu (9.8)) InsmVV12-96633mmii---
112MA1-758
Results on shared valid query set across LVLMs To enable a more rigorous comparison, we evaluate the models using the shared valid query set. Tab. presents the 25% subset of the Im- ageNer200 results using this shared valid set. QWEN is excluded due to its exceptionally low valid ratio. The overall performance trends are consistent with those observed on the models' individual valid query sets. GPT-4o outperforms all other compared models, and proprictary models demon- strate superior performance compared to open-source models, aligning with the results observed with their own valid query sets.
Table B.2.2: Comprehensive results the shared valid query set of the ImageNet200 benchmark. For clarity, the full names of the compared models are provided in the footnoees below the table. 4 and mean that lower and higher values are better, respectively. The AURC values are multiplied by 10², and the other values are percentages. Bold represents the best performance among generative LVLMs.
ImageNec200
ID
NINCO
Newr OaD
SSB Hard
INaturalist
OaD Textures
Opezimagee 0
All OwD
Models
Valid
ACC (f)
ECE
AURC(())
FPR-0955 <Jy AUBOC (1)
OpoCLIP
87.06
20.52
65.42/83.74
73.01
42.00
41,33/91,44
45.67/91.55
63.06 /8.09
GPT-4o
91.74
2.79
14.86
1831//11.99
/7.96/77.15
1/2//97.00
6,09 /95
2.56/ /9.55
25,60 /66.4
Clande 3.5 Summet
67.64
7.56
75.22
58 66.34
K3.51
95.51
9:04 /94 81
18.91/80. 14.34
$1.95 71.09
Gamini Pro 1.5
(5.27))
90.48
2.53
23.71
25.76/ /9.51
62.37/6933
69.57
0 18.18
6.27 /95
4.88 1.8./90 96.69
35.45 15545/82.00 82.00
ILaVA-v1
3.21
4,07
$60.46
100000 52.92
100,00/440 40.22
100000/ $0.02
100,00 90 80
100.00 48: 19
100,00/ 50.57
GLM-4y
69.04
2.04
81.70
100000 72.70
100.00 66.14
1000000 $2.98
100,00
83.65
100.00 (::00/822:00 92.90
100.00/ 74.02
InternVL2-2611
91.28
8.37
88.29
81.91/55 55.87
96.00/
51.88
83.06 35.5
21.77
0177/88 94
433.33 78815
69
InternVL2-7e1
90.00
6.42
85.00
61.69/ 17.38
83.00
56.66
2.20/97.12
9/78/19933
13.90/91.22 126
50.44 23.00
"OpenCLPP WT 8-32 40
(2004 (8) 061,
L.L.A v1.6 Misral 70, GLM dv 918 Bw..n/ 9hi-3
mini 403, InteraV1.22 Intern..22222... Chat 260,, InterV112 LLAMA33768
Scalability with image resolution. As shown in Tab. B.2.3 GPT-4o consistently outperforms Open- CLIP in both image recognition and OoDD tasks, particularly excelling in OoD distinguishability with AUROC of 98.26% and FPR of 2.61%. However, GPT-4o returns the lowest number of valid responses among all compared models. QWEN processes low-resolution images better than high- resolution ones, though its overall performance remains poor. As shown in its AUROC on Ool
20

---

Published as a conference paper at ICLR 2025
datasets, QWEN assigns higher confidence scores to ID classes for OoD imputs than for ID inputs. LLaVA-v1.6 processes low resolution images better than high resolution images. However, GLM-4v performs worse on the CIFARIO benchmark than on the ImageNet200 benchmark. including valid response rate. One notable observation is that V1.2-266 outperforms Intern VL.2-66B overall on the CIFARIO benchmark.
Highly biased confidence scores. The compared models tend to produce OoD score either 0.0 or 100.00, as shown in Fig. [bb] This indicates that the models typically classify input images as either ID or OoD with extremely high certainty, e.g, assigning 100.00 to only one class and 0.00 to all other classes. VLt VL22-66 exhibites this tendency more strongly than GPT-4o. It produces almost no responses with OoD scores between 0.0 and 100.00 whereas GPT-4o generates some responses in this range.
Table B.2.3: Comprehensive results on the CIFARIO benchmark. For clarity, the full names of the compared models are provided in the footnotes below the table." and f mean that lower and higher values are better, respectively. The AURC values are multiplied by 10 and the other values are percentages. 'Valid' indicates the ratio of valid responses out of total of 40, 568 image-prompl queries. The number in brackets represents the number of valid responses. Bold represents the best performance among the compared models among generative LVLMs.
CEPARIO
ID
NeD CIFARIOO
MNIST
SVHN
Touaz
Placa965
Tizy ImazeNee
All Onf
Models
Veld(())
AUREII)
FER TVW (337 AUROIC (1)
Open IP
10:00:4001100
45.33
15.15/91
197787
0088/00 /97.99
45 194445
41 993.99933
46.59/9265
26.76
GPT-4o
66.42 (26340)
05.34
1.18
10.30
5.49 96.64
0.55 99.32
2.67 19819
0.51:99.55
5 96.82
/1.3997.3
2 99.26
LLaVA-LA
83.51
100,00 76
GLM-4+
73.81 (40.095)
35.96
0.39
283.09
10800/ 57.
100.00 68.34
100000 68.31
100,001 68 30
1000001 66. 21
100.001572
103003/67799
OWEN
72.81 (29 537)
16:08
76.87
703.73
100.00 11 91
61 1919977760 77
594 63 05
73 TM /1377
96. 17777-7700 05
56.95 75-21
95.01/ 13
(1
81
15.00
113.73
10.31
02 (93-79
17 19105/91::4
44 93-55-8111
100100.78
16.87 (387/194.0
99.95 (01,549)
98.93
100.00
100.00/00 95.90
100,00/ 96
100,00 95
100,00/ 90.00
100/00/90014
100.00: 24.29
OpeeCLIP VTT-B
GPT-4o (2024-05 06) WA 6-M G1.M-4x-900
OWEN-VI VC.. (9:58), Intar M0 11A.:N11222L..AA5-22000
B.3 DETAILED RESULTS FOR REASONING IN SEC. 3.6
In Sec. 3.6, we provide the reasoning results of GPT-4o and InternVL2-266 on the ImageNet200 benchmark. As shown in Tab. GPT-40 demonstrates strong visual recognition abilities. It provides detailed descriptions of images, including aspects such as color, object shape, and relationships with other classes, leading to predictions in fine-gran fineggrained categories. For example, GPT-4o correctly identifies the bald eagle image by describing its distinctive features, such as the white head, yellow beak and large wingspan. For the image classified as lorikeet, GPT-4o provides highly detailed descriptions of the object's visual features, including its vibrant green plumage, and arrived at its conclusion by comparing these features with other possible classes among the provided options, such as toucans or gol df i nccee. Due to its impressive visual recognition ability, GPT-4o'ss misclassified predictions often involve classes closely related the ID class, such as cent c and mi 11 ipede.
pede
In contrast, the reasoning provided by InterVVL2-266 is less detailed than that of GPT-4o. Its descriptions are much more general, and at times, its explanations are based on common knowledge about the object in the image. For example, for an image of araneus genma, InternVL2-2611 describes it as spider on a green plant with a web. Similarly, for an image of a caracal, the explanation provides mainly general information, such as the species' native land or habitat. The analysis results indicate that high interpretability of visual features contributes to stronger OoDD capabilities.
We additionally present the reasoning results of LLaVA-v1.6 16 on the same benchmark to analyze its low performance, despite reporting valid response rates comparable to other models. As shown in Tab. B.3. LLaVA-v1.6 shows much rough image interpretation than GPT-4o. It primarily predicts images into coarse-level categories, such as birds or flowers, and its explanations rely more on common sense than visual features. This results in incorrect predictions, as evidenced in the cases where actual ID -> prediction OoD. It suggests that better interpretability of visual features is crucial for enhancing OoDD capabilities.
21

---

Published as a conference paper at ICLR 2025
Table B.3.1: Reasoning behind predictions of LLaVA-v1.. on the ImageNet120 benchmark, exclud- ing cases where the actual input is OoD but the prediction is ID. Due to the low valid response rate, none of LLaVA-v1.6's responses corresponds to this case. Bold highlights the predicted class and its confidence score before applying softmax. Bold italic highlights notable points. We omit some classes, such as those with a confidence score of 0.0.
Prodiction
ID
OnD
Ground uptt: Oszich
[estrich: nore thos clacos: 0:00;
Ground Jusco
imise prowided outice. a lorge flighttzs fning
(osos (hase classs: 100.00)
for
The image prosided is Mind, which does not bellong any the ziren 200 classes. Theeefore the peodiction "none of therr clasre with confidence 1001
disiective apperracc. inch not fisk, repile, munmal, hind inseet, so does not belong any the other classes pecsided. Theeetoee the prediction is "catrich" with confidecce of 100.00.
Actual
Groand- (Openinag--O) of these classes: 99.99 other classes:
0nl
N/A
0.01)
The image prowiee is flowerr which does not be to any of the giver 200 classes Therefore, the prediction foore thos claous with confedence 100
B.4 IMPLEMENTATION DETAILS FOR CLASS ORDER
To examine the influence of class order in the prompt, we evaluate Intern with three dif- ferent strategies: 1) Random, 2) Similar First, and 3) Similar Last. For Random, we randomly shuffle classes for every query. For Similar First and Similar Last, we group 200 classes into broad categories based on their conceptual similarity (e.g., aquatic animals, birds, domestic dogs, and vehi- cles). We request GPT-4o to group the given classes based on their conceptual similarity. Tab. B presents the class grouping results by GPT-4o, which are used for class order experiments Sec. 3.6 The prompt used for grouping classes can be found in Fig. B.6.3
By leveraging class groups, Similar First places the group containing the ground truth label of the input image at the beginning, while Similar Last places that group the end. For OoD input im- ages, we randomly shuffle the class order, since ground-truth label information is not available for OoD inputs. We evaluate these class order strategies Intern VL2-26B using 5% subset of the ImageNet200 benchmark, sampled in the same manner as the 25% subset.
Table B.4.1: Groups and corresponding classes suggested by GPT-40
Group
Classes
Aquatic Animals
goldfish, great white shark, hammerhead, stingray, eel, clown fish, puffer fish, grey whale, killer whale, sea lion. jellyfish, starfish. lobster. henmit crab, news, axolot. tree frog, snail
Birds
bald engle, vulture, flamingu, american egret, pelican, king penguin, duck, goose, black swan, hen, ostrich, peacock, lorikeet, bummingbird, toucan, goldiinch, junco
Reptiles & Amphihians
iguana, African chameleon, cobra, scorpion, tarantula, centipede
Insects
ladybug. fly, boc, ant, grassbopper, cockroach, mantis, dragonfly, monarch butterfly
Domestic Dogs
chibuahua, shih tzu, yurkshire lerrier, bostom lerrier, scottish terrier, west highland white terrier, pug. pomeranian, toy poodle, heagle, basset hoand, cocker spaniels, french halldog, pemhroke welsh corgi. afghan hound, bloodhound, weimaraner, golden retriever, labrador retriever, collie, border collie, notlweiler, german shepherd dog. boxer, saint berardd, husky, dalmatian, chow chow, standard poodle, italian grey- hound, whippet
Wild Mammals
timber wolf, hyena, red fox, tabby cat, loopard, snow loopard, lion, tiger, cheetah, wood rabbit, porcupine, squirell heaver, guinea pig, zehra, pig. hippopotamus, bison, gazelle, llama, skunk, hadger, polar bear, orangutan, gorilla, chimpanzec, gibbon, baboon, meerkat, panda, koala
Vehicles & Transportation
ambulance. jeep, school bus. pickup truck, tractor, steam locomotive, fire engine, canoe, pizate ship. schooner, submarine, space shutte,, military aircraft, parachute
Weapons & Military
assuult rifle, cannon, missile, revolver. tank
Sports Equipment
basketball, rugby ball, soccer ball, tennis ball, scuba diver, bascball player
Musical Instruments
accordion, electric guilar, flute, grand piano, hammenica, hurp, saxophone, trombone, violin
Food & Drinks
cream, bagel, pretzel, cheesehurger, hotdog, pizza, burrito, espresso, cabbage, hroccoli, eucumber, hell popper, mushroom, Grancy Smith, strawberr, lemon, pincapple, banana, pomogranate, acort
Household
backpack, wheelharoow, bathtuh, beer glass, binoculars, birdhouse, bow lie. broom, bucket, cauldron, can- dle, mobile phone, cowboy hat, gasmask, joystick. lab cost. lawn mower, lipstick, mailbox, mitten. sandal, shield, spider web, scarf, vase, wine bottle
Landmarks Structures
barn, lighthouse, carousel, castle, volcano
Tools & Machines
hammer, hatchet, guillotine
22

---

Published as a conference paper at ICLR 2025
B.5 DETAILED RESULTS FOR FAILURE CASES
The number of each failure case. Tab, B.5.1 summarizes the frequency of each failure case. Com- failure
can be categorized into six groups: mismatch between the predicted class and the class with the maximum confidence score, 2) failure to provide confidence scores for all classes, 3) responses that are not structured the given format, 4) responses indicating an inability iden- tify the given image, 5) inclusion of classes not listed among the given, and 6) class duplication. For Case we post-process the responses by assigning a confidence score of 0.0 to the missing classes, 200 classes can be considered substantial number to process effectively. Each failure case occurs uniformly across all datasets.
Table B.5.1: The number of each failure case each model with the 25%
ImageNet200 benchmark
Models
(1)
(3)
(5)
(6)
GPT-4o Claude 3,5 Sonnet
105
1
2
3,225
10
0
28
1
0
2,550
1,940
0
Gemini Pro 1.5
1558
208
22
LLaVA-v1.6
6,533
2
0
0
0
GLM-4v
775
1,419
339
0
QWEN
180
2,937
13,160
7
5,857
InternVL2-76B
212
0
0
353
42
GPT-44 12024-08-00 -6), LLaVA-v1 1...--A--..--M-rrall.. ---sstral--BB GLM-4v-9B, QWEN-
VL-Chat (9.6B),
Example responses for each failure case. Tab. B.5.2 provides two examples for each failure case. Case omitted as it is self-explanatory. Bold indicates the elements that contribute to the response being classified as failure case.
Table B.5.2: Example responses for each failure case
Fallure
Example
1) A mismatch between the predicted class and the class with the maximum confi- dence score
Prediction: 'great white shark' Confidence: ('goldfish': 0.0. 'great white shark": 0.0, 'hammerhead': 100.0, ..., 'ostrich": 12.1, "none of these classes'; 0.0}
Prediction: 'goldfinch' Confidence: ('goldfish': 0.0, 'great shark': 0.0, 'junco':
99.98. 'ostrich': 12.1, of these classes": 0.0)
Example: goldfish
3) Responses that are not structured according to the given format
Prediction: {goldfish, great white shark. hammerhead, stingray, hen, ostrich, goldfinch, junco, bald eagle, vulture, newt, axoloti, tree frog, iguana, African chameleon, centipede, sea lion, none of these classes)
4) Responses indicating an inability identify the given image
I'm sorry. can't 'tn't identify specific elements in the image beyond what 've provided in the description.
I apologize, but do not feel comfortable classifying describing this image in detail.
5) Inclusion of classes not listed among the given
Prediction: basketball player
Prediction: birthday
Confidence: ('goldfish': 0.0, 'great white shark?: 0.0, ..., "ostrich:: 99.9, ..., fos- trich': 12.1, 'none of these classes': 0.0)
6) Class duplication
Confidence: ('goldfish': 0.0, 'great white shark': 0.0. 'boxer": 9.5, ..., 'boxer":
10.0, 'none of these classes": 0.0)
Examples for inappropriate content. Tab, B.5.3 shows examples of images and their responses from each proprietary model. All inappropriate or harmful images found are from SSB Hard.
23

---

Published as a conference paper at ICLR 2025
As discussed in Sec. 3.6 All three proprietary models decline to follow given prompt if input image contains such content, instead they return responses with "I'm sorry, can't help with that" or "I apologize, but I do not feel comfortable classifying describing this image in detail". Claude 3.5 Sonnet rarely generates responses corresponding Case 4, but its frequency increases for the inappropriate classes in SSB Hard, such undies or uzi. However, all open-source models generate predictions for them.
Table B.5.3: Examples of inappropriate images and the corresponding refusal responses from pro- prietary models
Image
Model
Response
GPT-4o
I'm sorry, I can't help with that.
Claude 3.5 Sonnet
apologize, but I don't actually image attached analyze this conversation.
undies
Gemini 1.5 Pro
cannot assist with this request as it is sexually explicit in nature.
GPT-4o
I'm sorry, I can't help with identifying or analyzing this image.
Claude 3.5 Sonnet
apologize. but feel comfortable classifying describing this image in detail.
1
The effectiveness of guidelines and examples. We examine the effects of guidelines and examples mitigate invalid responses using Intern VL.2-76B the 5% subset of the ImageNer200 bench- mark. Baseline refers to a prompt that includes only the basic elements: a task description and an explanation of the rejection class, As shown in Tab, B.5.4. the inclusion of guidelines and examples results in nearly 90% reduction in failure cases compared to prompts without them. Examples help address all types of failure cases, while guidelines are especially effective in reducing Failure Case (2), where responses do not follow the required format.
Table B.5.4: Ablation results on the effects of guidelines and examples in reducing failure cases with VL2e77L22-7B
Prompt
(1)
(2)
(3)
(4)
(5)
(6)
Total
Baseline
193
886
201
0
1,280
ID example
18
731
100
0
849
Guidelines
493
128
631
+ OoD example
26
34
79
8
147
24

---

Published as a conference paper at ICLR 2025
B.6 PROMPT FORMAT
We provide the full prompt formats used for experiments in Sec. 3.5 and 3.6 In the prompt for- mar, yellow, green, red, blue boxes indicate task description, explanation of the reject class, guidelines, and examples for response format, respectively.
B.6.1 PROMPT FOR BASELINE SEC.
First, we define and explain the task that the LVLM will serve. We provide all in the prompt format and ask the LVLM to answer the prediction and confidence scores each for x. Note that the sum of the confidence scores
Fig. B.6.1 illustrates the baseline prompt used for experiments in Sec. 3.5
limited to constant value since empirically observe that the removal of the limit leads to enhanced outputs including much proper confidence score representations and improved prediction, particularly for OoD imputs. The red text indicates that the contents vary depending on the choice of benchmark.
Your task to classify the given image into one of these C classes: {ID classes, none of these classes and assign confidence values your prediction
classify the given image other classes, you are not sure whether an image belongs to one of the given other C classse, or if you think you need other classes other than the given other classes to classify the given image.
of these classes': you cannot classify given image any of the given
The following are guidelines for your respanse. Please respond according to these guidelines.
You should provide your confidence for each class between 0.00 and 100.00.
The confidence value can be expressed as a float.
The class You've assigned the max confidence should be your prediction.
Your confidence should be above
one the classes given. is not allowed assign
confidence to every class.
should provide your confidence the Your prediction should be in the given C classes. You are not allowed to predict the given ima ge into any other given classes.
given,
your prediction hone
classes'.
classes than the
There is hone of these classes' in the given C + 1 classes. This means that your prediction should be hone of these cla sses's your prediction is not in of (ID classes ).
Strictly follow guidelines
Here examples of your response: The first ane is when given image predicted into the classes classes The second one is when given image predicted into 'none of these classes'.
respond following examples
Example when image predicted into classes classes }
Prediction: exomple prediction
Confidence: classes: example conf (idence)
Example 2) when a given image predicted into 'none these classes'
Prediction: none these classes Confidence: (ID classes: example con fidence, none of these classes: example con (nffddece)
These above examples are confidences for two different images. When you pruvide your a answer, your answer format should be sa same as between dashed examples. Strictly achere the examples format provided
deviate from the format.
Figure B.6.1: Prompt used for experiments in Sec. 3.5
25

---

Published as a conference paper at ICLR 2025
B.6.2 PROMPT FOR REASONING
Fig. B.6.2 illustrates the prompt used for reasoning in Sec. 3.6 The red text indicates that the contents vary depending on the choice of benchmark. The text highlighted in yellow corresponds to the parts added for reasoning in the baseline prompt.
Your classify the given image confidence values for your prediction to each class, and give your rationale behind your prediction and confidence va values.
classes: classes,
assign
You can classify the given image into 'one of these classes': you cannot classify given image into any of the given other classes, if you are not sure whether an image belongs to one of the given other C classes, or if you think you need
other than the given other
classify the given image.
The following are guidelines for
response. Please respond according to these guidelines.
should provide your confidence each class between 0.00 100.00.
The confidence value can be expressed as float.
you've assigned the confidence should your prediction. Your confidence should be above 0.0 in at least one of the C + classes given. It is not allowed to assign 0.0 confidence
every class.
should provide your confidence the classes given, even your prediction is hone these classes'. Your prediction should be the given C classes. You are not allowed to predict the given ima into any other
given classes.
There is 'none of these classes' in the given C' classes. This means that your prediction should be hone of these
classes' your prediction not
of (ID classes
Strictly follow the guidelines above.
examples your response. (ID classss The second one is when a given image predicted into 'none of these classes'.
a given image predicted given
Please respond with the following examples format:
Example when given image is predicted given classes (ID classes }
Prediction: exomple prediction
Confidence: ( ID classes: example con fidence
Reasoning: example reasoning
Example a given image predicted into 'none classes'
Prediction: none these classes Confidence: classes: example con confidence, none of these classes: example confid (nneeeee)
Reasoning: etample reasoning
These above examples confidences for two different images. When you provide your answer, your answer format should be sa same as the format between the dashed lines examples.
Strictly
the examples provided
Figure B.6.2: Prompt used for reasoning in Sec. 3.6
26

---

Published as a conference paper at ICLR 2025
B.6.3
PROMPT FOR CLASS ORDER
Fig. B.6.3 illustrates the prompt used for grouping classes in the ImageNet200 dataset based on their conceptual similarity.
have 200 category names, and my goal is to group these categories into broader concepts for easier management. Here are the 200 category names have.
Goldfiih, great white shark, hammerhead, stingray, hen, estrich, goldfinch, junco, bald eagle, vulture, newt, axoloti, tree frog, Iguana, African chameleon, cobra, scorpion, tarantula, centipede, peacock, lorikeet, hummingbird, toucan, duck, goase, black swan, koala, jellyfish, snail, lobster, hermit crab, flamingo, american egret, pelican, king penguin, grey whale, killer whale, sea lion, chihuahua, shih tzu, afghan hound, basset hound, beagle, bloodhound, italian greyhound, whippet, weimaraner, yorkshire terrier, boston terrier, scattish terrier, west highland white terrier, golden retriever, labrador retriever, cocker spaniels, collie, border collie, rottweller, german shepherd dogg boxer, french build saint bernard, husky, dalmatian, pug, pomeranian, chow chow, pembroke welsh corgi, toy poodle, standard poodle, timber wolf, hyena, red fox, tabby cat, leopard, snow leopard, lion, liger, cheetah, polar bear, meerkat, ladybug, fly, bee, ant, grasshopper, cockroach, mantis, dragonfly, monarch butterfly, starfish, wood rabbit, parcupine, squirel, beaver, guinea pig, zebra, pig, Nppopotamus, bison, gazelle, llama, skunk, badger, orangutan, gorilla, chimpanzee, gibbon, baboon, panda, eel, clown fish, puffer fish, sccordion, ambulance, assault rifle, backpack, bam, wheelbarrow, basketball, bathtub, lighthouse, beer glass, binoculars, birdhouse, bow tie, broom, bucket, cauldron, candle, cannon, cance, carousel, castle, mobile phone, cowboy guitar, fire engine, flute, gasmask, grand piano, guillotine, hammer, harmonica, harp, hatchet, jeep, joystick, lab coat, wn mower, lipstick, mallbox, missile, mitten, parachute, pickup truck, pirate ship, revolver, rugby ball, sandal, saxophone, school bus, schooner, shield, soccer ball, space shuttle, spider trombone, vase, violin, military aircraft, wine bottle, ice cream, bagel, pretzel, cheeseburger hatdog, cabbage, br broccoli, cucumber, bell pepper, mushroom, Granny Smith,, strawberyy lemon, pineapple, banana, pomegranate, pizza, burrito, espresso, volcano, baseball player, scuba diver, acorn
electric
locomotive, scarf, submarine, tank, tennis ball, tractor,
Here is the initial grouping attempted.
Goldfish, great white shark, hammerhead, stingray, grey whale, killer whale, sea lion,
hen, ostrich, goldfinch, junco, bald eagle, vulture, flamnigo, american egret, pelican, king penguin, jellyfish, snail, newt, axoloti, tree frog, iguana, African chameleon, cobra,
eel, clown fish, puffer fish, starfish,
lobster, hermit crab, scorpion, tarantula, centipede,
peacock, lorikeet, hummingbird, toucan, duck, goose, black swan,
chihuahua, shih tzu, afghan hound, basset hound, beagle, bloodhound, italian greyh whippet, waimaraner, yorkshire terrier, boston terrier, scottish terier, west highland white terrier, golden retriever, labrador retriever, cocker spanlels, collie, border collie, rottweller, german shepherd dog, boxer, french bulldog, saint bernard, husky, dalmatian, pug, pomeranian, chow chow, pembroke welsh corgi, toy poodle, standard poodle,
timber walf, hyena, red fax, tabby cat, leopard, snow leopard, lion, tiger, cheetah, meerkat, lady, fly, bee, ant, grasshopper, cockroach, mantis, dragonfly, monarch butterfly, wood rabbitt porcupine, fox squirel,, beaver, guinea pig, zebra, pig, hippopotamus, bison, gazelle, llama, skunk, badger, polar bear,
orangutan, gorilla, chimpanzee, gibbon, baboon, panda, koala, ambulance, jeep, cance, school bus, pickup truck, pirate ship, space shuttle, submarine, tractor, parachute,
assault rifle, cannon, missile, revolver, tank, military aircraft, backpack, bam, wheelbarrow, bathtub, lighthouse, beer glass, binoculars, birdhouse, tie, broom, bucket, cauldron, candle, carousel, castle, mobile phone, cowboy hat, fire engine, gasmask, guillotine, hammer, basketball, rugby ball, soccer ball, tennis ball, accordion, electric guitar, flute, grand piano, harmonica, harp, saxophone, trombone, violin, hatchet, joystick, lab coat, lawn mower, lipstick, mailbox, mitten, sandal, schooner, shield, spider web, steam locomotive, scarf, vase, wine bottle,
cream, bagel, pretzel, cheeseburger, hotdog, pizza, burrito, espresso
cabbage, broccoli, cucumber, ball pepper, mushroom, Granny Smith, strawberry, lemon, pineapple, banana, pomegranate, acorn
volcano
baseball player, scuba diver
Could you refine the grouping of these classes to make them clearer?
Figure B.6.3: Prompt used for GPT-4o to group classes for class order in Sec. 3.6
27

---

Published as a conference paper at ICLR 2025
C EXPERIMENTAL DETAILS FOR REFLEXIVE GUIDE
C.I. SUGGESTED CLASS POST-PROCESSING
Despite the guidelines provided in Stage 1, responses that deviated from the guidelines are observed. These failure cases can be broadly categorized into four cases: 1) inclusion of ID) class names, 2) failure to suggest exactly N classes for each near- and far-OoD group, 3) duplicated classes, and 4) no suggestion due to failure in recognizing the image. We post-process the suggested answers to refine them into feasible set of auxiliary OoD classes, except for Case 4. For Case I, we exclude suggestions that are character-level identical to ID classes. For Case 2. if the total number of suggestions exceeds 2N, we sample only 2NN classes from the suggestions. If the total number of suggestions is less than 2N,, we use the suggestions as is without any supplementation. For Case 3, we remove one of duplicated classes.
C.2 DETAILED RESULTS FOR SEC.
Tab. C.2. presents the comprehensive results for ReGuide.
Table C.2.1: ReGuide results on 5% subset of the ImageNer200 benchmark. The AURC values are multiplied by 10 and the other values are percentages. 'Valid' indicates the ratio of valid responses out of total of 4, 170 image-prompt queries. Bold denotes the best performance among the results from each model.
ID
Nea
Fa-Oub
Opwaraage-
Modile
Velud
(TC
(CE
AURC (1)
FARE LLV RSS LUV AUROC (1)
$1.0 (2.54)
91.23
9.12
02.27
32.13/2
(58.31
$4,34-
$2.1
55,29
28.89/ 25-86 $8.25
15.72
23.12
(63.3
GPTare
$4.66
11:56
$5.18
(9 11/66
85 9536585
NURY (7600
T0V 24 20/
19
/11/// 61
11.94
/30.19
14864
$3.00
3.97
70.97
51.21
51.28 71.89
71
02:02771
52 00
2.20 2:20 99643
10.30 T67 0976//906//333 /19 15
14:27
14.27 90.40
44.45 (44 75.30
GPT
(7.88(3445)
9066
235
10.44
8.57
14.66 96
54.507 7%
0.8//183 98.11
94.61
GEP-A (7808-0-00) www.://............. WWw.M//--W Chos MID Www.W..LLL....-- 700
we evaluate the effectiveness of ReGuide on the shared valid query set. As shown in Tab. C.2.2 the overall performance trends among the LVLMs remained consistent with their performance on their own valid query sets. ReGuide enhances OoDD performance across all models, consistent with the results observed on their own valid query sets. We additionally evaluate their FPR at TPR threshold of 90%, considering the high sensitivity to TPR thresholds, as discussed in Sec. 3.6 At the 90% TPR threshold, FPR is significantly reduced, and ReGuide consistently demonstrates lower FPR compared to the baseline across all OoD datasets.
As in Sec. B.2
Table C.2.2: ReGuide results on the shared valid query set of the 5% ImageNet200 benchmark. and mean that lower and higher values are better, respectively. The AURC values are multiplied by 103, and the other values are percentages. Bold represents the best performance among the results from each model.
ID InngeNee200
NISCO
S5B Had
Testues
Openinage O
Modeb
Vold
(F)(
AURC(
FFROW4STT (1) (STRONON TPR ELLY AURDC
M5.30
6.15
61.3
WEBT
91-78
JAES 54.00 (83.0)
13-1145 27/
54.00
5413
11
76
76.48 (M))
LOST 1,B 798366
10:97 HAST A5
18.07 18.80 9916
18117 48 13
GPT4
08.85
1,82
1
9.001933 19856
32.517
34,8 90156
1X51 L85 /97.22
1/01 1990
1547/19550 53.62
RoGaide
12.18
11:22
/2.000 96.42
9:00 (1/449991
000/25.15/7.18
040/31.00/00.
5.46)
CPE-4 (2004-10440))
Tab. C.2.3 C.2.44 present example responses of each models Tab. The first row in Tab. C.2.3 indicates the classes suggested by GPT-4o when ID class names are provided (i.e., GPT-text). The prompt used to obtain class suggestions of GPT-text can be found in Sec. C.5) A class name below each image is the ground truth of the corresponding image. The class suggestion set in GPT-text remains fixed regardless of the input images, as ID classes are static information, whereas ReGuide's suggestions change based on the input image due to its image-adaplle nature.
The results on InterVL2--26B/77BB and GPT-4o demonstrate the effectiveness of ReGuide. Firstly, ReGuide suggests helpful classes for OoDD. For example, shown in Tab. C.2.3 In-
28

---

Published as a conference paper at ICLR 2025
tem suggests near-OoD classes for dotted image, such polka dot, cherry red, or salmon-colore which are highly close to its ground-truth. These suggested classes successfully guide the model to classify OoD inputs into of the auxiliary OoD classes. For softball image, the prediction of ReGuide also closest to the ground-truth.
Table C.2.3: Example responses of Intern V1.2-26B on 5% subset of ImageNer200 benchmark
Near-DoD
Far-OoD
GPT-text suggested OoD classes
motoreycle, bulldozer. windmill, skyscraper. piano keyboard, surf- board, basketball hoop. bookshelf, drone, traffic light, skis, space rover, diving board, ferris wheel, telescope, typewriter, electric fan, hot air halloon, sewing machine, xylophone
siamese cat, bengal riger cub, chee- lah cub, lynx, arctic wolf, manalee, walrus, otler, arctic fox, albatross, macuw, snowy owl, peregrine arctic hare. hedgehog- wood- pocker, puffin, scahorse, swordfish, caracal
Prediction
1eeaVL2-268
GPTacs
ReGlaide
ReGuide suggested OoD classes
marlin, bonito, remora, white shark, suilfish, tiger shark, sword- tail, flying fish, barracouta, men- haden, electric cel, smelt, pilotfish. thresher shark, squid, tuna. herring, bluefin tuna, mackerel, pufferfish
nautilus, sea urchin, sand dollar, starfish, oyster, krill, cuttlefish, sea spider, snail, horseshoe crab, mahimahi, conch, guppy. copepod, abalone. sunfish, barracuda, octo- pus, brittle star, mullet
great white shark
ImageNer2200
swordfish
rhead
hammerhead
baseball diamond, buseball uni- form, bascball field, baschall helmet, umpire, athlete, baseball game, softball player, baseball team, baseball, outfielder, baseball bat, bascball jersey, second base- man, pitcher
pants, fielder, catcher, shortstop. baseball stadium, sports player, first baseman. baseball glove, baseball cap. bascball card, third baseman, coach, haschall mitt, baseball player, baseball bat- ter, referee
baseball player
SSB Hard
baseball player
ftbb11
softball player
scarf
polka dot, cerise, cherry red, blush, cherry blossom, fuchsia, garnet, coral, rose, peach, roseate, salmon- colored
cherry. pink, vermilion, crimson,
Textures
wine red, salmonpink, scarlet, ma- genta, ruby, marmon, hurgundy, salmon pink, mauve
none these classes
dotted
polka dot
29

---

Published as a conference paper at ICLR 2025
Table C.2.4: Example responses of GPT-4o on 5% subset of ImageNe1200 benchmark
ReGuide suggested OoD classes
Prediction
Near-OoD
Far-OoD
Cir1-4a
ReGande
umbrella, wristwatch, bicycle, elec- Irie fan, soccer cleat, refrigera- lor, telescope, skutebourd, basket- ball hoop, vacuum cleaner. wushing machine, piano, coffee maker, lap- top, microwave, tennis racket, desk chair, bookcase, alarm click
greyhound, lurcher, saluki, rhode- sian ridgehack, vizsla, basenji. irish wolfhound, doherman, ibizan hound, deerhound, basque shep- herd, great dane, galgo, pharuoh hound, borzoi
ImaggeNe2200
greyhound
whippet
wh-ppet
umbrella, wristwatch, bicycle, elec- tric fan, soccer cleat, ularm clock, tennis racket. refrigerator, tele- scope, skatcboard, basketball hoop, vacuum cleaner, washing machine, piano, laptop, micnowave, desk chair, bookcase, coffee maker
oriole, bunting. grosbeak, virco, sparrow, kinglet, finch, linnet, wren, pine siskin, nuthatch, phuebe, treecreeper, tanager. canury, star- ling, chickadee, warbler, titmouse
goldfinch
SSB Hard
none of these clusses
ye
fish scales, coral, chainmail, tur- de shell, reptile skin, armadillo shell, rocky surface, emoodille skin, lexture, mesh fabric, min- eral formation, mosuic, sandstone. fossil texture. lizard skin, dragon scales, pangolin scales, pebbles, snake skin, textured leather
umbrella, wristwatch, bicycle, elec- tric
Textures
cleat, alarm clock, tennis racket, refrigerator, tele- scope, skateboard, basketball hoop, vacuum cleaner, washing machine, piano, laplop, microwave, desk chair, bookcase, coffee maker
goldfish
fish scales
scaly
C.3 ANALYSIS OOD SCORE WITH REGUIDE
To further investigate the FPR increase in larger models such as InternV22-76B and GPT-4o in Tab. [4] analyze the ROC curves for each model with and without ReGuide. Fig. C.3.1 shows the ROC curves for Intern VL2-26B and GPT-4o. Both models exhibit improved ROC curves after applying ReGuide; however. the FPR declines more gradually for GPT-4o-ReGuide compared to Intern V1..-226---eeiiii..
as TPR decreases from 100% as shown in Fig. Ib) and C.3.I
While the reduction FPR for GPT-4o+ReGüidde is less pronounced at high TPR thresholds (e.g., 100%-95%), it achieves significantly lower FPR as the TPR threshold approaches 90%.
57
10
83
20
12
(a) Intem2LL2226B
(b) IntemVLD-26BBRoouudd
(c) GPT-4o
(d) GPT-to-ReGuide
Figure C.3.1: ROC curves for Intern VL2-26B and GPT-4o
To explore the causes of the FPR change observed across models in Fig. C.3. we analyze exam- ples at the two extremes 10% and 0.0%- of the ROC curves, as well as the shifts OoD score distributions induced by ReGuide. This analysis reveals that the increase in FPR @95%TPPPP for larger models such as Intern VL2-76B and GPT-40 stems from their superior visual recognition ability, which leads more confident predictions with highly suitable class suggestions.
presents examples on the extremes of Fig. Cb). and C.3.I d). This highlights that la- beling issues arise from insufficiently thorough annotations. First, small subset of datasets exhibits a conceptual mismatch between the objects images and their ground- labels. For instance, the grand pi ano image demonstrates misprediction caused by such labeling issues, the object is
Tab. C.3.1
30

---

Published as a conference paper at ICLR 2025
more accurately described as a grand piano mechani sm, class suggested by GPT-4o, rather than the ground-trutth label grand piano. Another case involves images with ground-truth labels corresponding to objects that occupy only small portion of the image, shown in the bee and ball examples. In the bee image, the dominant object is a plant resembling a ca uss yet the ground-truth label is bee. For such images, LVLMs often suggest classes based on the dominant objects and classify them accordingly.
Table C.3.1: Examples the extremes of the ROC curves for the ImageNet200 benchmark. Bold indicates the predicted class and its confidence score before applying the softmax function. Classes with a confidence score of 0.00 have been omitted.
Made
GPT-4e
IatereVL.3-668
100,000
Neze acDranie keys, harpechend, hamsesum healpheees, clann caass, oment grasd synacsess lesbeand aradd pizso tncchation
Neas OcD: saallowes dusaktion, masigold daie, tuip, mos, libs. oichid, sis, cornatin, posst, chayaathennsr pany, viskss by-
Ditasit: ImageSSe200
huirdrys clectiic fie, naket, Kender, refiggratre Poodetior good paro machasion
buyele,
OuD dandetion cledi, alse dandelion head
Predicios. castos flove
(grand plato meluriin 99.99 ron thow choone (0))
(carten Blwer 108000 more these chaser 0.00)
grand plano
Dutset SSB Hard
Ditactt NINCO
Nea- OuD: tres, mouniis
gane, bandball diansod,
Neu. OrD: cogges sorry, eatfoh, pile: salsoon fane, sworifils
basbbll luchhall
Fas OuD: unduclla, hamne action westweklh, buyele, lainkye, keep deekk fes, leess nokket Mender, celigerato
OaD unbeella. lannes setea, wistwakll, cleptar Jlisocens, ednss, cariaal tasiish, boit Prodition.
Pioletion
hall
LUBI THANS
The suggested classes provided by GPT-4o for aabel-mismatched inputs are significantly fine- grained and contextually appropriate. Tab. C.3.2 presents the classes suggested by GPT-4o for im- ages with ambiguous or inaccurate labels. For example, GPT-4o suggests classes such as model ape, gorilla fi guri ne, orangutan statue, and monkey figurine as near-Ool) classes. Based on the shape and texture of the object the image, these suggestions are highly rele- vant and more suitable classifications. Similarly, for the cockshell image, GPT-4o suggests near- OoD classes such as mot categories closely related to the depicted object. As shown in Tab. C.3.2 GPT-4oo's 's strong image recognition capabilities allow it to provide fine arainer contextually appropriate classifications for images with ambiguous or inaccurate labels.
sailboat, rowboat, and kayak, which are fine-grained
Table C.3.2: ReGuide examples for images with ambiguous or inaccurate labels from the ImageNet200 benchmark. Bold indicates the predicted class and its confidence score before apply- ing the softmax function. Classes with confidence score of 0.00 have been omitted.
ID inputs classitied into OoD dlases
OnD isputs dassiiieed into ID clasess
Nise-Daii: nowing hoot, pabills jackst, risctank, repids, raf, oaer roftigg paddling, infutaale kayaking gear, tiver salling, influabes,
Dunset ImageNer200
Datacct: SSH--Had
terr. commana, jacket, publlcsou,
dock,
sponts laysk, adventtre. Neycle, wriewatch, sic fin, soaer clai, acuiis nickss belazi, refigemator, ielewooee skavehrerd, hudeohall howp. vocum craaee: washing mochine, pi- azo, lapoo,, nicrowave desk chair, bookesse, coffee makes, diara clock
Fan-Del untrella
sundala skiff, mains. yacls, cance pubdle, postoce bout. kuyak Pan-Ovi): ambrrlla, Neydle incwmilll, elecnic fin. anthosoppe. refrigeratr, reclicer, teleccppe skuteooad typentiter, baskettull kuep, varaum clanee wadiing machine, sowing machins arkiee. nicowwaee wasser brietcase, daam
Peddcction kayaking seuz
(kayaking year:
claece: 0.00)
9K00, these clases
Dasset: lmnggNNe220
Dituset:
collectllle primate, action spe, poilla fineiss, prinate igun, urine, lyy huboon, oranquian diiee asthoopod lyy, plasiic munkey. chimpanzee my plush monkry, stiteed chimpannen balwon rppics. siman stute, catcaure gocils, ape wolstare monkey model Par-OuD snbrilla, beycl, heslpphonas lamp, shrass clock, retiggeato,, pencil guisa. computer, sorebioti. cat, chiit, peruz
possey,
podbellce piz. wild boar,
to
dlebook rig.
sunkatire jasclias, mestun piz, amwerth dune galito pig. kune lons, vemamre Pig. lirge whire Fie OcD: unbeella wiiswath, bicyel, clestri fan, socce: clud, rafegentte scloung, skenhvand, hookcase bookettall heop vas- vun cleaee, wakiing muchine, plane. lapoop, micowaae, leani Pediction (pig 95,00 sim of thece dassn: 5.00)
unglavas
Piediction monkey
(manke liuurin:: 96.06, aonz of those clases: 3.013
Dasaiet ImageNer2(1
Dataior: SSB Hand
Near-Gul: Grf> cascs, car, wheel rim, unieycle wapos, seilly wheel Ntycle wheel pusicon. band inik sonorr ginden bamns, dolly. wäeel, yar canl, sindles, tiyyele guede molley, motoreyle whecl. ter
Nov- OcD: damucily, moth, kamile, crickes bunisletee. fio, ophid homee. lrcoinng snik bug prayng mannss beetle, di- cada, åretly, katdda
Re----l unteells, wriewatch, bicycle, eleaticc wacor cket, leatis rackst, refiierutos telescope, skutebourd, lypipp washing
OuD: unbeella, westwath, electic fan, paintrush, soccer bestetbil hoop.
hoop,
eirignntee shlrrape machine. lapoop.
deaser, micnnaon, desk chair, bookeser cofiee maker alam clock Prodiction wheehanow ( (sbedlarrow: 98,00 whod: 90.00, pore of dese classes: 0.00)
desk bookcase, alann clock
Prodiction: morucch buterlly (mnnarchh buttortty 95,00. none ot these clusses 5.0)
wheelbwTTON
peacock
31

---

Published as a conference paper at ICLR 2025
Fig. C.3.2 a) and b.3.2 depict the OoD score distributions for Intern VL2-266 and GPT-4o, spectively. As shown in Fig. C.3.2 a), with ReGuide, IntemVL.2-26B produces OoD scores ranging between 0.00 and 100.00, which not observed without ReGuide. However, the effect of ReGuide on GPT-44 differs from that on Intern V1.2-26B. The OoD score distribution for GPT-4o becomes more biased with ReGuide, illustrated in C........)b) The number of responses with Ool scores between 0.00 and 100.00 decrease significantly. We infer that this outcome is due to the strong vi- sual recognition ability of GPT-4o. As shown in Tab. 2 the suggestions of GPT-4o include classes that are highly similar to the given image.
100
IntemVL2-266
100
GPT-4
Frequency
IntemVLL2268
ReGuide
Frequency
GPT-4o
75
75
50
25
25
0
25
50
75
100
0
25
50
75
100
OoD score
OoD score
(a) IntemVL2-26B
(b) GPT-4o
Figure C.3.2: OoD score distributions for Intern VL2-26B and GPT-4o
For more detailed analysis of the ReGuide effect, we examine the OoD score distributions for ID and OoD inputs separately. Fig. C.3.3 displays OoD score distributions for ID and OoD inputs on InternVL2-22 and GPT-4o.
150
Feequency
HaarsLa333 faGuia
Frequency
Froquency
GF5
Trequencyy
G754)
800
1.00
800
50
50
200
200
25
50
75
100
0
25
50
25
50
100
100
0:0
Scone
OoD score
OnD score
GoD score
(a) IntemVL2-266 ID
(b) LLtemVL2-226B OoD
(c) GPT-4o ID
(d) GPT-4o OoD
Figure C.3.3: OoD scores distributions of each ID and OoD for Intern VL2-26B and GPT-4o
A common observation regarding ReGuide for Intern VL2-26B and GPT-4o is the shift in the OoD score distribution OoD inputs. As shown Fig. CC..3 and C.3.3 d). ReGuide increases the number of OoD inputs with an OoD score of 0.0 while reducing those with score of 100. This shift demonstrates that the models effectively classify OoD inputs into one the suggested OoD classes, thereby improving OoD detectability and confirming that ReGuide functions intended.
The increase inputs with an OoD score of 0.0 is also observed for ID inputs. As previously discussed in Tab. C this is caused by ID images with ambiguous inaccurate labels. For such inputs, GPT-4o+RRGuudd effectively classifies them into a single class with higher confidence, as the suggested classes often include one closely aligned with the ground-truth label. This explanation further supported by the improvement in FPR @ 90%990%PPPP TPR, where the influence of this small subset of images is reduced, as shown in Tab.
32

---

Published as a conference paper at ICLR 2025
C.4 COMPARISON OF OoD SCORE FUNCTIONS
When incorporating available auxiliary OoD classes, one of effective scoring strategies is to include confidence estimates for these auxiliary OoD classes. For example, NegLabel (Jiang al. 2024) uses the scoring function as the ratio of the sum of ID softmax to the sum of both ID and OoD softmax, and EOE (Cao al. 2024) utilizes the difference between max ID softmax and max OoD softmax.
We compare three different scoring functions: the maximum ID softmax, 2) the difference be- tween the sum of ID softmax and OoD softmax, and 3) the difference between the maximum ID softmax and maximum OoD softmax. Each of the scoring functions is represented, order, as follows:
ex/TT
Smax (x, V, Aaux)
max e
EKEYLA C5K/T
edilT
Ssumsub (x, Aaux)
Σ
E
EkEYLAm
eaalT
EkERUAm eaa/T
.EY/
jeAuu
@aL/T
031/T
Smassub (x, Asux) = max kyy ΣREDUAM eoa/T
max jeAu
ΣkEYLAm 6#c/T
(3)
where T is temperature.
Tab. C.4.1 presents the results with these scoring functions. All results are on a 5% subset of ImageNet200 benchmark using Intern VL2-266B-eGuruide No significant differences are observed across scoring functions. LVLMs tend to assign high confidence to a single predicted class, re- sulting in similar results regardless of these different scoring designs. Therefore, all results reported in our paper are measured by Smax.
Table C.4.1: Comparison results between different scoring functions on 5% subset of ImageNet200 benchmark using Intern VL.2-26B
Near- Near-OoD
Far-OoD Textures
All OoD
NINCO
SSB-Hard
iNaturalist
Openimage-0
FPR@95%TPR (L) AUROC (t)
Smax
22.89/ / 86.53
15.21 / 99.41
1.39 / 98.02
3.93 97.05
2.04 29.4/9
10.27 99.19
Saumsub
23.88 86.46
15.21 90.58
10.65 97.45
3.49 96.90
2.33/ /97.69
11.59 93.119
Smaxsub
23.38 86.53
15.27 90.59
1.39 97.45
393/96.90
99.69
10.33 93.26
33

---

Published as a conference paper at ICLR 2025
C.5 PROMPT FORMAT
We provide the full prompts used for experiments in Sec. 4.1 In the prompt, yellow, green, red, blue boxes indicate task description, an explanation of rejection class, guidelines, and examples for response format, respectively.
C.5.1 PROMPT FOR CLASS SUGGESTION OF GPT-TEXT
Fig. C.5.I illustrates the prompt used for class suggestion of GPT-text. The guidelines consist of two key instructions: first, avoid suggesting classes that are too similar to the image, and second, to avoid suggesting classes that are identical to the ID class. The red text indicates that the contents vary depending on the choice of benchmark.
Your task to provide 2N class names based on the given class names: (ID classes). The first N names are visually similar to visually dissimilar belong different domains, should not provide
given The last class names.
Avoid suggestions that are very similar to each other. For example, suggestions that are part of a large object or in the same broad category, such (example class 1) (erample class set
Do not provide the following category names your suggestions: (ID clusses, none of these classes)
Figure C.5.1: Prompt used for class suggestion of GPT-text
C.5.2 PROMPT FOR REGUIDE STAGE 1
Fig. C.5.2
illustrates the prompt used for class suggestion of ReGuide Stage Similar to the original prompt used for evaluating OoDD and the prompt used for class suggestion on GPT-text, we provide few guidelines prevent receiving uninformative class names, such as ID class names, overly similar class names or just a repetition of class name. Without an example, the response format of image-based suggestions deviates more compared to that of text-based suggestions. Thus, we include the response format an illustrative example. The red text indicates that the contents vary depending on the choice of benchmark.
Your task to provide 2.V class names based on the given image. The first N names are visually similar to image. The last names are visually dissimilar belong to different domains. You should not provide over 2N class names.
Avoid suggestions that very similar each other. For example, suggestions that of large object in the same broad category, such as (example class set 1) or (example class set 2).
provide following category names as your suggestions: classes,
classes)
example your response: Example) When the given image is 'exumple label'
Class suggested: example class suggestions
Strictly follow this format for respanse.
Figure C.5.2: Prompt used for ReGuide Stage
34

---

Published as a conference paper at ICLR 2025
C.5.3
PROMPT FOR REGUIDE STAGE 2
illustrates the prompt used for ReGuide Stage 2. The red indicates that the contents vary depending on the choice of benchmark. The blue indicates the locations of the suggested OoD classes from Stage 1. The highlighted text in yellow indicates modifications made reflect these suggested classes. GPT-text employs the same prompt format for GPT-4o's suggested classes.
Fig. C.5.3
Your task to classify the given image into one these C classes: (ID classes, auxiliary OoD classes, none of these classes) and assign confidence values for your prediction to each class.
You can classify the given image into 'none of these classes': if you cannot classify the given image into any of the other classes given, if you need other classes other than the given other C + 2N classes to classify the given image.
whether image belongs
given other C 2N classes, think you
fallawing are guidelines your response. Please respond according
guidelines.
You should provide your confidence for each class between 0.00 and 100.00.
expressed The class You've assigned the max confidence should be your prediction.
The confidence
float.
Your confidence should be above 0.0 in least one of the + 2N classes given. It not allowed to assign 0.0
confidence to every class.
You should provide your confidence in the C+ 2N classes given, even your prediction is 'none of these classes'. allowed predict given image into any
Your prediction should be
given classes than the C + 2N 4 given classes.
classes. You
-Thre 'none of these classes' in the given & 2N + classes. This means your prediction should be 'none of these
classes' your prediction is not in one of the other classes.
Striclly follow the guidelines above.
Here is an example of your response.
Please respond with the following examples format:
Example)
Prediction:
your prediction
Confidence: (ID classes: your prediction euxiliary OoD classes: your prediction, none of these classes: your prediction)
These above examples are confidences for two different images. When you provide your answer, your format should be the the format between the dashed lines the examples. Strictly adhere to the examples format provided above. Do not deviate from the above format.
Figure C.5.3: Prompt used for ReGuide Stage 2
C.6 INFERENCE COST ANALYSIS
While ReGuide, as a two-stage approach, incurs higher computational costs compared to single- turn generation (Section 3), the inference cost does double. This is because caching image representations eliminates the need to process the same image twice, and the shorter prompt in Stage (class suggestion) reduces the token count. For open-source models, we analyze the inference cost from computational complexity perspective, whereas for proprietary models, the analysis focuses on API usage costs.
We estimate the inference time FLOPss following Li et al. (0024)). This standard practical estimate based on a simplified scaling law, assuming that the inference cost of the LLM scales linearly with both the number of tokens processed and the model size. The inference time FLOPs is estimated as:
(4) where N is the number of parameters in the LLM and T denotes the total number of tokens involved in the inference process. Specifically, T the of Q. V. and G, which correspond to the number of text input tokens, image tokens, and generated tokens, respectively. For example, for VL2erVVL2-76B. the prompt depicted Fig. B.6.I with ImageNet200 classes includes 5,350 input tokens. The number of generated tokens is 1,936, assuming the output follows the provided example structures, and the number of image tokens is 256. Thus, the baseline FLOPs is estimated as 7 x 10 x (256 + 5, 350 + 1, 1,93)..
FLOPs = O(N x T)
Similarly, for ReGuide, F FLOPs estimated as:
FLOPs O(N (V + Qstegel Gstagel + Qstage2 + Gtagge2)
(5)
35

---

Published as a conference paper at ICLR 2025
where Q Stagel, Gstagel and Qtoage2, Gtage2 denote the number of text input and generated tokens for Stage and Stage 2, respectively. Although ReGuide involves two stages, the input image is processed only once the same image is reused across both stages and can be cached. Note that the inference cost of the vision encoder is excluded, as it remains consistent for both the baseline and ReGuide within the same VLM..
Tab. C.6.1
presents FLOPs for the baseline and ReGuide on Intern VL2 models. These values estimated on one input pair (image-prompt) of the ImageNet200 benchmark, assuming that the generated outputs follow one of the provided example structures.
ReGuide requires only 1.14x the computational cost compared to the baseline. It is because 1) the prompt used in Stage 1 is short and simple (e. 917 tokens for Stage 1), and 2) the image tokens are processed only once. Consequently, the increase in computational cost arises solely from the processing of the Stage prompt. Additional optimization techniques for multi- turn inference could further reduce the computational cost gap between the baseline and ReGuide.
Table C.6.1: Estimated computational costs for InternVL2 models (in units of 10t3 FLOPs)
Models
Baseline
+ReGuide
Intern VL.2-26B
15.086 52.794
17.052 60.
Intern VL,2-76B
Tab. C.6.2 shows API costs for proprictary models un- der the baseline and ReGuide experiments on the 5% ImageNet200 benchmark. From the perspective of API costs, ReGuide exhibited a higher proportional increase compared to computational costs above, with a 1.47x increase relative the baseline. Since processed each stage separately, the API cost reflects a scenario where image token caching is not applied. Additionally, we did not employ multi-turn inference, as provided no advantages in inference time due the absence of batch inference support. Implementing multi-turn batch inference could significantly improve efficiency and reduce costs.
Table C.6.2: API costs (USD) for proprietary models, calculated per 1 million input and output tokens
Models
Input token
Output token
Baseline
ReGuide
GPT-4o
1.25
73
Claude Sonnet
15
228
Gemini Pro 1.5
3.5
10.5
193
GPT-4o (2024-08-06)
One possible approach to reduce the inference cost gap between the baseline and ReGuide inte- grate the two-stage prompts into single prompt while maintaining ReGuide's process of providing suggested classes. This adjustment would allow ReGuide to maintain its effectiveness with a single forward pass. Fundamentally, it is essential to develop LVLMs to overcome their tendency to limit responses to the options explicitly provided in the user-defined prompt. Such advancements would enable LVLMs to explore and utilize information beyond the immediate constraints of the prompt, establishing robust models capable of generating safe and reliable responses to a wider range of inputs.
C.7 LIMITATIONS REGUIDE
While the proposed ReGuide improves performance in both image classification and OoDD tasks, there remain limitations. These challenges are primarily caused by the difficulty in exerting pre- cise control over the behavior of the VLM. Due to the inherent complexity of understanding the implicit dynamics of LVLMs, controlling their behavior through prompting does not always ensure the desired outcome. Although we provide additional guidelines based on the model's failure cases to mitigate unintended responses, this approach is insufficient for full control over outputs. To ad- dress unintended outputs, can utilize LLMs iteratively refine prompts based the model's responses. Leveraging external LLMs for this purpose expected to function similarly to model en- semble principles, resulting in prompts better suited to the model compared to heuristic refinements.
36

---

Published as a conference paper at ICLR 2025
From a visual perspective, ReGuide relies on the image understanding capabilities of LVLMs. Any potential improvement depends on the model's ' ability to recognize visuals accurately. Additionally, since our proposed approach is image-adaptive, the quality of the suggested class set is influenced by the image context. For instance, the target object occupies only small portion of the image, an LVLM may focus more other objects in the background. This tendency can result in class suggestions that are either uninformative or overly broad.
37


Reviews:
[{"review_id": "ahImXhcSzm", "reviewer": "Reviewer_Mv3D", "summary": "The goal for the paper is to evaluate the out-of-distribution detection (OoDD) capabilities of LLMs (some private and open models)", "questions": "- Why we need to evaluate the capabilities \n- as LLM gets huge weekly, the measurement still makes sense?", "limitations": "", "rating": 3, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- easy to read\n- easy to duplicate", "weaknesses": "- motivation for the work\n- limited impacts\n- extending from a prior work in text-based method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "SYPqSHH4AS", "reviewer": "Reviewer_kMC5", "summary": "This work proposes a self-guided prompting approach, termed Reflexive Guidance (ReGuide), aimed at enhancing the OoDD capability of LVLMs by leveraging self-generated image-adaptive concept suggestions. Experimental results demonstrate that our ReGuide enhances the performance of current LVLMs in both image classification and OoDD tasks", "questions": "see weakness", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "Strengths:\n•This paper represents the first effort to extend zero-shot Out-of-Distribution Detection (OoDD) to generative Large Vision-Language Models (LVLMs) in reference to CLIP. Building upon this foundation, the authors define the task and propose corresponding evaluation methodologies, which offer valuable insights for future research；\n•The paper evaluates and analyzes the OoDD capabilities of various proprietary and open-source LVLMs. Furthermore, the authors propose a two-stage self-guided prompting approach, termed Reflexive Guidance (ReGuide), to unlock the potential of LVLMs. Experimental results demonstrate that the ReGuide enhances the performance of LVLMs in both image classification and OoDD tasks.", "weaknesses": "Weaknesses:\n•The paper introduces new tasks and evaluation metrics. However, certain details in this regard are somewhat lacking. For instance, the paper mentions using only 25% of the benchmark subsets but does not specify which portions are utilized. It would be preferable to provide a list of these subsets to facilitate the replication and comparison of subsequent work. Additionally, other similar details need to be verified.\n\n•In the experimental evaluation, the ‘valid ratio’ varies across different methods. And only valid responses are included when measuring performance, which results in inconsistencies in the samples used to calculate metrics for each method. This may not be justifiable. Have you considered adopting a more consistent evaluation approach?\n\n•The novelty of this paper is somewhat limited.  Firstly, it introduces the OoDD task for LVLMs; however, the rationale behind this task remains debatable. Considering the robust capabilities of LVLMs and their ongoing advancements, LVLMs are anticipated to directly recognize objects within images. In comparison, the method proposed in this paper is overly complex and excessively reliant on prompts. Secondly, the approach presented in the paper appears to be more akin to prompt engineering, lacking substantial innovative contributions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2OD8ee3uGN", "reviewer": "Reviewer_BDm6", "summary": "Overall this is a solid paper. This paper addresses the underexplored trustworthiness of foundation models, particularly focusing on the out-of-distribution detection (OoDD) capabilities of large vision-language models (LVLMs). The authors highlight the gap between the demonstrated potential and practical reliability of LVLMs, raising concerns about their safe deployment. To address this, the researchers evaluate and analyze the OoDD capabilities of various proprietary and open-source LVLMs. They propose a novel approach called Reflexive Guidance (ReGuide), which aims to enhance the OoDD capability of LVLMs by using self-generated image-adaptive concept suggestions. Experimental results show that ReGuide improves the performance of current LVLMs in both image classification and OoDD tasks.", "questions": "Please see Weaknesses.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "[Promising research topic] This paper focuses on an under-explored topic, the reliability of large vision-language models. [Benchmark contribution] This paper define the concept of Out-of-Distribution Detection on LVLMs, and conduct extensive experiments to build a benchmark and give insightful analysis. [Method contribution] Furthermore, they propose a method ReGuide on the prompt to boost the performance of LVLMs. They verify the effectiveness of the proposed method on GPT-4o and InternVL.", "weaknesses": "Since the out-of-distribution detection on vision-language foundation models is a new definition proposed in this paper, I would suggest to add a demonstration figure to show this concept in a visually straightforward way. You can compare the conventional OOD, OoDD on CLIP and OoDD on LVLMs.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "e9PXmEkFn3", "reviewer": "Reviewer_wyWF", "summary": "This paper explores to enhance out-of-distribution detection (OoDD) in large vision-language models (LVLMs). The authors emphasize the increasing importance of these models in various domains and highlight the challenges related to their trustworthiness, specifically in dealing with out-of-distribution (OoD) inputs. To address this, they propose a self-guided prompting approach called Reflexive Guidance (ReGuide), which leverages LVLMs’ visual interpretation capabilities to improve OoD detection by generating image-adaptive concept suggestions. The experiments demonstrate that ReGuide significantly boosts the performance of both open-source and proprietary models in OoD detection tasks.", "questions": "N/A", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The introduction of the ReGuide method is interesting. By using image-adaptive concept generation, the authors provide a unique solution to enhance OoD detection in LVLMs. This approach leverages the strong visual recognition capabilities of these models, offering a novel way to bridge gaps in OoD detectability.\n\n2. Comprehensive Evaluation: This paper comprehensively evaluates both proprietary and open-source LVLMs, providing a balanced comparison across different models. This may be helpful for future research.\n\n3. Detailed Analysis: The authors provide a well-rounded analysis, including experimental results, visual feature interpretation, and the effectiveness of ReGuide on near- and far-OoD datasets. This depth of analysis gives credibility to their findings and demonstrates the robustness of their proposed method. Furthermore, it demonstrates the potential of introducing LVLMs to the area,", "weaknesses": "1. Limited Explanation of Failures: Although the paper provides examples of failure cases, it does not fully delve into the underlying causes or offer detailed solutions for these issues. While the authors acknowledge the problem of overconfidence in models such as GPT-4o with ReGuide, further exploration of these limitations could strengthen the study. There is still not a robust method to effectively introduce LVLMs to the OoD tasks.\n\n2. Model-Specific Insights: The paper focuses on generic findings across models, but a deeper investigation into how specific models (e.g., GPT-4o vs. InternVL2) behave differently when ReGuide is applied could add nuance to the conclusions. For example, the differences in false positive rates (FPR) between models with and without ReGuide should be presented for a better comparison. \n\n3. Scalability and Practicality: While the ReGuide method shows a promising direction, the computational overhead and API limitations mentioned in the paper could present challenges for practical, large-scale implementation. This issue is touched upon but not sufficiently addressed in terms of how ReGuide might be optimized for deployment at scale. Meanwhile, the inference cost analysis can further improve the paper's quality and inspire further work.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)