Title: Enhancing Zero-shot OOD Detection with Pre-trained Multimodal Foundation Models
Venue: NeurIPS 2025
Accepted: no
Keywords: OOD detection
Abstract: Out-of-distribution (OOD) detection is essential for reliable deployment of deep models in real-world scenarios. Advances in pre-trained multimodal foundation models have enabled zero-shot OOD detection using only in-distribution (ID) labels. Recent methods of this direction expand the label space with auxiliary labels to facilitate the discrimination between IDs and OODs.
Inspired by the probabilistic formulation via Binomial distribution, we further discover the key factors that theoretically affect zero-shot OOD detection performance: the cardinality of the auxiliary label set, the similarity between labels and samples, and the uncertainty of the similarity scores. From the theoretical analysis, existing methods that construct fixed, single-modality auxiliary labels surely have limited effectiveness. To address these issues, we propose Refer-OOD, a framework that adaptively generates, filters, and retrieves multimodal references that explicitly account for these factors. It consists of three modules: reference acquisition, feature mapping, and decision module. Experiments across multiple benchmarks demonstrate that Refer-OOD consistently improves zero-shot OOD detection with both vision-language models (VLMs) and multimodal large language models (MLLMs).


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nRefer-OOD introduces a zero-training framework for zero-shot OOD detection that dynamically assembles sample-specific multimodal references. For each test image, a multimodal LLM first generates candidate ID/OOD labels, a relevance-and-diversity filter prunes noisy concepts, and a web retriever fetches matched reference images. The test image and the retrieved references are jointly encoded by a frozen VLM, and a tailored activation-ratio metric drives the final ID/OOD decision.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths\n- This paper introduces sample-level multimodal references to enhance the activation probability of out-of-distribution samples without excessively increasing ID false rejections. Through filtering strategies and adaptive thresholds, the method is insensitive to the scale of the reference set and is pluggable into any pre-trained VLM, thereby achieving lower FPR95 and higher AUROC than multiple existing SOTA methods without needing to fine-tune model weights.\n\n### Weaknesses\n- This paper mentions “We use Bing image search and a Chrome retrieval plugin for retrieval”, but does not provide specific latency measurements or inference cost analyses.\n\n- This paper states “However, these methods are computationally expensive and may undermine the generalization ability of pre-trained VLMs.” but in fact LAPT only consumes computation during training, and at inference only requires a single forward pass + dot product. Your work needs to Generate, Filter, Retrieve for each image. \n\n- This paper has some similarity with the paper *Zero-Shot Out-of-Distribution Detection Based on the Pre-trained Model CLIP*(Generate unique OOD text for each image) and *LAPT: Label-driven Automated Prompt Tuning for OOD Detection with Vision-Language Models* (Generate/retrieve images for reference) . The innovation of its differences is limited, slightly insufficient workload.\n\nQUESTIONS:\n- How accurate is the candidate OOD label dynamically generated by MLLM? Is its quality convincing?\n- What differences are there in the quality of OOD labels generated by different prompts?\n- What is the impact if online search retrieval is unavailable or rate-limited? What is the fallback scheme?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 2, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 1, "ORIGINALITY": 2, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes Refer-OOD, a three-stage framework that (i) dynamically generates candidate OOD text labels with an MLLM, (ii) retrieves complementary web images, and (iii) fuses visual–textual similarities (or MLLM reasoning) to decide ID vs OOD. A binomial-style analysis motivates why a larger, more diverse auxiliary label set with lower ID similarity should improve FPR95. Experiments on four coarse- and four fine-grained benchmarks show large gains: e.g. FPR95 = 0.03 % on coarse datasets with CLIP-ViT/B-16 and +13.65 pp FPR95 improvement on fine-grained tasks relative to EOE . The authors acknowledge higher inference cost and potential security risks from online retrieval.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. Performance improvements are consistent across both VLM and MLLM back-ends, coarse and fine-grained settings.\n2. The three modules (generation, filtering, retrieval) are ablated, and each is shown to contribute.\n\nWeaknesses:\n1. Limited novelty over prior retrieval-augmented OOD work. Generating labels with an LLM and retrieving images at inference resembles existing pipelines; the main difference is fusing both modalities, which feels incremental.\n2. Some strategies of the method are designed experimentally without a theoretical foundation or clear explanations. For example, how does the method addresses MLLMs are known for producing hallucinated contents, e.g., [a].\n3. the proposed method additionally introduce at least 6 hyper-parameters. The ablation studies imply that the method are sensitive to the hyper-parameters. Does all ID datasets used in this paper share the same hyper-parameter settings.\n4. Compared with CUB-200,Stanford-Cars,Food-101 and Oxford-Pet, ImageNet-1k is significantly more standard benchmark in VLM-based OOD dection. However, why the results on ImageNet-1k is given in the appendix. \n5. After checking the results, I find the improvements are not convincing. For example, 1) A few previous works' results on ImageNet-1k are not consistent with the number in the papers, especially NegLabel and CLIPN; 2) the results of NegLabel in table 8 (appendix) are signfiicantly lower than that in [b]\n\n[a] Hallucination of Multimodal Large Language Models: A Survey\n[b] Conjugated Semantic Pool Improves OOD Detection with Pre-trained Vision-Language Models\n\nQUESTIONS:\nSee Weaknesses.\n\nLIMITATIONS:\nNA", "TITLE": "Official Review", "RECOMMENDATION": 2, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors identify key limitations in existing zero-shot OOD detection methods (static single-modality references) and propose Refer-OOD, which dynamically generates/filters text labels and retrieves visual references using multimodal foundation models. This adaptive approach significantly outperforms prior methods on coarse- and fine-grained benchmarks with both vision-language models (VLMs) and multimodal LLMs.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n1. The theoretical insights (e.g., variance reduction via multiple references) and unified adaptive framework are novel. Dynamic reference construction diverges significantly from static auxiliary-label methods (e.g., CSP, EOE).\n2. The paper is well-structured, with clear figures (e.g., Fig 1–2) and section flow. The problem formulation (Section 3.1) and method description (Section 4) are logically presented.\n3. The theoretical analysis (Section 3) rigorously derives factors influencing OOD detection and motivates the adaptive approach. Experiments are extensive, covering 10+ datasets, multiple backbones (CLIP, Qwen-VL), and ablation studies (e.g., module contributions, hyperparameters). The method consistently outperforms strong baselines (e.g., 13.65% average FPR95 improvement over EOE in fine-grained tasks).\n\nWeaknesses\n1. The computational cost of online generation/retrieval is not quantified (e.g., latency vs. baselines). Theoretical assumptions (e.g., i.i.d. similarity scores) are not empirically validated.\n2. Dependency on external APIs (Bing) for retrieval raises reproducibility concerns. Limited discussion on real-time applicability (e.g., latency constraints in safety-critical systems).\n3. The core idea of using MLLMs for label generation builds on EOE, and retrieval-augmented methods (e.g., RAG) are established—though their integration for OOD detection is innovative.\n\nQUESTIONS:\n1. Computational Efficiency: Online retrieval/generation incurs latency. Could you report inference time vs. baselines (e.g., on ImageNet-O)? Would caching references per class be feasible without sacrificing performance?\n2. Theoretical Validity: The analysis assumes similarity scores are i.i.d. and approximates aggregates as Gaussian. Did you validate these assumptions empirically (e.g., score distributions on ID/OOD data)?\n3. Retrieval Robustness: How sensitive is performance to the retrieval API (Bing vs. Google/Baidu in Table 17)? Could adversarial/noisy references degrade results?\n4. Ablation Baseline: Does adaptive generation alone (without retrieval) outperform static methods (e.g., EOE)?\n\nLIMITATIONS:\nCheck the Weakness.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 2, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nAuthors propose Refer-OOD, a theoretical framework inspired by modeling OOD scores with the Binomial distribution to adaptively generate, filter, and retrieve multi-modal references to discriminate between ID and OOD examples by maintaining the activation probability of ID samples while increasing the activation probability of OOD samples. Authors examine zero-shot OOD detection performance the lens of the cardinality of the auxiliary set, the similarity between labels and samples, and the uncertainty of similarity scores. The proposed method has three modules, including reference acquisition for obtaining relevant references, feature mapping to evaluate the relevance of the input image to the constructed reference, and a decision module to classify if a sample is ID or OOD.  Authors demonstrate that their approach consistently improves zero-shot OOD detection with VLMs and MLLMs across several benchmarks including iNaturalist, SUN, Places, Texture, CUB, Stanford-Cars, Food, and Oxford-Pet.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n- Simple Approach. The proposed approach is simple and leverages RAG from search engines to augment the reference set.\n- Extensive Experiments and Ablations. Authors conduct extensive analysis on many datasets and evaluate several baseline models. They also ablate the key hyperparameters in their model. I would encourage authors to put the key takeaways from the ablation experiments in the text as well as including the table of results to improve readability. \n- Novelty. Authors highlight that prior works try to identify ID/OOD using purely ID labels, which others introduce auxiliary OOD labels. However, constructing this set of OOD labels is non-trivial. The primary contribution of this work is leveraging VLMs/MLLMs to construct this OOD using RAG-style methods using multiple diverse multi-modal references per class rather than a single reference label. \n\nWeaknesses\n- Struggles with Fine-Grained Classes. VLMs/MLLMs struggle with distinguishing between fine-grained classes, such that the generation process is not able to accurately generate relevant references. This problem will likely get worse when encountering concepts not typically found in VLM / MLLM pre-training (e.g. medical images, aerial images, etc.).\n- Writing Lacks Clarity. The current version of the paper relies heavily on theoretical frameworks and equations, when the proposed method can be explained more simply. Given that the proposed approach is simple, I would encourage the authors to directly explain the idea and intuition without relying so heavily on equations, and potentially swapping Section 4 with Section 3 or move Section 3 to the supplement. Many of the equations are redundant and unnecessary (e.g. 5,6,7). I would also recommend authors review their work for grammatical errors as this hinders clarity.\n\nQUESTIONS:\n- How do you disentangle the performance between the VLM / MLLM and search engine? To what degree does the search engine compensate for poorly generated textual references?\n- How might retrieval accuracy change if using image-based prompting for similar images to the test image (e.g. reverse image search) instead of text-based search?\n- Why didn't authors use more recent VLM / MLLMs (e.g. Qwen 2.5VL and SigLIPv2)?\n- How might performance change when the set of ID and OOD classes are swapped? How does the specific problem setup of the specific datasets influence downstream accuracy? What would happen if we dropped datasets from the ID set?\n\nLIMITATIONS:\nYes, authors address limitations in Section 6.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}