Title: Conjugated Semantic Pool Improves OOD Detection with Pre-trained Vision-Language Models
Venue: NeurIPS 2024
Accepted: yes
Keywords: out-of-distribution detection

zero-shot OOD detection

pre-trained vision-language models
Abstract: A straightforward pipeline for zero-shot out-of-distribution (OOD) detection involves selecting potential OOD labels from an extensive semantic pool and then leveraging a pre-trained vision-language model to perform classification on both in-distribution (ID) and OOD labels. In this paper, we theorize that enhancing performance requires expanding the semantic pool, while increasing the expected probability of selected OOD labels being activated by OOD samples, and ensuring low mutual dependence among the activations of these OOD labels. A natural expansion manner is to adopt a larger lexicon; however, the inevitable introduction of numerous synonyms and uncommon words fails to meet the above requirements, indicating that viable expansion manners move beyond merely selecting words from a lexicon. Since OOD detection aims to correctly classify input images into ID/OOD class groups, we can "make up" OOD label candidates which are not standard class names but beneficial for the process. Observing that the original semantic pool is comprised of unmodified specific class names, we correspondingly construct a conjugated semantic pool (CSP) consisting of modified superclass names, each serving as a cluster center for samples sharing similar properties across different categories. Consistent with our established theory, expanding OOD label candidates with the CSP satisfies the requirements and outperforms existing works by 7.89% in FPR95. Codes are available in https://github.com/MengyuanChen21/NeurIPS2024-CSP.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a novel method utilizing pre-trained vision-language models (VLMs) to enhance out-of-distribution (OOD) detection. Their core idea lies in constructing a Conjugated Semantic Pool (CSP) to enrich the pool from which OOD labels are drawn. Unlike simply expanding lexicons with synonyms and uncommon words, the CSP leverages modified superclass names that serve as cluster centers for samples with similar properties across categories. This, the authors theorize, not only increases pool size but also amplifies the activation probability of OOD labels while maintaining low label dependence. Extensive experiments demonstrate significant performance improvements over existing approaches.\n\nSTRENGTHS:\nOriginality: The paper introduces a unique concept, conjugated semantic labels, for enhancing the semantic pool, offering a valuable contribution to OOD detection.\nTheoretical Underpinnings: The authors provide a robust theoretical framework, including mathematical models and justifications demonstrating how the CSP improves OOD detection.\nEmpirical Validation: Extensive experiments showcase the effectiveness of the proposed method, achieving significant advancements over current state-of-the-art techniques.\nGeneralizability: The method's applicability across diverse VLM architectures suggests broad utility and adaptability.\nClear Problem Definition: The paper clearly identifies limitations in existing methods, particularly regarding simple lexicon expansion, and proposes a well-reasoned solution.\n\nWEAKNESSES:\n1. Ablation Study & OOD Score Function:\nThe ablation study in Table 11 showcases that competitive performance on iNaturalist and SUN datasets is achievable without the CSP. However, it remains unclear how the proposed OOD score function (designed by the authors) contributes to performance when CSP is not employed. Further clarification is needed on: (1)Performance Source: If strong results are achievable without CSP on these datasets, what factors besides the CSP are driving the overall performance improvement observed in the main experiments? (2) OOD Score Function Usage: How can the authors' OOD score function be effectively utilized in scenarios where CSP is not implemented?\n2. Synonym Handling and CSP Overlap:\nThe paper acknowledges the limitations of including synonyms in larger lexicons. However, it lacks a deeper exploration of how the CSP specifically addresses this issue. Additionally, the potential for semantic overlap within the CSP itself needs to be addressed. It would be beneficial to see a discussion on: (1) CSP vs. Synonyms: How does the design of the CSP inherently mitigate the problems introduced by synonyms in traditional lexicon expansion? (2) Mitigating Overlap: How do the authors handle potential semantic overlap between elements within the CSP? Are there strategies to ensure distinctiveness between labels?\n3. Systematic Analysis on Place and Texture Datasets:\nA more systematic analysis of the CSP's impact on the Place and Texture datasets, particularly concerning the design principles of semantic pooling, would be valuable. The authors' own experiments suggest that designing effective semantic pooling requires considering multiple factors. Therefore, a more in-depth exploration of: (1) Place and Texture Performance: How does the CSP specifically improve performance on Place and Texture datasets? Can these improvements be linked to specific design choices within the CSP? (2) Semantic Pooling Design Factors: Building on the authors' findings, what factors are crucial for designing effective semantic pooling mechanisms, especially in the context of Place and Texture datasets?\n\nOverall, this paper proposes a promising approach to improve OOD detection by innovatively using conjugate semantic labels to expand the semantic pool, which is supported by theoretical analysis and empirical results. However, I still have some questions about the method that the authors need to address.\n\nQUESTIONS:\nSee Weakness\n\nLIMITATIONS:\nSee Weakness", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a novel zero-shot out-of-distribution (OOD) detection pipeline that enhances performance by expanding the semantic pool with a conjugated semantic pool (CSP), which consists of modified superclass names that cluster similar properties from different categories. The approach aims to increase the activation probability of selected OOD labels by OOD samples while ensuring low mutual dependence among these labels. By moving beyond traditional lexicon-based expansion and using the CSP, the paper's method outperforms existing works by 7.89% in FPR95, demonstrating a significant improvement in OOD detection without the need for additional training data.\n\nSTRENGTHS:\n1. The paper is well-written and easy to understand.\n2. The supplement to the shortcomings of the NegLabel theory is very enlightening.\n3. Extensive experiments demonstrate the superiority of the method.\n\nWEAKNESSES:\n1. The primary contribution of this work is its enhancement of the semantic pool expansion issue within NegLabel, representing an incremental step within the NegLabel framework.\n\n2. There is ambiguity regarding the distinction between citations and contributions. For instance, the content related to Lemma 1 in both the preliminary section and the appendix A.1, A.2 seems to originate directly from the original NegLabel paper, yet this source is not explicitly referenced in the main text and appendix.\n\n3. In Section 3.3, the authors think that OOD images may not include their true label \"white peacock\" among the negative labels is a issue. However, this argument appears debatable. As evidenced in NegLabel's visualizations and Figures 6 and 7 of this paper, OOD images may not consistently exhibit the highest similarity to their ground truth (GT) label and could show significant similarity with multiple negative labels. Thus, even if the true GT label of an OOD image is absent from the negative labels, the NegLabel method can still effectively detect OOD data.\n\n4. From an implementation standpoint, how does including terms like \"smartphone\" and \"cellphone\" as negative labels impact the overall OOD score? The claim in Section 3.2 about the impact of synonym disruption on Lemma 1's independence seems somewhat overstated.\n\n5. I am curious whether, aside from the semantic pool, all other aspects of the technical implementation directly derive from NegLabel. Specifically, does the NegMining Algorithm, the OOD score calculation, and the Grouping Strategy align exactly with those in NegLabel? This aspect requires further clarification.\n\n6. Additionally, is Figure 3 a representation of actual distribution distance data or a schematic created by the authors? Utilizing real experimental data to support the findings presented in Figure 3 would strengthen its validity. If it is a schematic, it may introduce subjective biases from the authors.\n\n7. The paper extensively discusses the limitations of previous methods and the intended outcomes in Section 3, yet provides relatively minimal detail on the actual implementation of the proposed method. It would be beneficial for the authors to expand on the specifics of their proposed approach in Section 3.3. For instance, further elaboration on how superclass selection is conducted and integrated with adjectives would be valuable.\n\n8. According to the description in the paper, the method used to select cluster centers appears crucial. Have the authors considered employing clustering techniques for superclass selection rather than relying solely on manual selection?\n\n9. Another point of confusion arises regarding how variations in corpus size impact the acquisition of semantic pools. Does this necessitate selecting additional superclasses?\n\nQUESTIONS:\nsee Cons\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-08"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper explores how to set potential OOD labels to facilitate OOD detection with vision-language models. The paper first conducts a theoretical analysis, revealing that in addition to increasing the negative label space, it is also important to increase the probability of negative labels being activated and to reduce mutual dependence between negative labels. Therefore, the paper proposes a new strategy for constructing negative labels by introducing modified superclass names to construct a conjugated semantic pool (CSP). Experiments are conducted on standard benchmarks.\n\nSTRENGTHS:\n1. The paper theoretically analyzes the criteria for selecting negative labels and proposes a new method for designing negative labels.\n2. The motivation is clear, and the method design is straightforward and effective.\n3. The proposed method achieves state-of-the-art results on standard benchmarks.\n\nWEAKNESSES:\nWhile the overall method design is agreeable, there are some details that need clarification:\n\n1. In Line 276, the authors claim that CSPs overlapping with ID classes will not be selected as negative labels. How is this implemented, and does it significantly impact the results?\n2. In Line 321, the authors introduce the superclasses used and mention combining these superclasses with adjectives to form the CSP. Could you provide more details on the adjectives used and the selection process? Is the pairing with superclasses done randomly?\n3. How many negative labels are ultimately used? Beyond the 10,000 classes in the negative label pool, how many new negative labels are introduced? Does the number of newly introduced negative labels significantly affect OOD detection performance?\n4. In my experiments, on ImageNet, performance initially improves and then declines as the number of negative labels increases; however, in experiments with CIFAR as the ID dataset, performance continuously improves with more negative labels. Have the authors observed similar phenomena, and can they provide an explanation?\n5. It is suggested to follow Neglabel and refer to the introduced classes as \"negative labels\" instead of \"OOD labels\" to avoid confusion with the labels of practical OOD datasets.\n\nQUESTIONS:\nSee weaknesses\n\nLIMITATIONS:\nNA", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents improvements to zero-shot OOD detection methods based on pre-trained vision-language models. The study first models factors that influence the performance of existing pipelines and theoretically derive two necessary conditions for enhancing performance: expanding the OOD label candidate pool and maintaining low interdependence. Subsequently, the author analyzes why simple expansion methods do not meet these conditions and proposes constructing a conjugated semantic pool for expansion. This method meets the theoretical conditions and achieves SOTA performance on several public benchmarks.\n\nSTRENGTHS:\n1. The article presents reliable mathematical modeling and theoretical derivations, and the proposed method is simple and efficient.\n\n2. The proposed method achieves considerable performance improvements over the latest methods.\n\n3. The article features a clear structure and logical coherence, demonstrating a high standard of presentation and writing quality.\n\nWEAKNESSES:\n1. The description in the caption of Table 1 is unclear. In the experimental section, the author mentioned that the upper part of Table 1 represents traditional OOD detection methods, while the lower part pertains to detection methods based on pre-trained models like CLIP. However, this distinction is not explicitly marked in the caption, which might lead readers to misunderstand that all methods are based on the CLIP framework.\n\n2. There is a possibility that in some cases, the chosen OOD labels from CSP may also have high similarity to ID images. The authors should provide more explanation on this point.\n\n3. I do not quite understand the significance of the experiments in Table 5. I suggest the authors provide a more detailed discussion.\n\n4. In Table 9, the meaning of \"Size\" of the corpus is not clear. Does it refer to the total number of words in the dictionary or the number of selected OOD labels?\n\nQUESTIONS:\nNA\n\nLIMITATIONS:\nPlease see the weaknesses.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}

comments: SUMMARY:
The paper proposes a method for zero-shot out-of-distribution (OOD) detection using an expanded semantic pool of modified superclass names. By leveraging a pre-trained vision-language model, the approach aims to improve OOD classification performance by ensuring low mutual dependence among selected OOD labels. This method outperforms existing techniques by 7.89% in FPR95, highlighting its effectiveness in handling OOD detection tasks.

STRENGTHS:
1. The theoretical analysis of expanding NegLabel using the u-function is very clear and intuitive.
2. The experiments are conducted very comprehensively.

WEAKNESSES:
1. The method proposed in this paper lacks innovation; its main framework and basic performance are entirely derived from NegLabel.
2.The preliminary section lacks clear references to the original theory of NegLabel and the proofs in the appendix.
3.The description of the proposed method is very concise, which makes it unclear for readers to understand the details of the method design. It is recommended to provide pseudocode for the algorithm to clarify the method's design.

QUESTIONS:
see weakness

LIMITATIONS:
YES
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2024-07-14