Fine-grained Abnormality Prompt Learning for Zero-shot Anomaly Detection

Authors: JIAWEN ZHU, Yew-Soon Ong, Chunhua Shen, Guansong Pang
Primary area: applications to computer vision, audio, language, and other modalities
Keywords: Zero-Shot Anomaly Detection; Prompt Learning; Visual Anomaly Detection

Abstract:
Current zero-shot anomaly detection (ZSAD) methods show remarkable success in prompting large pre-trained vision-language models to detect anomalies in a target dataset without using any dataset-specific training or demonstration. However, these methods are often focused on crafting/learning prompts that capture only coarse-grained semantics of abnormality, $e.g.$, high-level semantics like "damaged", "imperfect", or "defective" on carpet. They therefore have limited capability in recognizing diverse abnormality details that deviate from these general abnormal patterns in various ways. To address this limitation, we propose FAPrompt, a novel framework designed to learn Fine-grained Abnormality Prompts for more accurate ZSAD. To this end, we introduce a novel compound abnormality prompting module in FAPrompt to learn a set of complementary, decomposed abnormality prompts, where each abnormality prompt is formed by a compound of shared normal tokens and a few learnable abnormal tokens. On the other hand, the fine-grained abnormality patterns can be very different from one dataset to another. To enhance their cross-dataset generalization, we further introduce a data-dependent abnormality prior module that learns to derive abnormality features from each query/test image as a sample-wise abnormality prior to ground the abnormality prompts in a given target dataset. Comprehensive experiments conducted across 19 real-world datasets, covering both industrial defects and medical anomalies, demonstrate that FAPrompt substantially outperforms state-of-the-art methods by at least 3%-5% AUC/AP in both image- and pixel-level ZSAD tasks.

Decision: reject
Reviewer scores: 5, 5, 5, 5

Reviews and discussion:
<thread>
creation date:
1730004229046

thread_type:
official_review

<thread_object>
creator: reviewer_rtd8

<content>
summary:
The paper focuses on zero-shot anomaly detection by learning a set of fine-grained abnormality prompts based on CLIP. These abnormality prompts consist of shared normal tokens and specific abnormal tokens, providing fine-grained semantics of abnormality. To account for dataset variance, the authors further propose a data-dependent abnormality prior (DAP) to generate instance-aware prompts from the image patch features. Extensive experiments are conducted on 19 datasets, covering both industrial and medical scenarios, comparing the proposed method with a sufficient number of zero-shot anomaly detectors.

strengths:
1. The design of compounding normal-abnormal tokens is interesting and novel.
2. The experiments are conducted extensively and demonstrate favorable performance.

weaknesses:
1. The design of abnormality prompts is not entirely consistent with the concept of "fine-grained." Although CAP leverages a set of abnormality prompts with an orthogonal constraint loss, the final abnormality embedding is derived as the average of all abnormality prompt embeddings, resulting in a vector-based abnormality prompt prototype that essentially reduces the diversity of abnormalities. If there are multiple types of anomalies within an image, is a single abnormality prompt prototype sufficient?
2. The number of abnormality prompts, K, shows very limited difference to pixel-level detection results (see Figure 7), making it difficult to evaluate the necessity of multiple abnormality prompts. Additionally, Figures 4 and 7 are missing the results when K=1.
3. The proposed DAP is technically similar to CoCoOp, and it uses the features of selected image patches instead of the global image feature. Moreover, the parameter M appears to be quite sensitive for pixel-level ZSAD.

questions:
1. I have some doubts about L087-L089. Since abnormality prompts have a different number of tokens compared to normal prompts, does this not make them easily distinguishable from normal prompts?
2. The first contribution is not accurate. The update of prompts indeed depends on human annotations in the cross-dataset scenario.
3. I suggest moving Figure 7 to the main manuscript, as both image-level and pixel-level zero-shot anomaly detection (ZSAD) are equally important for a comprehensive evaluation.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730091683971

thread_type:
official_review

<thread_object>
creator: reviewer_agpf

<content>
summary:
The paper introduces FAPrompt, a novel framework designed to improve ZSAD by enabling fine-grained detection of abnormalities without requiring dataset-specific training or detailed human annotations. Unlike existing ZSAD methods that primarily capture coarse abnormality features, FAPrompt focuses on recognizing diverse, specific abnormality details across a range of datasets, such as industrial defects and medical anomalies.

strengths:
1. FAPrompt addresses a key limitation in current ZSAD methods, which often struggle to identify fine-grained, specific abnormalities. By introducing learnable fine-grained abnormality prompts, FAPrompt improves both the accuracy and applicability of ZSAD in diverse contexts.
2. Comprehensive experiments show that FAPrompt outperforms state-of-the-art methods by notable margins (3%-5% improvements in AUC/AP) across both image- and pixel-level ZSAD tasks.

weaknesses:
1. The novelty of FAPrompt is limited. The design of CAP seems a simple combination of prompt tuning and prototype learning.
2. Besides CoOp and CoCoOp, other state-of-the-art prompt tuning approaches are expected for comparisons (e.g. PromptSRC[1] and TCP[2]).

*Reference*

[1] Khattak, Muhammad Uzair, et al. "Self-regulating prompts: Foundational model adaptation without forgetting." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.

[2] Yao, Hantao, Rui Zhang, and Changsheng Xu. "TCP: Textual-based Class-aware Prompt tuning for Visual-Language Model." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.

questions:
1. In table 4, image-level results on medical datasets show that CAP achieves the best result compared with FAPrompt, is there any further explanation?
2. How to demonstrate that CAP's ability of learning fine-grained abnormality semantics? In fig 3, fine-grained prompts (with different colors) seems not to be seperable.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730617751574

thread_type:
official_review

<thread_object>
creator: reviewer_nnbo

<content>
summary:
This study proposes FAPrompt to learn fine-grained abnormalities in Zero-shot Anomaly Detection (ZSAD) and classify anomalies based on data-dependent characteristics. To address the limitations of focusing on coarse-grained abnormality semantics, the authors introduce compound abnormality prompting (CAP) and a data-dependent abnormality prior (DAP) module, along with distinct loss terms to effectively train the CAP and DAP modules.

strengths:
The motivation of this study is reasonable, and the proposed method is novel. Additionally, the method has been validated across diverse datasets.

weaknesses:
1. The description of the learnable layers in the DAP module is missing.
2. The visualization in Figure 1 of the proposed methodology conflicts with the actual experimental results shown.

questions:
1. Please provide specific examples or criteria that distinguish 'coarse-grained' prompts from the proposed fine-grained prompts. Clarifying this would help determine if the label is used due to a single prompt approach or if there is a deeper rationale.
2. Consider alternatives to averaging the abnormality prompts in the CAP module, such as selecting the most similar prompt for each detected abnormality. If other approaches were tested, sharing the rationale or results behind choosing the current method would add clarity.
3. Could the authors provide additional evidence or clarification on how the fine-grained distributions in Figures 1 and 3 align with each other? Additional context or visual consistency would strengthen the claim that each abnormality prompt represents a specific abnormal state.
4. The explanation regarding the effectiveness of the compound prompt seems insufficient. Could more information be provided on the outcomes when combining the normal learnable token with the abnormality token? Specifically, what results were achieved if CAP and DAP were applied independently, without this combination?
5. Could the authors clarify how this approach differs from existing prompt ensemble methods? Specifically, what distinguishes the process of averaging the abnormality prompts from a traditional ensemble approach, and in what ways does this differ from selecting the highest anomaly score across individual abnormality prompts?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730627936792

thread_type:
official_review

<thread_object>
creator: reviewer_ttqm

<content>
summary:
The paper introduces a novel framework for zero-shot anomaly detection (ZSAD) that focuses on learning fine-grained abnormality prompts without requiring detailed human annotations or text descriptions. Key contributions include the development of a Compound Abnormality Prompting (CAP) module for generating complementary abnormality prompts and a Data-dependent Abnormality Prior (DAP) module aimed at enhancing cross-dataset generalization. The authors assert that their method, FAPrompt, significantly outperforms existing state-of-the-art solutions in both image- and pixel-level ZSAD tasks across 19 real-world datasets.

strengths:
1. The paper is clearly written and well-organized, making complex ideas accessible. Diagrams and figures effectively illustrate key ideas, enhancing reader comprehension.
2. The experiments are thorough, encompassing 19 diverse datasets from both industrial and medical domains. The results demonstrate substantial improvements over current state-of-the-art methods, reflecting the high quality of the proposed approach.

weaknesses:
1. There is marginal improvement in pixel-level ZSAD in Table 2. In contrast, the simple AnomalyCLIP achieves comparable, and even superior, results on industrial datasets.
2. The design of DAP is very similar to CoCoOp, and using a fixed M across images with varying scales of anomalous regions is unreasonable.
3. Missing necessary baseline: AnomalyCLIP with an ensemble of multiple abnormality prompts with orthogonal constraint loss; otherwise, it is difficult to justify the fine-grained prompts.

questions:
1. Could you provide additional evidence to substantiate the claim that the prompt-wise anomaly scores visualized in Figure 3 demonstrate the discriminability of FAPrompt? A comparison with visualizations from baseline models would effectively highlight the advantages.
2. To better verify the necessity of using multiple prompts, it is suggested to include more detailed ablation experiments, such as k=1 with and without CAP/DAP, similar to those in Table 4.
3. Ablation studies on the number of layers with learnable tokens and the length of the tokens should be included.
4. Minor: I recommend merging Table 3 and Table 4 for improved method comparison.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732303336325

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response (1/5)

comment:
We sincerely thank all the reviewers for their time, insightful feedback, and constructive suggestions. We have carefully addressed the concerns that were common across multiple reviewers below, followed by detailed responses to individual questions and comments from each reviewer. Changes in the manuscript (beyond typos) are highlighted in **orange** in the revised version of the paper. We hope they help address the reviewers’ concerns and we are looking forward to engaging in an interesting discussion.

> ### **Shared Concern \#1. Difference between DAP and CoCoOP, and the fixed Value of $M$ [TTqm, agpf, RtD8].**

**Conceptual difference between DAP and CoCoOP.**  While the data-dependent adaptation in DAP is inspired by CoCoOp, as we mentioned in Line 261-262, there are significant differences in terms of both intuition and  design:
- **Intuition.**  DAP is designed in a way to adaptively capture abnormality-aware local features in each image as the abnormality prior. This allows our method to concentrate on various localized abnormal features for learning fine-grained abnormality in the other module of $\texttt{FAPrompt}$. In contrast, CoCoOp focuses on global class semantic in the images, which involves **no** i) adaptive local normality/abnormality modeling and ii) interaction with fine-grained pattern learning modules, rendering it ineffective for our anomaly detection objectives.
 - **Design.**  To ensure that DAP can adaptively and accurately capture the sample-wise local abnormalities, we introduce a novel abnormality prior learning loss that is specifically designed for our task and not involved in any way in CoCoOp. This loss function mitigates the influence of normal samples on abnormality tokens, allowing the learning of abnormality prompt to concentrate on truly abnormal features from abnormal samples.

These two unique characteristics are key innovations of DAP, well differentiating it from CoCoOp. 

**Difference in detection performance.**  In addition, the comparative results between CoCoOp and $\texttt{FAPrompt}$ demonstrate that these designs effectively enhance both image- and pixel-level ZSAD performance (See Table 1 and Table 2). Even in our ablation study, when we removed the CAP module, the performance of DAP alone still surpasses CoCoOp (See Table 3). This further demonstrates the difference of our DAP module and its significance to ZSAD.

**Hyperparameter $M$.**  Regarding the fixed value of $M$ across images, we have conducted a sensitivity analysis in the original manuscript, as illustrated in Figure 4. The abnormality prior in DAP is derived from the top $M$ most abnormal patches. Setting $M$ too large can introduce noise or irrelevant patches, diminishing the abnormality prompt’s effectiveness. In contrast, a very small $M$ might fail to capture sufficient local abnormality features. Our experiments indicate that $M = 10$ is a balanced choice, resulting in superior performance across the datasets.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732627485753

thread_type:
official_comment

<thread_object>
creator: authors

<content>
comment:
Dear Reviewers,

Thank you once again for your constructive feedback and thoughtful suggestions. We would like to kindly remind you that we have thoroughly addressed your concerns by clarifying our contributions, emphasizing the significant performance improvements of our method, and including additional comparisons with baseline models as well as ablation studies.

As the paper revision period is near to close, we look forward to hearing your response and remain available to address any further comments or questions you might have.

Best regards,
Authors
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper is well-written and the experimental results are strong and the figures are well-drawn. What reviewers are most concerned about is the novelty of this paper, they recognized that the ideas if this paper come from the prompt tuning and prototype learning, like CoCoOp. The author's careful response solves part of the problems. However, AC still believes that the inspiration and insights of this paper is not novel enough to meet the standards of this top meeting. So the final vote is rejection.