GlocalCLIP: Object-agnostic Global-Local Prompt Learning for Zero-shot Anomaly Detection

Authors: Jiyul Ham, Yonggon Jung, Jun-Geol Baek
Primary area: applications to computer vision, audio, language, and other modalities
Keywords: Zero-shot anomaly detection, Prompt learning, Object-agnostic glocal semantic prompt design, Deep-text prompt tuning, Glocal contrastive learning

Abstract:
Zero-shot anomaly detection (ZSAD) is crucial for detecting  anomalous patterns in target datasets without using training samples, specifically in scenarios where there are distributional differences between the target domain and training data or where data scarcity arises because of restricted access. Although recently pretrained vision-language models demonstrate strong zero-shot performance across various visual tasks, they focus on learning class semantics, which makes their direct application to ZSAD challenging. To address this scenario, we propose GlocalCLIP, which uniquely separates global and local prompts and jointly optimizes them. This approach enables the object-agnostic glocal semantic prompt to effectively capture general normal and anomalous patterns without dependency on specific objects in the image. We refine the text prompts for more precise adjustments by utilizing deep-text prompt tuning in the text encoder. In the vision encoder, we apply V-V attention layers to capture detailed local image features. Finally, we introduce glocal contrastive learning to improve the complementary learning of global and local prompts, effectively detecting anomalous patterns across various domains. The generalization performance of GlocalCLIP in ZSAD was demonstrated on 15 real-world datasets from both the industrial and medical domains, achieving superior performance compared to existing methods.

Decision: reject
Reviewer scores: 5, 5, 3, 5

Reviews and discussion:
<thread>
creation date:
1730194173637

thread_type:
official_review

<thread_object>
creator: reviewer_bsxx

<content>
summary:
This paper aims to adapt CLIP for the challenging task of zero-shot anomaly detection. The authors propose enabling object-agnostic learning to capture both global and local anomaly semantics. Experimental results seem to demonstrate the effectiveness of the proposed method.

strengths:
1. This paper focuses on a challenging and valuable field that is practical to the real world.
2. The authors conduct experiments across diverse datasets to support their claims.

weaknesses:
1. This paper presents similar technological contributions and organization to AnomalyCLIP. However, the authors do not provide a comprehensive comparison and discussion with AnomalyCLIP. A detailed analysis in the introduction section is necessary to explicitly tell the main technological differences from AnomalyCLIP.
2. The manuscript lacks proper citations in several sections, notably in 3.2 (Prompt Design) and 3.5 (Training and Inference). The authors should carefully review and appropriately cite previous research to acknowledge foundational work in this field.
3. The illustration in the paper is unclear. For example, it is not evident what the term "anchor" refers to in Eq. 5, and the rationale for using V-V attention instead of other attention mechanisms, such as Q-Q or K-K attention, should be clearly explained.

questions:
See Weaknesses
</content>
</thread_object>

</thread>
<thread>
creation date:
1730310975694

thread_type:
official_review

<thread_object>
creator: reviewer_1cz8

<content>
summary:
This work investigates the task of zero-shot anomaly detection and proposes a model named GlocalCLIP, which employs a dual-branch approach for both global and local modeling of image and text inputs based on CLIP. Results across multiple datasets demonstrate the effectiveness of the proposed method.

strengths:
- The paper is clearly presented, including both text and images, making it easy to reproduce based on existing work.
- Quantitative and qualitative experiments on multiple datasets.

weaknesses:
- The novelty of the method is limited compared to AnomalyCLIP. The framework still follows the standard Winclip framework and subsequent works. The authors' claim of the "first framework" is exaggerated, as prompt tuning and V-V attention have already been used in anomaly detection, particularly in AnomalyCLIP.
- The design of global and local branches is not novel. The authors should provide a more detailed explanation of what the two types of tokens model in zero-shot anomaly detection.
- The global loss does not use multi-stage feature fusion, unlike the local loss. Additionally, the novelty of the GCL loss is trivial, lacking any impressive design.
- The introduction resembles related work, and the motivation is not convincing.
- For Fig. 1, it is unclear whether the authors aim to highlight the novelty of the zero-shot anomaly detection task or the disadvantages of other settings. I disagree with the authors' claim in Sec. 2.4 that few-shot and multi-class models (e.g., UniAD) are unfriendly for practical applications.
- The improvement over AnomalyCLIP is not significant.
- Please report the differences in training overhead, computational costs, and efficiency comparisons of different methods.

questions:
Please refer to the Weaknesses section.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730483103825

thread_type:
official_review

<thread_object>
creator: reviewer_5eoh

<content>
summary:
The authors propose GlocalCLIP, which introduces separable global and local prompts through an object-agnostic glocal semantic prompt design and jointly optimizes them to address the zero-shot anomaly detection task. They also employ contrastive learning to enhance the learning of global and local visual features. The effectiveness of the method is validated on 15 datasets.

strengths:
The authors propose GlocalCLIP, which introduces separable global and local prompts through an object-agnostic glocal semantic prompt design and jointly optimizes them to address the zero-shot anomaly detection task. They also employ contrastive learning to enhance the learning of global and local visual features.

weaknesses:
See questions.

questions:
1. The structure of the proposed method appears to be almost identical to AnomalyCLIP, except for improvements in prompt design and contrastive learning. There is no fundamental change at the framework level.
2. The design of the V-V attention layer has already been experimented with and used in AnomalyCLIP.
3. I believe the incremental experiments in Table 3 should be conducted starting from AnomalyCLIP to demonstrate the effectiveness of the newly proposed innovations.
4. In Table 3, adding F4, which corresponds to GCL in Figure 4, does not seem to significantly improve the pixel-level metrics and even shows some decline. However, in Figure 4, adding GCL shows a clear difference in pixel-level performance compared to not adding GCL. Please explain the reason for this discrepancy.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730687171073

thread_type:
official_review

<thread_object>
creator: reviewer_fhmb

<content>
summary:
This paper introduces a novel method for zero-shot anomaly detection. GlocalCLIP separates global and local prompts and optimize them together, enabling detection of general anomalies without object dependence. Through refined text prompts and a V-V attention layer for detailed image features, GlocalCLIP effectively captures both normal and abnormal patterns. Tested on 15 industrial and medical datasets, it outperforms current ZSAD methods.

strengths:
+  An object-agnostic global  and local prompts are supposed to learn normal and abnormal patterns.
+ Align visual features and text features by jointly optimizing global and local prompts through contrastive learning to enhance robustness.
+  The proposed method demonstrate excellent performance on multiple datasets.

weaknesses:
This paper primarily focuses on improvements to AnomalyCLIP, particularly with regard to enhancements in text prompts. The paper has some unclear explanations that can make it difficult to understand.

questions:
+ How to demonstrate the differences between global prompts $g_n$, $g_a$and local prompts $l_n$, $l_a$?
+ Does 'semantic prompt design' refer to Sec3.3 ? Why does it perform poorly on 'single', and even lead to a decrease in performance in table 4?
+ What causes the F3 to reduce image-level performance so drastically in medical applications in table 3? Could you provide a detailed explanation and analysis?
+ What' s the anchor prompts?
+ Some minor issues, such as line 316.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732761228785

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General response (1/3)

comment:
> ### Concern about the difference between AnomalyCLIP and Ours.

We would like to acknowledge and express our gratitude to the foundational works, including OpenCLIP [1], WinCLIP [2], PromptAD [3], AnomalyCLIP [4], which have greatly inspired this study. GlocalCLIP extends the concept of AnomalyCLIP by explicitly distinguishing between global and local information and employing Glocal Contrastive Learning to complementarily learn these two types of information.
This study highlights the critical role of prompts in decision-making for anomaly detection. By focusing on prompt learning, we aim to enhance the effectiveness of ZSAD through a complementary approach that integrates global and local perspectives.
In conclusion, we demonstrate the strength and effectiveness of our approach, which is both simple and effective, as evidenced by Tables 1, 2 and Fig. 3. We also extend our thanks to Reviewer 5eoh for recommending incremental experiments. Following your suggestion, we conducted ablation studies starting with AnomalyCLIP to clearly demonstrate the incremental improvements.

| Module | Industrial domain |  | Medical domain |  |
|:---:|:---:|:---:|:---:|:---:|
|  | **Pixel-level** | **Image-level** | **Pixel-level** | **Image-level** |
| **AnomalyCLIP** | (94.2, 81.0) | (86.1, 88.7) | (89.1, 72.0) | (94.4, 95.0) |
| **+Semantic design** | (95.0, 82.4) | (85.6, 88.3) | (90.0, 74.5) | (94.6, 95.2)   |
| **+Global local branch**| (95.3, **84.0**) | (86.2, 88.5) |(90.2, 74.4) | (89.8, 91.1) |
| **+GCL** | (**95.3**, 83.3) | (**86.7, 89.3**) | (**90.3, 74.8**) | (**94.9, 95.4**) |

We noted improvements in the Industrial domain, with Pixel-level AUROC increasing by 1.2%, AUPRO by 3.7%, Image-level AUROC by 0.7%, and AP by 1.4%. Similarly, in the Medical domain, Pixel-level AUROC increased by 1.3%, AUPRO by 3.9%, Image-level AUROC by 0.5%, and AP by 0.4%.

[References]

[1] Cherti, M., Beaumont, R., Wightman, R., Wortsman, M., Ilharco, G., Gordon, C., ... & Jitsev, J. (2023). Reproducible scaling laws for contrastive language-image learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 2818-2829).

[2] Jeong, J., Zou, Y., Kim, T., Zhang, D., Ravichandran, A., & Dabeer, O. (2023). Winclip: Zero-/few-shot anomaly classification and segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 19606-19616).

[3] Li, X., Zhang, Z., Tan, X., Chen, C., Qu, Y., Xie, Y., & Ma, L. (2024). Promptad: Learning prompts with only normal samples for few-shot anomaly detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 16838-16848).

[4] Zhou, Q., Pang, G., Tian, Y., He, S., & Chen, J. (2023). Anomalyclip: Object-agnostic prompt learning for zero-shot anomaly detection. arXiv preprint arXiv:2310.18961.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732761308730

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General response (2/3)

comment:
> ### Interpretation of the roles of global prompts and local prompts, as well as the differences between them.

The third row of Fig. 4 highlights the differences between global and local prompts. In the case of w/o GCL, the global and local prompts are independently trained. When the roles are switched, using the global prompt for pixel-level anomaly localization reveals limited performance in detecting localized anomalous regions. This serves as qualitative evidence demonstrating the distinct characteristics of global and local prompts. Specifically, the global prompt focuses on the overall semantics of an object, whereas the local prompt concentrates on identifying localized defects. Consequently, the two prompts learn to represent general normality and abnormality from different perspectives. Additionally, visualizations illustrating these differences can be found in Appendix Fig. 6 to Fig. 8.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732761390572

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General response (3/3)

comment:
> ### Explanation and Analysis of Anchor Prompts.

| Anchor prompt | Industrial domain |  | Medical domain |  |
|:---:|:---:|:---:|:---:|:---:|
|  | **Pixel-level** | **Image-level** | **Pixel-level** | **Image-level** |
| **Local prompt** | (95.1, 83.0) | (86.6, 89.0) | (90.2, 74.8) | (94.8, 95.3) |
| **Global prompt** | (**95.3, 83.3**) | (**86.7, 89.3**) | (**90.3, 74.8**) | (**94.9, 95.4**) |

The anchor prompt in Glocal Contrastive Learning is based on the global prompt. This design choice stems from the hierarchical characteristics of image semantics. While the global prompt captures the overall image context, the local prompt focuses on fine-grained details. Therefore, the global text embedding was selected as the anchor to guide the alignment of the local text embeddings.
Specifically, the anchor prompts were defined as the global normal prompt and the global anomaly prompt, enabling dual alignment that ensures the prompts are learned relative to the semantic context of global normality and abnormality. (Please refer to the updated Equations 2 and 3 for further details.)
We conducted an ablation study on the choice of anchor prompts. As shown in the table, using the global prompt as the anchor resulted in better prompt learning performance, validating the effectiveness of this design choice.
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper proposes a zero shot anomaly detection method using CLIP features. There are several such methods and the contributions on the method lie in  using a dual-branch approach to separately model global and local anomaly semantics. These are then jointly optimized via global and local prompts. The proposed method is tested on 15 industrial and medical datasets and outperforms existing ZSAD methods, demonstrating its efficacy. Reviewers like the proposed dual branch approach and the fact that it is tested on multiple datasets. However, they all feel that  improvements over AnomalyCLIP is limited. The rebuttal did not seem to address the revivers concerns. Hence, given the ratings of the paper, it is not ready for acceptance at ICLR.