AnomalyCLIP: Object-agnostic Prompt Learning for Zero-shot Anomaly Detection
Authors: Qihang Zhou
Guansong Pang
Yu Tian
Shibo He
Jiming Chen
Venue: ICLR 2024 poster (2024)
Keywords: Anomaly detection
Zero-shot anomaly detection
CLIP
Industrial defect inspection
Primary area: representation learning for computer vision, audio, language, and other modalities
TL;DR: We propose a novel approach, namely AnomalyCLIP, to adapt CLIP for accurate zero-shot anomaly detection.

Abstract: Zero-shot anomaly detection (ZSAD) requires detection models trained using auxiliary
data to detect anomalies without any training sample in a target dataset. It
is a crucial task when training data is not accessible due to various concerns, e.g.,
data privacy, yet it is challenging since the models need to generalize to anomalies
across different domains where the appearance of foreground objects, abnormal
regions, and background features, such as defects/tumors on different products/
organs, can vary significantly. Recently large pre-trained vision-language
models (VLMs), such as CLIP, have demonstrated strong zero-shot recognition
ability in various vision tasks, including anomaly detection. However, their ZSAD
performance is weak since the VLMs focus more on modeling the class semantics
of the foreground objects rather than the abnormality/normality in the images. In
this paper we introduce a novel approach, namely AnomalyCLIP, to adapt CLIP
for accurate ZSAD across different domains. The key insight of AnomalyCLIP
is to learn object-agnostic text prompts that capture generic normality and abnormality
in an image regardless of its foreground objects. This allows our model to
focus on the abnormal image regions rather than the object semantics, enabling
generalized normality and abnormality recognition on diverse types of objects.
Large-scale experiments on 17 real-world anomaly detection datasets show that
AnomalyCLIP achieves superior zero-shot performance of detecting and segmenting
anomalies in datasets of highly diverse class semantics from various defect
inspection and medical imaging domains. Code will be made available at https://github.com/zqhang/AnomalyCLIP.

Published as a conference paper at ICLR 2024
ANOMALYCLIP OBJECT-AGNOSTIC PROMPT LEARN- ING FOR ZERO-SHOT ANOMALY DETECTION
Qihang Zhou'; Guansong Pang"*, Yu Tian', Shibo Meli,Jiming Chen 't
"Zhejiang University 2 Sinnappree Management University {zqhang, s s18he, ejm]@zju.ee edu. cn ?gspang& & smmu. edu.sg Pytian11 @meei harvard.edu
'Harvard University
ABSTRACT
Zero-shot anomaly detection (ZSAD) requires detection models trained using aux- iliary data to detect anomalies without any training sample in target dataset. It is a crucial task when training data is not accessible due to various concern, e.g., data privacy, yet it is challenging since the models need to generalize to anoma- lies across different domains where the appearance of foreground objects, abnor- mal regions, and background features, such defects/tumors ucts/organs, can vary significantly. Recently large pre-trained vision-language models (VLMs), such as CLIP, have demonstrated strong zero-shot recognition ability in various vision tasks, including anomaly detection. However, their ZSAD performance is weak since the VLMs focus more on modeling the class semantics in the images.
different prod-
of the foreground objects rather than the abnormality/norman this paper we introduce a novel approach, namely AnomalyCLIP, to adapt CLIP for accurate 7SAD across different domains. The key insight of Anoma- lyCLIP is to learn object-agnostic text prompts that capture generic normality and abnormality in an image regardless of its foreground objects. This allows model to focus on the abnormal image regions rather than the object semantics, enabling generalized normality and abnormality recognition diverse types of objects. Large-scale experiments on 17 real-world anomaly detection datasets show that AnomalyCLIP achieves superior zero-shot performance of detecting and segmenting anomalies in datasets of highly diverse class semantics from vari- ous defect inspection and medical imaging domains. Code will be made available at /attps://githut ghubbb. com/ z www.nnn//Aomaaaaam LYCLIP
INTRODUCTION
Anomaly detection (AD) has been widely applied in various applications, such industrial defect
inspection (Bergmann et al. [019 2020 Liznerski et al. 2020 Pang et al., [2021a] Roth et al. 2022]
Huang et al. 2022 Mou et
2022 Chen et al. 2022 You et al. [022]] Ding et al.) [222 Reiss &
Hoshen| Xie et al. 2023
Zhou el al. ; Cao et al. 2023) and medical image analysis (Pang
let 202 Tian et al.
2021 Femmando et al. [021]] Qin et al. 2022; Ding et al. 2022] Liu al.
2023 Tian et al. 2023). Existing AD approaches typically assume that training examples in target
application domain are available for learning the detection models (Pang et all 202 bb Ruff et al.] 2021]. However, this assumption may not hold in various scenarios, such as i) when accessing training data violates data privacy policies (e.g., to protect the sensitive information of patients), or ii) when the target domain does not have relevant training data (e.g., inspecting defects in manufacturing line of new products). Zero-shot anomaly detection (ZSAD) is an emerging task for AD in such scenarios, to which the aforementioned AD approaches are not viable, it requires detection models to detect anomalies without any training sample in a target dataset.
Since anomalies from different application scenarios typically have substantial variations in their visual appearance, foreground objects, and background features, e.g., defects on the surface of one product that the other products, lesions/tumors on different organs, or industrial defects vs. tumors/lesions in medical images, detection models with strong generalization ability w.r.t such variations are needed for accurate ZSAD. Recently large pre-trained vision-language models (VLMs) (Radford et al. 2021 Kirillov et al., 2023) have demonstrated strong zero-shot recognition ability in various vision tasks, including anomaly detection (Jeong et al. 2023). Particularly, being
Equul contribution. Corresponding authors.

---

Published as a conference paper at ICLR 2024
Audilary data
Test
Original leat prompes
Tationed teet prompte
Learmable lect prompts
Obyect-egnaslle taxt prompts
CArgal Fur
Copsulo
Mose piats
Bran
18a
PEB
(D)
(D4GM)
(CalonD))
(OUI))
[WIGLIP)
(Cobp)
(ANonINCLPP)
Figure 1: (a)
(b)
(c)
(d)
(e)
(i)
Comparison of ZSAD results on (b) test data using (c) original text prompts in CLIP
Radford et al. (22T)) (d) tailored text prompts for AD in WinCLIP (Jeong et al. 2023), (e) learnable
text prompts for general vision tasks in CoOp (Zhou et al. 2022a), and (f) object-agnostic text prompts in our AnomalyCLIP. (a) presents a set of auxiliary data we can use to learn the text prompts.
The results are obtained by measuring the similarity between text prompt embeddings and image embeddings. The ground-truth anomaly regions suffer from poor generalization across different domains, while our AnomalyCLIP in (f) can well generalize to anomalies in diverse types of objects from different domains.
circled in red in (a) and (b). (c), (d), and (e)
pre-trained using millions/billions of image-text pairs, CLIP (Rafford et al., 2021) has been applied
to empower various downstream tasks (Zhou al., 2022b Rao et al. [2022] | Khattak et al.) [023]
Sain et al. 2023) with its strong generalization capability. WinCLIP (Jeong al., (023) is a seminal
work in the ZSAD line, which designs a large number artificial text prompts to exploit the CLIP's generalizability for ZSAD. However, the VLMs such CLIP primarily trained to align with the class semantics of foreground objects rather than the abnormality/normalii in the images, and as a result, their generalization in understanding the visual abnormality/normallll
is restricted, leading to weak ZSAD performance. Further, the current prompting approaches, using either manually defined text prompts (Jeong al., 2023) learnable prompts (Sun et al., [022] Zhou et al., (022a), often result in prompt embeddings that opt for global features for effective object semantic alignment (Zhong al. 2022; Wu al., 2023). failing capture the abnormality that often manifests in fine-grained, local features, as shown in Fig. Id and Fig. Ie
In this paper we introduce novel approach, namely AnomalyCLIP, to adapt CLIP for accurate ZSAD across different domains. AnomalyCLIP aims to learn object-agnostic text prompts that capture generic normality and abnormality in an image regardless of its foreground objects. first devises simple yet universally-effective learnable prompt template for the two general classes normality and abnormality and then utilizes both image-level and pixel-level loss functions to learn the generic normality and abnormality globally and locally in our prompt embeddings using auxiliary data. This allows our model to focus on the abnormal image regions rather the object semantics, enabling remarkable zero-shot capability of recognizing the abnormality that has similar abnormal patterns to those auxiliary data. As shown in Fig. La and Fig. [b]] the foreground object semantics be completely different in the fine-tuning auxiliary data and target data, but the anomaly patterns remain similar, e.g., scratches on metal nuts and plates, the misplacement of transistors and PCB, tumors/lesions on various organ surfaces, etc. Text prompt embeddings in CLIP fail to generalize across different domains, as illustrated in Fig. Ic but object-agnostic prompt embeddings learned by AnomalyCLIP can effectively generalize recognize the abnormality across different domain images in Fig. IF In summary, this paper makes the following main contributions.
We reveal for the first time that learning object-agnostic text prompts of normality and abnormality is a simple yet effective approach for accurate ZSAD. Compared to current text prompting approaches that primarily designed for object semantic alignment (Zhou jet al. 2002bb] Jeong et al.] 2023). our text prompt embeddings model semantics of generic abnormality and normality, allowing object-agnostic, generalized ZSAD performance.
We then introduce novel ZSAD approach, called AnomalyCLIP, in which utilize an object-agnostic prompt template and a glocal abnormality loss function (i.e., a combination of global and local loss functions) to learn the generie abnormality and normality prompts using auxiliary data. In doing so, AnomalyCLiP largely simplifies the prompt design and can effectively apply to different domains without requiring any change on its learned two
2

---

Published as a conference paper at ICLR 2024
object
Textual embedding
Normal lext prompt
Layer
N LIPPER
Similanty score
globall
(damaged object
9, Anomalous prompt
Objed-egnoedic leamable lext prompt
Text encoder
Ground truth
Locl
Similartt map
Block
Block
Block
Local visual embedcing
Auallary image
Layo
Layer
Layer
Layer
Block
Block
Visual embedding
Learnable
Vision encoder
Frozen
Layer
DPAM boyer
Layer
Original layer
Cosire simiarty
Element-wiise
Maximum
Figure 2: Overview of AnomalyCLIP. To adapt CLIP to ZSAD, AnomalyCLIP introduces object- agnostic text prompt templates capture generic normality and abnormality regardless of the object semantics. Then, we introduce glocal context optimization to incorporate global and fine- ene-ggained anomaly semantics into object-agnostic text prompt learning. Finally, textual prompt tuning and DPAM are used to enable the prompt learning in the textual and local visual spaces of CLIP.
prompts, contrasting to existing methods like WinCLIP whose effectiveness relies heavily on extensive engineering on hundreds of manually defined prompts.
Comprehensive experiments 17 datasets from various industrial and medical domains demonstrate that AnomalyCLIP achieves superior ZSAD performance of detecting and segmenting anomalies in datasets of highly diverse class semanties from defect inspection and medical imaging domains.
PRELIMINARY
CLIP consists of text encoder and visual encoder denoted T(-) and F(.). respectively. Both encoders are mainstream multi-layer networks such as VIT (Vaswani al. 2017 Dosovitskiy al.) 2020). Using text prompts is typical way to achieve the embeddings of different classes for zero- recognitio.. Particularly, a text prompt template G with the class name a can be passed through T(-) to obtain its corresponding textual embedding dc RR. The text prompt template commonly used in CLIP looks like photo of a [cls], where [cls] represents the target class name. Then F(.) encodes image T; to derive visual representations, where the class token is treated as its visual embedding (global visual embedding). and patch tokens f E R x DD referred
RD
local visual embeddings. CLIP performs zero-shot recognition by measuring the similarity between textual and visual embeddings. In specific, given target class set C and an image Ii, CLIP predicts the probability of x; belonging as follows:
eapp 9os /T)
ply cl²;) P(ge: f.) Ecee exp(<< 9c. fr >)/T)*
where 7 a temperature hyperparameter, and the operator <. represents the computation of co- sine similarity. Unlike many vision tasks that involve many objects and use the name of the objects as the class name [cls], we posit that performing ZSAD tasks using CLIP should be object-agnostic, so we propose to design two classes of text prompts (i.e., normality and abnormality) and compute classes according to Eq. [ We denote the probability of being abnormal P(g., Si) as the anomaly score. The computation is extended from global visual embeddings to local visual embeddings to derive corresponding segmentation maps S., R H and S, E H where each entry (j. k) are computed as P(gn. f. fm(li)) and P(g., pm)().))).
the possibility of these
ANOMALYCLIP. OBJECT-AGNOSTU PROMPT LEARNING 3.1 APPROACH OVERVIEW
In this paper, we propose AnomalyCLiP to adapt CLIP to ZSAD via object-agnostic prompt learn- ing. As shown in Fig. 2, AnomalyCI.IP first introduces object-agnostic text prompt templates, where
3

---

Published as a conference paper at ICLR 2024
design two generic object-agnostic text prompt templates of In. and 9. to leam generalized em- bedding for the normality and abnormality classes, respectively (see Sec. 3.2). To learn such generic text prompt templates, we introduce global and local context Optimization to incorporate global and fine-grained anomaly semantics into object-agnostic textual embedding learning. In addition, tex- tual prompt tuning and DPAM are used to support the learning in the textual and local visual spaces of CLIP. Finally, we integrate the multiple intermediate layers to provide more local visual details. During training, all modules are jointly optimized by the combination of global and local context optimization. During inference, we quantify the misalignment of textual and global/local visual embeddings to obtain the anomaly score and anomaly score map, respectively (see Sec. 3.3 3.3.
3.2 OBJECT-AGNOSTO TEXT PROMPT DESIGN
Commonly used text prompt templates in CLIP, like photo of [cls], primarily focus on object semantics. Consequently, they fail to generate textual embeddings that capture anomaly and normal semantics to query corresponding visual embeddings. To support the learning of anommly-diserininative textual embeddings, we aim to incorporate prior anomaly semantics into text prompt templates. A trivial solution to design the templates with specific anomaly types, such as A photo of a [cls] with scratches. However, the pattern of anomaly is typically unknown and diverse, so it is practically difficult to list all possible anomaly types. Therefore,
is important to define text prompt templates with generic anomaly semantics. For this purpose, we can adopt the text damaged [cls] to cover comprehensive anomaly semantics, facilitating the detection of diverse defeets such as scratches and holes. Nevertheless, utilizing such text prompt templates poses challenges in generating generic woomaly-discriminttig textual embeddings. This is because CLIP's original pre-training focuses aligning with object semantics instead of the ab- normality and normality within images. To address this limitation, we can introduce learnable text prompt templates and tune the prompts using auxiliary AD-relevant data. During the fine-tuning process, these learnable templates can incorporate both broad and detailed anomaly semantics, re- sulting in textual embeddings that are more discriminative between normality and abnormality. This helps avoid the need for manually defined text prompt templates that require extensive engineer- ing (Jeong et al.) 2003).. These text prompts are referred to as object-aware text prompt templates and defined as follows:
[V]][[]] ... [Ve][elss
[Wi]] [[]]][]].... [ [damaged][css)
9.
where [V], and [W]: (i E (I,...,
E}) are learnable word embeddings in normality and abnormality
text prompt templates, respectively.
ZSAD tasks require models to detect anomalies in previously unseen target datasets. These datasets often exhibit significant variations in object semantics among different objects, like various defects on one product vs. another, or discrepancies between industrial defects and medical imaging tu- mors. However, despite these substantial differences in object semantics, the underlying anomaly patterns could be similar. For instance, anomalies like scratches on metal nuts and plates, or the misplacement of transistors and PCB, as well as tumors on the surface of various organs, can share similar anomaly patterns. We hypothesize that the key of accurate ZSAD is to identify these generic anomaly patterns regardless of the varying semantics of different objects. Therefore, the inclusion of object semantics in object- text prompt templates is often unnecessary for ZSAD. It can even hinder the detection of anomalies in classes that have not been seen during the learning process. More importantly, excluding the object semantics from text prompt templates allows learnable text prompt templates to focus on capturing the characteristics of anomalies themselves, rather than the objects. Motivated by this, we introduce object-agnostic prompt learning, with the aim capture generic normality and abnormality within images regardless of the object semantics. Different from object-aware text prompt templates, as shown below, the object-agnostic text prompt templates replace the class name in In and 9n with ob ject. blocking out the class semantics of objects:
In. [Vi][V2] ... [Ve]jobject)
9a = [W][W2]] ... [dddmmogedd] [object].
This design empowers the object-agnostic text prompt template to learn the shared patterns of dif- ferent anomalies. As a result, the generated textual embeddings are more generic and capable of identifying anomalies across diverse objects and different domains. Further, this prompt design is versatile and can be applied to different target domains without any modification, e.g., requiring no knowledge about the object name or anomaly types in target dataset.
4

---

Published as a conference paper at ICLR 2024
3.3 LEARNING GENERIC ABNORMALITY AND NORMALITY PROMPTS
Glocal context optimization To effectively learn the object-agnostic text prompts, we devise a joint optimization approach that enables the normality and abnormality prompt learning from both global and local perspectives, namely global and local context optimization. The global context op- timization aims enforce that our object-agnostic textual embeddings are matched with the global visual embeddings of images of diverse objects. This helps effectively capture the normal/abnormal semantics from a global feature perspective. The local context optimization is introduced to enable object-agnostic text prompts to concentrate on fine-gre tne-grained, local abnormal regions from M inter- mediate layers of the visual encoder, in addition to the global normal/abnormal features. Formally, let MM be the set of intermediate layers used (i.e., M [M), our text prompts are learned by minimizing the following glocal loss function:
Liotal = Lytobal + A EMEEM Liocal
(2)
where À is a hyperparameter to balance the global and local losses. Iiglobat is a cross-entropy loss that matches the cosine similarity between the object-agnostic textual embeddings and visual embed- dings of normal/abnormal images from auxiliary data. Let S E Hi x W woaaee be the ground-truth segmentation mask, with S₁k= = 1 if the pixel is as an anomaly and Sjk= 0 otherwise, then we have
s(3,2) M, = P(gn: mm(),))) suith = P(gas STMM. where j E [1, H]. E [1, Llocal = (U
So,M.l). S) + Di S) + Dice U (U(((U((Spp (.
where Focal(.,.) -) and Dice(,,)) denote a focal loss (Lin et al. 2017) and Dice loss (Li et al. 2019) respectively. The operators Up(-) and [ represent the unsampling and concatenation along with the channel, and I represents the full-one matrix. Since the anomalous regions are typically smaller than the normal ones, we use focal loss to address the imbalance problem. Furthermore, to ensure that the model establishes accurate decision boundary, we employ the Dice loss measure the overlaps between the predicted segmentation pp(S,, M. MU. and the ground truth mask.
Refinement of the textual space To facilitate the learning of a more discriminative textual space via Eq. 2 inspired Jia al. (2022) and Khattak al. (2023). introduce text prompt tuning to refine the original textual space of CLIP by adding additional learnable token embeddings into its text encoder. Specifically, we first attach randomly initialized learnable token embeddings t'm into Tm. the m-th layer of the frozen CLIP text encoder. Then, we concatenate Kn, and the original token embeddings mm along the dimension of the channel, and forward them to Tm to get the corresponding r'm r'+1 and fm-1- To ensure proper calibration, we discard the obtained r' and initialize new learnable token embeddings t t Note that even though the output r rm+1 is discarded, the updated gradients can still be backpropagated to optimize the learnable tokens l' due to the self-attention mechanism. We repeat this operation until we reach the designated layer M'. During fine-tuning, these learnable token embeddings are optimized to refine the original textual space. More details see Appendix D
Refinement of the local visual space Since the visual encoder of CL.IP is originally pre-trained to align global object semantics, the contrastive loss used in CLIP makes the visual encoder produce representative global em- bedding for class recognition. Through the self-attention mechanism, the attention map in the visual encoder fo- cuses on the specific tokens highlighted within the red rectangle in 3bg Although these tokens may contribute to global object recognition, they disrupt the local visual semantics, which directly hinders the effective learning of text
the fine-graing fineggrained abnormality in our prompts (Li et al., 2023b). We found empirically that a diagonally prominent attention map helps reduce the disturbance from other tokens, leading to im- proved local visual semantics. Therefore, we propose a mechanism called Diagonally Prominent Attention Map to refine the local visual space, with the visual encoder kept frozen during training. To this end, we replace the original Q-K attention in the visual encoder with diagonally prominent attention, such Q-Q, K-K, and V-V self-attention schemes. As demonstrated in Fig 3c Fig 3d and Fig. Be the refined DPAM attention maps are more diagonally prominent, resulting in sub- stantially improved segmentation maps in both original CLIP and our AnomalyCLIP. Compared to
Figure 3: DPAM visualization.
5

---

Published as a conference paper at ICLR 2024
CLIP that is based on global features and manually defined text prompts, the text prompts learned by AnomalyCLIP are more fine ane-grained, enabling substantially more accurate alignment between the aormality/bnoormality prompt embeddings and the local visual embeddings across four different self-attention schemes. This, in turn, allows AnomalyCIIP to generate accurate S, and S, for the joint optimization in Eq. [] Unless otherwise specified, AnomalyCIIP utilizes V-V self-attention due to its superior overall performance. The performance of different self-attention mechanisms is analyzed in Sec. D We also provide detailed explanation about DPAM in Appendix C
Training and Inference During training, AnomalyCLIP minimizes the loss in Eq. using an auxiliary AD-related dataset. As for inference, given a test image Ig, we use the similarity score P(ga, [i) as the image-level anomaly score, with the anomaly score leaning toward one when the anomaly textual embedding Ja is aligned with global visual embedding fi. For pixel-wise pre- dictions, we merge the segmentation Sy and S. of all selected intermediate layers, fol- lowed by interpolation and smoothing operation. Formally, anomaly score map Map e ( pUp( where G. represents a Gaussian filter, and a controls smoothing.
IR Himaaee x Wimage is computed as Map = Go EE
4
EXPERIMENTS
4.1 EXPERIMENT SETUP
Datasets and Evaluation Metries We conducted extensive experiments on 17 publicly available datasets, covering various industrial inspection scenarios and medical imaging domains (including photography, endoscopy, and radiology) to evaluate the performance of AnomalyCLIP. In industrial inspection, we consider VVTec AD (Bergmann et al. 2019)) VisA (Zou et al.) 2022). MPDD (Jezek) et al.
2021), BTAD (Mishra et al.) (02T). SDD (Tabernik et al. 2020). DAGM (Wieler & Hahnj 2007 and DTD-S OTD-Synthetic (Aota et al., 2023). In medical imaging.
consider skin cancer de-
tection dataset ISIC (Gutman et al. 2016), colon polyp detection datasets CVC-ClinieDB (Bernal
ct al.
2015), CVC-ColonDB (Tajbakhsh et al. 2015). Kvasir (Jha et al. 2020), and Endo (Hicks
et al.
2021). thyroid nodule detection dataset TN3K (Gong et al.) (021). , brain tumor detection
datasets HeadCT (Salehi et al.) 2021)) BrainMRI (Salehi et al. 2021 Br35H (Hamada.) (2020), and
COVID-19 detection dataset COVID-19 (Chowdhury et al.) 2020; Rahman et al. (022)). The SOTA
competing methods include CLIP (Radford et al. (021). D. CLIP-AC (Radford et al. 2021). ,. Win-
CLIP (Jeong et al.) 2023). VAND (Chen et al. 2023). and CoOp (Zhou et al. 2022b). We provide more details about the methods and data pre- pree-prccesing in Appendix|A| The anomaly detection per- formance is evaluated using the Area Under the Receiver Operating Characteristic Curve (AUROC).
Additionally, average precision (AP) for anomaly detection and AUPRO (Bergmann et al. 2020) for anomaly segmentation are also used to provide more in-depth analysis of the performance.
Implementation details We use the publicly available CLIP mode (VIT (VITIVVVIIIILLVEEI L/148336px) as backbone. Model parameters of CLIP are all frozen. The length of learnable word embeddings E is set to 12. The learnable token embeddings are attached to the first 9 layers of the text encoder for refining the textual space, and their length in each layer is set to 4. We fine-tune AnomalyCLIP using the test data MVTee AD and evaluate the ZSAD performance on other datasets. As for MV'Tec AD, we fine-tune AomalyCIIIP on the test data of VisA. We report dataset-level results, which are averaged across their respective sub-datasets. All experiments are conducted in PyTorch-2.0.0 with a single NVIDIA RTX 3090 24GB GPU. More details can be found in Appendix A
4.2
MAIN RESULTS
ZSAD performance on diverse industrial inspection domains Table T shows the 7SAD results of AnomalyCLIP with five competing methods over seven industrial defect datasets of very different foreground objects, background, and/or anomaly types. AnomalyCLIP achieves superior ZSAD per- formance across the datasets, substantially outperforming the other five methods in most datasets. The weak performance of CLIP and CLIP-AC can be attributed to CLIP's original pre-training, which focuses on aligning object semanties rather than anomaly semantics. By using manually defined text prompts. WinCLIP and VAND achieve better results. Alternativly. CoOp adopts learn- able prompts to learn the global anomaly semantics. However, those prompts focus on the global feature and ignore the fine- -rained local anomaly semantics, leading to their poor performance on anomaly segmentation. To adapt CLIP to ZSAD, AnomalyCLIF learns object-agnostic text prompts to focus on learning the generic abnormality/normaliyy using global and local context optimization,
hhtps//gtthhh.cmmmmmfooonnnnnnnn clip clip
6

---

Published as a conference paper at ICLR 2024
Table 1: ZSAD performance comparison on industrial domain. The best performance is highlighted in red, and the second-best best is highlighted in blue. denotes results taken from original papers.
Tank
Cavegory
Dutneets
[C]
CLIP
CLIP-AC
WinCLIP
VAND
CoOp
AnumllyCLIP
Objy Aresttrr
MVTec
15
(74.1, 87.6)
(71.5, 86.4)
(918,96.5)
91.5)
(88.8, 91.8)
(91.5, 96.2)
VisA
12
(66.4, 71.5)
(60.0) 70.1)
(78.1, 81.21
(78.0, 81.4)
(62.8. 08.
(82.1.85.4
Image-level (AUROC, AP)
Ohj
MPDD
(54.3, 65.4)
(56.2.
(636. 68.9)
(73.0, 80.2)
(55.1,64.2)
(77.0, 82.00
BTAD
(34.5, 52.5)
(51.0.
(68.2, 70.9)
(73.6. 68.6)
(66.8.
(88.3, 87.3)
SDD
1
(65.7, 45.2)
(60.2. 45.7)
(84.3,
(79.8 71.4)
(74.9,05 00.1)
(84.7, 80.00
Texture
DTD-Synthetc
DAGM
12
10
(79.6, 59.0)
85.7)
(66.8,
(82.5,652 63.7)
(9.8. 79.5)
(86.4, 95.00)
(94.4, 83.8)
(87.5, 74.6)
(97.5 92.5)
(95.5, 97.0)
Obj &tastrre
MVIec AD
15
(38.4, 11.3)
(38.2, 11.6)
61.6)
(87.6, 44.00
(33.3, 6.7)
(91,181.4)
VisA
12
(46.6 14,8)
(47.8. 17.3)
(79.6, 56 86
(94.2, 96.8))
(24:2. 3.8)
(95.5,87.0)
Pixet-level (AUROC, PRO)
Obj
MPDD
BTAD
6
(62.1, 13.0)
(30.6,
(58.7. 29.1)
(32.8. K.3)
(76.4, 48.9)
27.3)
(60.8. 25.0)
(94.1.83.3)
(28.6. 3.8
(1544 2:3)
(94.2. 74.85
(9675,88.7)
SDD
65.1)
(28.9) 7.1)
Texture
DTD- DTD-Synihetic
DAGOM
12
10
(33.9. 12.5)
2.99
(23.7,5.5 5.5)
(32.7. 4.8)
(8..9.57.8)
(87.6, 65.7)
(95.3, 86.99
(82.4, 66.2)
(17.5, 2.1)
(--)
(95.691.0)
(97.9. 92.3)
Table 2: ZSAD performance comparison on medical domain. The best performance is highlighted
in red, and the second-best best is highlighted in blue. Note that the image-level medical AD datasets do not contain segmentation ground truth, so the pixel-level medical AD datasets are different from the image-level datasets.
Task
Category
Darasess
[C]
CLIP
CLIPAAC
WinCLIP
VAND
CoOp
AnomalyCLIP
HexKT
58.4)
(60.0, 60.7)
(81.8, 80.2)
(89.1,89.4
(78.4
(93.4, 91.6)
Image
Brain
(7..9.817))
86.4)
(89.3.90.99
(61.3.41.)
(90.3,92.2)
(AUROC, AP)
Br3511
(78.4.7888)
(82.7. 81.3)
(80.5, 82.2)
(93.1,92.9) 92.9)
(86.0.87.5)
(94.6, 94.7)
Chest
COVID-19 19
(23.7.42.4)
(75,0, 45.9)
(66.4. 42.99
(15.5. 8.5)
(25.3, 9.2)
(90.1, 58.7)
Skin
ISIC
(33.1.5.))
(3600 7.7)
(83.3, 55.1)
(89.4
15.9)
78.4)
CyCCColnn
15.3)
(49.5, 11.5)
(20.3.32.5)
(78.A, 64.6)
(40.5, 2.6)
(81.9, 71:31
Pixel-level (AUROC, PRO)
Colon
CVC-ClinieDB
(47.5, 18.9)
(48.5, 12.6)
(51.2,13.8)
(8I.5, 60.7)
(34-8, 2.4)
(82.9, 67.8)
Kvasir
(44.6, 17.7)
(45.0) 16.8)
(69.7, 24.5)
(75.0, 36.2)
(44.1. 3.5)
(78.9, 45.61
Fido
(45.2,
(46.6, 12.6)
54.9)
(84.1, 63.6)
Thyroid
TNSK
(42.3.
(35.6.5.2)
(70.7, 39.8)
(73.6 37.8)
(34 9.5)
(81.5, 50.4)
enabling the modeling of both global and local abnormaliyynorraaiiyy Our resulting prompts can also generalize to different datasets from various domains. To provide more intuitive results, we visualize the anomaly segmentation results of AnomalyCLIP, VAND, and WinCLIP across differ- ent datasets in Fig. 4 Compared to VAND and WinCLIP, AnomalyCLIP can perform much more accurate segmentation for the defects from different industrial inspection domains.
Generalization from defect datasets to diverse medical domain datasets To evaluate the gen- eralization ability of our model, we further examine the ZSAD performance of AnomalyCLIP on 10 medical image datasets of different organs across different imaging devices. Table 2 shows the results, where learning- using MVTec AD data. It is remarkable that methods like AnomalyCLIP and VAND obtain promis- ing ZSAD performance on various medical image datasets, even though they are tuned using a defect detection dataset. Among all these methods, AnomalyCLIP is the best performer due to its strong generalization brought by object-agnostic prompt learning. As illustrated in Fig, 4 Anoma- lyCLIP can accurately detect various types of anomalies in diverse medical images, such as skin cancer regions in photography images, colon polyps in endoscopy images, thyroid nodules in ultra- sound images, and brain tumors in MRI images, having substantially better performance in locating the abnormal lesion/tumor regions than the other two methods WinCLIP and VAND. This again demonstrates the superior ZSAD performance of AnomalyCLIP in datasets of highly diverse object semantics from medical imaging domains.
methods, including AnomalyCLIP, VAND and CoOp, are all tuned
Can we obtain better ZSAD performance if fine-tuned using medical image data? Comparing the promising performance in industrial datasets, AnomalyCLIP presents relatively low perfor- mance in medical datasets. This is partly due to the impact of auxiliary data used in our prompt learning. So, then we examine whether the ZSAD performance on medical images can be improved if the prompt learning is trained on an auxiliary medical dataset. One challenge is that there are no available large 2D medical datasets that include both image-level and pixel-level annotations for our training. To address this issue, we create such dataset based on ColonDB (More details see Appendix (A), and then optimize the prompts in AnomalyCLIP and VAND using this dataset and evaluate their performance on the medical image datasets. The results are presented in Table 3 AnomalyCLIP and VAND largely improve their detection and segmentation performance compared to that fine-tuned on MVTec AD, especially for the colon polyp-related datasets such as CVC- ClincDB, Kvasir, and Endo (note that these datasets are all from different domains compared to the fine-tuning ColonDB dataset). AnomalyCLIP also exhibits performance improvement in detecting
7

---

Published as a conference paper at ICLR 2024
brain tumors in datasets such HeadCT, BrainMRI, and Br35H. This is attributed to the visual similarities between colon polyps and brain tumors. Conversely, the symptom of the colon polyp differs significantly from that of diseased skin or chest, leading to performance degradation in ISIC and COVID-19. Overall, compared to VAND, AnomalyCI.IP performs consistently better across all datasets of anomaly detection and segmentation.
Industrial domain
Medical domain
Table 3: ZSAD performance on medi- cal iamges when fine-tuned by medical image datasets.
Truh
Oura
Casegocy
Dutasets
VAND
AnomalyCLIP
Clessitiation
HeadC1
189.1, 89.31
(93.5,901)) 95.13
VAND
Drain
BrainMRI
(89.3,909)
(95.5,97 91.2)
(93.1, 92.9
19
(15.5.83)
(7.79.33.7)
Segmentation
Skin
ISIC
(58.8,
(830)
82.3)
(924482.9)
Colon
Kinsir
(87.6. 39.3)
(92.5, 61.5)
Ende
(88.5, 81.99
(93.2, 84.8)
Figure 4: Segmentation visualization.
Thyrodd
TN3K
(60.5,
(79.2.
Object-agnostic vs. object-aware prompt learning To study the ef- fectiveness of object-agnostic prompt learning in AnomalyCLIP, we com- pare AnomalyCLIP with its vari- ant that uses an object-aware prompt template. The performance gain of AnomalyCLIP to its object-aware prompt learning variant is shown in Fig. where positive values indicate our object-agnostic prompt templates better than the object-aware one. It is clear that our object-agnostic prompt learning performs much better than, or on par with, the object-aware version in both image- level and pixel-level anomaly detection. This indicates that having object-agnostic prompts helps better learn the generic abnormality and normality in images, as the object semantics are often not helpful, can even become noisy features, for the ZSAD task.
Imae- pegee-Mvel mair
Pace mennc AUROC
Image level
AUPAO
Peformance gain
MVToc
MIA
MEDO
DAGH
DTD
Performance gain of using object-agnostic prompts compared object-aware prompts.
Figure 5:
4.3 ABLATION STUDY
Module ablation We first validate the effectiveness of different high-level modules of our Anoma- lyCLIP, including DPAM (Ti), object-agnostic text prompts (T2), added learnable tokens in text encoders (T3), and multi-layer visual encoder features (Ta). As shown in Table 4 each module contributes to the remarkable performance of AnomalyCLIP. DPAM improves the segmentation performance by enhancing local visual semantics (Ti). Object-agnostic text prompts focus on the abnormality/normaliyy within images instead of the object semantics, allowing AnomalyCLIP de- tect anomalies in diverse unseen objects. Therefore, introducing object-agnostic text prompts (7₂) significantly improves AnomalyCLIP Furthermore, text prompt tuning (Ta) also brings performance improvement via the refinement of original textual space. Finally. Ta integrates multi-layer visual semantics to provide more visual details, which further promotes the performance of ZSAD.
Context optimization Next we examine key modules in detail. The object- prompt learn- ing the most effective module, and driven by our glocal context optimization, so we consider two different optimization terms, local and global losses, in Eq. 2 The results are shown in Table Both global and local context optimization contribute to superiority of AnomalyCLIP. Global context optimization helps to capture global anomaly semanties, thus enabling more accurate image- level detection. Compared to global context optimization, local context optimization incorporates
Table Module ablation.
Table Context optimization ablation.
Midule
Pinel level
Image level
Pinel level
VisA
Image level
Local.
Global
Pisei-tevel
MV AD
Image-levat
Pixel-tevel
VisA
Image- level
+Ti
(68-4, 47-4)
15:4)
(06.3, 83.3)
(54.8 32.7)
17.1)
(54.4
18..8
31.77
(899, 95.4)
1866, 78.1)
62.1
(61.1.69) 69.1)
(82.2.84 84.9)
+7g
+Tg
$1.1) (90.1)
(89.5, 81
(91.)) 96.1)
090.8 96.0)
195.2, (0.0)
195.0. 35.3)
(81.9) 85.21
(81.7, 85-21
/
x
(91.)
(91.0, 80.43
(89.9 96.00
96.20
(95.5
(952 86.5)
(.2.1.85.4)
(79.5, 83.2)
(91.1,
8

---

Published as a conference paper at ICLR 2024
Pixel-level AUROC
AUPRO MV10c AD
Image-level AUROC
AP
MV AD
MVVicc AD
MV AD
100
ViSA
100
DTD
VISA
100
DTD
100
MPDD
DAGM MPDD
DAGM
DAGM
DAGM
BTAD
5D0
STAD
SDD
BTAD
SDD
BTAD
SDD
VY woard
O-0 attetton
K of
Figure 6: DPAM component ablation.
local anomaly semantics, which improves pixel-level performance and complements image-level performance. By synthesizing these two optimization strategies, AnomalyCLIP generally achieves better performance than using them individually.
DPAM strategy ablation AnomalyCLIP uses V-V self-attention by default. Here we study the effectiveness of using two other DPAM strategies, including Q-Q and K-K self-attention, result- ing in two AnomalyCLIP variants, namely AnomalyCLIP 2q and AnomalyCLIP The compari- son results are presented in Fig. 6 AnomalyCLiP qq achieves similar segmentation capabilities as AnomalyCLIP but suffers from degradation in detecting image-level anomalies. Conversely, while AnomalyCLIPss performs well in anomaly classification, its segmentation performance is less ef- fective than AnomalyCLIP and AnomalyCIIIP qq- The V-V self-attention is generally recommended in AnomalyCLIE Detailed analysis of DPAM can be seen in Appendix C
5
RELATED WORK
Zero-shot anomaly detection ZSAD relies on the model's strong transferability to handle un- seen anomalies (Aota al., 2023). CLIP-AD (Liznerski et al., 2022) and ZOC (Esmacilpour al. 2022) are early studies in utilizing CLIP for ZSAD, but they mainly focus on the anomaly classi- fication task. ACR (Li et al. 2023a requires tuning on turget-domain-relevaa ZSAD on different target datasets, while AnomalyCLIP can be applied to different datasets after it trained on one general dataset. A very recent approach WinCLIP (Jeong et al. (023) presents seminal work that leverages CLIP for zero-shot classification and segmentation. It uses large number of hand-crafted text prompts and involves multiple forward passes of image patches for anomaly segmentation. To tackle this inefficiency, VAND (Chen et al., 2023) introduces learnable linear projection techniques to enhance the modeling of local visual semantics. However, these ap- proaches suffer from insufficiently generalized textual prompt embeddings, which degrades their performance in identifying anomalies associated with various unseen object semantics. Anomaly- CLIP utilizes only two object-agnostic learnable text prompts optimize the generic text prompts of abnormality and normality, and it can obtain segmentation results with just a single forward pass. AnomalyGPT (Gu et al. 2023) is a concurrent work in utilizing foundation models for AD, but is designed for unsupervised/few-slle
auxiliary data for
AD with manually crafted prompts.
Rather than resorting to full network fine-tuning, prompt learning emerges as purameter-efficiees alternative to achieve satisfactory results (Sun et al. 2022 Zhou et al. 2022a Khattak et al. 2023 Kim et al. 2023). CoOp (Zhou et al. (022b) introduces learnable text prompts for few-shot classification. On this basis, DenseCLIP (Rao et al. 2022) extends prompt learning to dense prediction tasks with an extra image decoder. Instead, AnomalyCI.IP proposes object- agnostic prompt learning for anomaly detection, blocking out the potential adverse impact of the diverse object semanties on anomaly detection. Benefiting from the glocal context optimization, AnomalyCLit CLIP can capture local anomaly semantics such that we can simultancously perform clas- sification and segmentation tasks without an additional decoder network like Rao et al. (0022).
Prompt learning
CONCLUSION
In this paper, we tackle a challenging yet significant area of anomaly detection, ZSAD, in which there is no available data in the target dataset for training. We propose AnomalyCLIP to improve the weak generalization performance of CLIP for ZSAD. We introduce object-agnostic prompt learn- ing to learn generic abnormality/normaliyy text prompts for generalized ZSAD image datasets of diverse foreground objects. Further, to incorporate global and local anomaly semantics into Anoma- lyCLIP, we devise a joint global and local context optimization to optimize the object-agnostic text prompts. Extensive experimental results on 17 public datasets demonstrate that AnomalyCLIP achieves superior ZSAD performance.

---

Published as a conference paper at ICLR 2024
REPRODUCIBILITY STATEMENT
To ensure the reproducibility and completeness of this paper, we have included an Appendix consist- ing of five main sections. In Appendix A we provide more implementation details of AnomalyCLIP, as well as the reproduction of other baseline methods. Appendix B provides key statistics about datasets used in our experiments and the implementation of the auxiliary medical dataset for prompt tuning. Appendix D supplements the main paper with additional results and ablations. Further vi- sualizations of similarity scores and maps are detailed in Appendix Additionally, the main paper presents only the average performance in each dataset that contains a number of data subsets, for which we present their ine-grained detection results in Appendix F Our code will be made publicly accessible once the paper is accepted.
REFERENCES
Toshimichi Aota, Lloyd Teh Tzer Tong, and Takayuki Okatani. Zero-shot versus many-shot: Un- supervised texture anomaly detection. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp- 5564-5572, 2023.
Paul Bergmann, Michael Fauser, David Sattlegger, and Carsten Steger. Mvtec ad-a comprehen- sive real-world dataset for unsupervised anomaly detection. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pp. 9592-9600, 2019.
Paul Bergmann, Michael Fauser, David Sattlegger, and Carsten Steger. Uninformed students: Student-teacher anomaly detection with discriminative latent embeddings. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 4183-4192, 2020.
2020.
Jorge Bernal, F Javier Sánchez, Gloria Fernández-Esparrachh Debora Gil, Cristina Rodríguez, and Fernando Vilariño. Wm-dova maps for accurate polyp highlighting in colonoscopy: Validation vs. saliency maps from physicians. Computerized medical imaging and graphics, 43:99-111, 2015.
Tri Cao, Jiawen Zhu, and Guansong Pang. Anomaly detection under distribution shift. arXiv preprint arXiv:2303. 13845, 2023.
Xuhai Chen, Yue Han, and Jiangning Zhang. A zero-/few-shot anomaly classification and segmen- tation method for cvpr 2023 vand workshop challenge tracks 1&2: 1st place on zero-shot ad and 4th place on few-shot ad. arXiv preprint arXiv:2305.17338 2023.
Yuanhong Chen, Yu Tian, Guansong Pang, and Gustavo Carneiro. Deep one-class classification via interpolated gaussian descriptor. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pp. 383-392, 2022.
Muhammad E. H. Chowdhury, Tawsifur Rahman, Amith Khandakar, Rashid Mazhar, Muham- mad Abdul Kadir, Zaid Bin Mahbub, Khandakar Reajul Islam, Muhammad Salman Khan, Atif Iqbal, Nasser Al Emadi, Mamun Bin Ibne Reaz, and Mohammad Tariqul Islam. Can help in screening viral and covid-19 pneumonia? IEEE Access, 8:132665-332777 2020. doi: 10.11O9/AC 1100///ACCSSS22200300022222 2020.3010287.
Hanqiu Deng and Xingyu Li. Anomaly detection via reverse distillation from one-class embedding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp- 9737-9746, 2022.
Choubo Ding, Guansong Pang, and Chunhua Shen. Catching both gray and black swans: Open-set supervised anomaly detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.
2022.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenbomn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.1 1929, 2020.
10

---

Published as a conference paper at ICLR 2024
Sepideh Esmacilpour, Bing Liu, Eric Robertson, and Lei Shu. Zero-shot out distribbbin detec- tion based on the pre-trained model clip. In Proceedings of the AAAI conference on artificial intelligence, volume 36, pp. 6576, 2022.
Tharindu Fernando, Harshala Gammulle, Simon Denman, Sridha Sridharan, and Clinton Fookes. Deep learning for medical anomaly detection-a a survey, ACM Computing Surveys (CSUR), 54(7): 1-37, 2021.
Haifan Gong, Guanqi Chen, Ranran Wang. Xiang Xie, Mingzhi Mao, Yizhou Yu, Fei Chen, and Guanbin Li. Multi-task learning for thyroid nodule segmentation with thyroid region prior. In 2021 IEEE 18th international symposium on biomedical imaging (ISBI), pp. 257-261. IEEE, 2021.
Zhaopeng Gu, Bingke Zhu, Guibo Zhu, Yingying Chen, Ming Tang, and Jinqiao Wang. Anoma- lygpt: Detecting industrial anomalies using large vision-language models, 2023.
David Gutman, Noel C. F. Codella, Emre Celebi, Brian Helba, Michael Marchetti, Nabin Mishra, and Allan Halpern. Skin lesion analysis toward melanoma detection: challenge at the inter- national symposium on biomedical imaging (isbi) 2016, hosted by the international skin imaging collaboration (isic), 2016.
A. Hamada.
Br35h:
Brain tumor detection 2020.
Online. Available:
2020.
Steven A Hicks, Debesh Jha, Vajira Thambawita, Pål Halvorsen, Hugo Hammer, and Michael A Riegler. The endotect 2020 challenge: evaluation and comparison of classification, segmentation and inference time for endoscopy. In Pattern Recognition. ICPR International Workshops and Challenges: Virtual Event, January 10-15, 2021, Proceedings, Part VIII, pp. 263-274. Springer, 2021.
Chaoqin Huang, Haoyan Guan, Aofan Jiang, Ya Zhang, Michael Spratling, and Yan-Feng Wang. Registration based few-shot anomaly detection. In European Conference on Computer Vision, pp. 303-319. Springer. 2022.
Jongheon Jeong, Yang Zou, Taewan Kim, Dongqing Zhang, Avinash Ravichandran, and Onkar Dabeer. Winelip: Zero-/few-shot anomaly classification and segmentation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 19606-19616 2023.
Stepan Jezek, Martin Jonak, Radim Burget, Pavel Dvorak, and Milos Skotak. Deep learning-baat based defect detection of metal parts: evaluating current methods in complex conditions. In 2021 13th International congress on ultra modern telecommunications and control systems and workshops (ICUMT), pp. 66-71. IEEE, 2021.
Debesh Jha, Pia H Smedsrud, Michael A Riegler, Pal Halvorsen, Thomas de Lange, Dag Johansen, and Håvard Johansen. Kvasir-seg: A segmented polyp dataset. In MultiMedia Modeling: 26th International Conference, MMM 2020, Daejeon, South Korea, January 5-8, 2020, Proceedings, Part 11 26, pp. 451 462. Springer, 2020.
Menglin Jia, Luming Tang, Bor-Chun Chen, Claire Cardie, Serge Belongie, Bharath Hariharan, and Ser-Nam Lim. Visual prompt tuning. In European Conference on Computer Vision, pp. 709-727. Springer, 2022.
Muhammad Uzair Khattak, Hanoona Rasheed, Muhammad Maaz, Salman Khan, and Fahad Shah- baz Khan. Maple: Multi-modal prompt learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 19113- 2023.
Kwanyoung Kim, Yujin Oh, and Jong Chul Ye. Zegot: Zero-shot segmentation through optimal transport of text prompts. arXiv preprint (tt::2331.1277/1 2023.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv: Ieie::442266000 6980, 2014.
11

---

Published as a conference paper at ICLR 2024
Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander Berg, Wan- Yen Lo, et al. Segment anything. arXiv preprint arXiv:2330..00000 2023.
Aodong Li, Chen Qiu, Marius Kloft, Padhraic Smyth, Maja Rudolph, and Stephan Mandt. Zero-shot anomaly detection via batch normalization. In Thirty-seventh Conference on Neural Information Processing Systems, 2023a.
Xiaoya Li, Xiaofei Sun, Yuxian Meng, Junjun Liang, Fei Wu, and Jiwei Li, Dice loss for data- imbalanced nlp tasks. arXiv preprini arXiv: 1.02855, 2019.
Yi Li, Hualiang Wang, Yiqun Duan, and Xiaomeng Li. Clip surgery for better explainability with enhancement in open-vocabulary tasks. arXiv preprint arXiv:2304.05663, 2023b.
Tsung-Yi Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. In Proceedings of the IEEE international conference on computer vision, pp. 5980-2988, 2017.
Jie Liu, Yixiao Zhang, Jie-Neng Chen, Junfei Xiao, Yongyi Lu, Bennett A Landman, Yixuan Yuan, Alan Yuille, Yucheng Tang, and Zongwei Zhou. Clip-driven universal model for organ segmen- tation and tumor detection. arXiv preprint arXiv:2301.0078 2023.
Philipp Liznerski, Lukas Ruff, Robert A Vandermeulen, Billy Joe Franks, Marius Kloft, and Klaus- Robert Müller Explainable deep one-class classification. arXiv preprint 2rXiv:2007.01760 2020.
Philipp Liznerski, Lukas Ruff, Robert A Vandermeulen, Billy Joe Franks, Klaus-Rober Müller, and Marius Kloft. Exposing outlier exposure: What can be learned from few, one, and zero outlier images. arXiv preprint arXiv:2205. 1144, 2022.
Pankaj Mishra, Riccardo Verk, Daniele Fornasier, Claudio Piciarelli, and Gian Luca Foresti. Vt-adl: A vision transformer network for image anomaly detection and localization. In 2021 IEEE 30th International Symposium Industrial Electronics (ISIE), pp. 01-06. IEEE, 2021.
Shancong Mou, Xiaoyi Gu, Meng Cao, Haoping Bai, Ping Huang, Jiulong Shan, and Jianjun Shi. Rgi: robust gan-inversion for mask-free image inpainting and unsupervised pixel-wise anomaly detection. In The Eleventh International Conference on Learning Representations, 2022.
Guansong Pang, Choubo Ding, Chunhua Shen, and Anton van den Hengel. Explainable deep few- shot anomaly detection with deviation networks. arXiv preprint 20Xiv:2088.00422 2021a.
Guansong Pang, Chunhua Shen, Longbing Cao, and Anton Van Den Hengel. Deep learning for anomaly detection: A review. ACM computing surveys (CSUR), 54
Ziyuan Qin, Huahui Yi, Qicheng Lao, and Kang Li. Medical image understanding with pretrained vision language models: A comprehensive study. arXiv preprint arXiv:2209.15517. 2022.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference machine learning, pp- 8748-8763. PMLR, 2021.
Tawsifur Rahman, Amith Khandakar, Yazan Qiblawey, Anas Tahir, Serkan Kiranyaz, Saad Bin Abul Kashem, Mohammad Tariqui Islam, Somaya AI Maadeed, Susu Zughaier, Muhammad Salman Khan, et al. Exploring the effect of image enhancement techniques on covid-19 detection using chest x-ray images. Computers in biology and medicine, 132:104319, 2021.
Yongming Rao, Wenliang Zhao, Guangyi Chen, Yansong Tang, Zheng Zhu, Guan Huang, Jie Zhou, and Jiwen Lu. Denseclip: Language-guide dense prediction with context-aware prompting. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 18082-18091. 2022.
Tal Reiss and Yedid Hoshen. Mean-shifted contrastive loss for anomaly detection. In Proceedings of the AAAI Conference Artificial Intelligence, volume 37, pp. 2155-2162, 2023.
12

---

Published as a conference paper at ICLR 2024
Karsten Roth, Latha Pemula, Joaquin Zepeda, Bernhard Schölkopf, Thomas Brox, and Peter Gehler. Towards total recall in industrial anomaly detection. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 14318-14328, 2022.
Lukas Ruff, Jacob R Kauffmann, Robert A Vandermeulen, Grégoire Montavon, Wojeiech Samek, Marius Kloft, Thomas G Dietterich, and Klaus- Müller. A unifying review of deep and shallow anomaly detection. Proceedings of the IEEE, 109(5):56-775,, 2021.
Aneeshan Sain, Ayan Kumar Bhunia, Pinaki Nath Chowdhury, Subhadeep Koley, Tao Xiang, and Yi-Zhe Song. Clip for all things zero-shot sketch-based image retrieval, fine-grained or not. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp- 2775, 2023.
Mohammadreza Salchi, Niousha Sadjadi, Soroosh Baselizadeh, Mohammad H Rohban, and Hamid R Rabiee. Multiresolution knowledge distillation for anomaly detection. In Proceed- ings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 14902-14912, 2021.
Ximeng Sun, Ping Hu, and Kate Saenko. Dualcoop: Fast adaptation to multi-label recognition with limited annotations. Advances in Neural Information Processing Systems, 35:30569-30582, 2022.
Domen Tabernik, Samo Šela, Jure Skvarč, and Danijel Skočaj. Segmentation--asee deep-learning approach for surface-defect detection. Journal of Intelligent Manufacturing, 31(3):759-776, 2020.
Nima Tajbakhsh, Suryakanth R Gurudu, and Jianming Liang. Automated polyp detection in colonoscopy videos using shape and context information. IEEE transactions on medical imaging, 35(2):300644, 2015.
Yu Tian, Guansong Pang, Fengbei Liu, Yuanhong Chen, Scon Ho Shin, Johan W Verjans, Rajvin- der Singh, and Gustavo Carneiro. Constrained contrastive distribution learning for unsupervised anomaly detection and localisation in medical images. In Medical Image Computing and Com- puter Assisted Intervention-MICCAI 2021: 24th International Conference, Strasbourg, France, September 27-Ociober 1, 2021, Proceedings, Part 24, pp. 128-140. Springer, 2021.
Yu Tian, Fenghei Liu, Guansong Pang. Yuanhong Chen, Yuyuan Liu, Johan W Verjans, Rajvinder Singh, and Gustavo Carneiro. Self-supervised pseudo multi-class pre-training for unsupervised anomaly detection and segmentation in medical images. Medical Image Analysis, pp. 102930, 2023.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural informa- tion processing systems, 30, 2017.
Matthias Wieler and Tobias Hahn. Weakly supervised learning for industrial optical inspection. In DAGM symposium in, volume 6, 2007.
Size Wu, Wenwei Zhang, Sheng Jin, Wentao Liu, and Chen Change Loy. Aligning bag of regions for open- opeallocab object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 15254-15264, 2023.
Guoyang Xie, Jingbao Wang, Jiaqi Liu, Feng Zheng, and Yaochu Jin. Pushing the limits of fewshot anomaly detection in industry vision: Graphcore. arXiv preprint arXiv:2301. 2023.
Zhiyuan You, Lei Cui, Yujun Shen, Kai Yang, Xin Lu, Yu Zheng, and Xinyi Le. A unified model for multi-class anomaly detection. Advances in Neural Information Processing Systems, 35:4571- 1- 4584, 2022.
Yiwu Zhong, Jianwei Yang, Pengchuan Zhang, Chunyuan Li, Noel Codella, Liunian Harold Li, Luowei Zhou, Xiyang Dai, Lu Yuan, Yin Li, et al. Regionclip: Region- based language-iman pretraining. In Proceedings the IEEE/CVF Conference on Computer Vision and Pattern Recog- nition, pp. 15793-16803 2022.
13

---

Published as a conference paper at ICLR 2024
Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Conditional prompt learning for vision-language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.
2022a.
Kaiyang Zhou, Jingkang Yang. Chen Change Loy, and Ziwei Liu. Learning to prompt for vision- language models. International Journal of Computer Vision, 130(9))2337-2348, 2022b.
Qihang Zhou, Shibo He, Haoyu Liu, Tao Chen, and Jiming Chen. Pull & push: Leveraging dif-
ferential knowledge distillation for efficient unsupervised anomaly detection and localization. IEEE Transactions on Circuits and Systems for Video Technology, 33(5)2176-2189, 2023. doi: 10.1100/TCSVT::222.22115
Yang Zou, Jongheon Jeong, Latha Pemula, Dongqing Zhang, and Onkar Dabeer. Spot-the-differeme self-supervised pre-training for anomaly detection and segmentation. In European Conference on Computer Vision, pp. 392 022400..66 Springer, 2022.
A IMPLEMENTATION DETAILS AND BASELINES
A.1 IMPLEMENTATION DETAILS
In this paper, we use the publicly available CLIP model (VII-L/140336px as our backbone. Model parameters of CLIP all frozen. The length of learnable text prompts M is set 12. These trainable text tokens are attached to the first 9 layers of the text encoder, and each text token has length of 4. We fine-tune AnomalyCLIP on the test data on MVTec AD and test the performance for other datasets. As for MVTec AD, we fine-tune AomalyCLIP on test data on VisA. To provide adequate visual details, we extract local visual embeddings ex, from the 6-th, 12-th, 18-th, and 24-th layers of the visual encoder. Starting from the 6-th layer, we apply DPAM to the architecture of the visual encoder according to Sec. 3.3 Additionally, we set the balanced weight
in our loss function. The input images are resized to size of 518 with batch size 8, and we use the Adam optimizer (Kingma & Ba 2014) with learning rate of 0.001 to update model parameters. During testing, apply Gaussian filter with 4 to smooth the anomaly score map. The epoch is 15 for all experiments, which are performed Py PoTrcch-2.0m 2.0.0 with a single NVIDIA RTX 3090 24GB GPU.
A.2 BASELINES
To demonstrate the superiority of Anomlay-CL.n compare AnomlayCLIP with broad SOTA baselines. Implementation and reproduction details are given as follows:
CLIP (Radford et al.] 202I). CLIP is a powerful zero-shot classification method. two classes of text prompt templates
To perform the anomaly detection task, A photo of a normal [cls] and A photo of an anomalous [cis], }, where cls denotes the target class name. The anomaly score is computed according to Eq. [ As for anomaly segmentation, we extend the above computation to local visual embedding derive the segmentation.
CLIP-AC (Radford et al. 2021). Different from CLIP, CLIP-AC employs an ensemble of text prompt templates that are recommended for ImageNet dataset (Radford et al. [222)) We average the generated textual embeddings of normal and anomaly classes respectively, and compute the probability and segmentation in the same way as CLIP.
WinCLIP (Jeong et al.) 2023). WinCLIP is SOTA ZSAD method. They design large set of hand-crafted text prompt templates specific to anomaly detection and use window scaling strategy to obtain anomaly segmentation. All parameters are kept the same in their paper.
VAND is an improved version of WinCLIP. They first adjust the text prompt templates and then introduce learnable linear projections to improve local visual semantics to derive more accurate segmentation. All parameters are kept the same in their paper.
VAND (Chen et al. 2023
14

---

Published as a conference paper at ICLR 2024
Table 6: Key statistics the datasets used.
Normal and
Dataset
Category
Modalities
C|
Usage
anomalous samples
MVTec AD
Obj &texture
Photography
15
(467, 1258)
Industrial defect detection
VisA
Photography
12
(962, 1200)
Industrial defect detection
MPDD
Photography
(176, 282)
Industrial defect detection
Obj
BTAD
Photography
3
(451, 290)
Industrial defect detection
SDD
Photography
1
(181, 74)
Industrial defect detection
DAGM
Photography
10
(6996, 1054)
Industrial defect detection
Texture
DTD-Synthetic
Photography
12
(357, 947)
Industrial defect detection
ISIC
Skin
Photography
1
(0, 379)
Skin cancer detection
CVC-ChmecDB
Endoscopy
(0, 612)
Colon polyp detection
CVC-ColonDB
Endoscopy
(0, 380)
Colon polyp detection
Kvasir
Endoscopy
(0, 1000)
Colon polyp detection
Endo
Endoscopy
(0, 200)
Colon polyp detection
Radiology
TN3K
Thyroid
1
(0, 614)
Thyroid nodule detection
(Utralsound)
Radiology
HeadCT
(100, 100)
Brain tumor detection
(CT)
Radiology
BrainMRI
(98, 155)
Brain tumor detection
Brain
(MRI)
Radiology
Br35H
(1500, 1500)
Brain tumor detection
(MRI)
Radiology
COVID-19
Chest
(1341, 219)
COVID-19 detection
(X-ray)
CoOp (Zhou et al. 2022b). CoOp is representative method for prompt learning. To adapt CoOp to ZSAD, we replace its learnable text prompt templates [V.][V)]...[VvV] ccs]] with normality and abnormality text prompt templates, where V, is the learnable word em- beddings. The normality text prompt template defined as [Vi [Vi][[]] [V [Vx][[[ [
[c|s]] s).
and the abnormality one is defined as [V [V2- [VN]] [Ww]]annoo anoma ous [cls]. ss). Anomaly proba- bilities and segmentation are obtained in the same way as for Anomaly CLIP. All parameters are kept the same in their paper.
B DATASET
More dataset details In this paper, we conduct extensive experiments on public datasets span- ning two domains and three modalities to validate the effectiveness of our methods. Since we just use the test data of Datasets, we present the relevant information of their test sets in Table 6 We apply the default normalization of OpenCI.IP to all datasets. After normalization, we resize the images to a resolution of (518, 518) to obtain an appropriate visual feature map resolution. It should be noted that the original image size of SDD has width of 500 and height ranging from 1,240 to 1,270. Before processing, we vertically divide the original 500 1,250 image into two images and assign pixel-wise annotations each image.
Fine-tuning medical dataset We cannot find publicly available 2D medical AD datasets that in- clude both category labels and segmentation ground truths simultaneously. To fill the blank, this paper, create such medical dataset by combining two existing 2D medical datasets. Particularly, we use the colon polyp detection dataset ColonDB (Tajbakhsh et al. 2015) to provide pixel-level an- notations. Meanwhile, considering the normal samples in the same domain, of Endo classification dataset (Hicks et al., 2021) to combine with ColonDB. As result, the new medical dataset contains 163 normal samples and 380 anomaly samples, supporting both anomaly classification and segmentation tasks.
choose the test split
15

---

Published as a conference paper at ICLR 2024
C DETAILED ANALYSIS OF DPAM
Since the visual encoder of CLIP is originally pre-trained to align global object semantics, such as cat and dog, the contrastive loss used in CLIP makes the visual encoder produce a representative global embedding for recognizing semantic classes. Through the self-attention mechanism, the attention map in the visual encoder focuses on the specific tokens highlighted within the red rectangle in Fig. Bb Although these tokens may contribute to global object recognition, they disrupt the local visual semantics, which directly hinders the effective learning of the fine- tine-grained abnormality in our object-agnostic text prompts. For segmentation purposes, it's crucial for the visual feature map to emphasize the surrounding context to capture more local visual semantics.
Formally, let aij be attention score in the attention score matrix, where i, E [1, w]. then the i-th output of Q-K K attention can be written as:
ęK VD
a VDJ
Attention(Q, softmar
V
0zj
aij
Note that vectors (i.c., qi kis 0;) are represented as row vectors, Attention( (Q, K, V); can be re- garded the weighted average of vj using dij the weight. Assuming that the original attention map focuses on the specific tokens at index m, it is clear that qi only produces the large attention score with km in all ky. Therefore, aim is the largest score among other Oıy so Attention( Q, K, V); is dominated by Um. which causes the local visual embedding at index i to be disturbed by the lo- cal visual embedding at index m. In Figure 3(b), the attention score map presents vertical activa- tion and suggests that every O: produces large attention score with km. In such a case, several Attention (Q, K, V) is dominated by Um and results in weak anomaly segmentation in Figure 3(b) even though Um may be important for original class recognition. Some prior studies (Rao et al.) 2022 Gu et al. )023) use an additional decoder to recover the local visual semantics. In this paper, directly use local visual embeddings for segmentation and point out that an ideal attention map for local visual semantics should exhibit a more pronounced diagonal pattern. For this purpose, DPAM is proposed to replace the original Q-K attention with analogous components, including Q-Q, K-K, and V-V self-attention. Therefore, dg is changed into:
vang a = VD,
oj oij= = vD
0ij =
This modification ensures that qi ky and hold significant weight in forming Atlention(4) Q. V)., Atlention(K, K K, V)i and V thereby preserving local visual semantics. As result, the produced attention maps exhibit more diagonal prominence compared to the original Q-K attention, leading improved performance tation, as shown in Fig 3c Fig and Fig. 3e However, since Q and K consist of the original attention map, other important tokens index n for class recognition within themselves may also produce relatively large scores (a;,)) (e.g., 9s has strong relevance with on besides q:) to disturb Allention (Q, Q, V), and Allention K, K, V): Fig 3c and Fig 3d In contrast to Q-Q and K-K, V-V does not participate in computing the original attention map, reducing the unexpected bias to different tokens in V for the purpose of anomaly segmentation. Therefore, V does not produce large weight (a:y) with ey and generates larger weight (a,) to form Attention (V, V, V)., preserving more information of in and experiencing diagonally prominent attention map (minimal disturbance), depicted in Fig. 3e This is the reason why V-V achieves the best results.
anomaly segmen-
D ADDITIONAL RESULTS AND ABLATIONS
Module ablation by removing modules. We dive into the effectiveness of each module in AnomalyCLIP in Table 7 We test the contribution of one module by removing one module and maintaining the rest module.
1. The effectiveness of DPAM (Ti). When we remove DPAM, the results show decrease from 91.1% AUROC to 87.9% AUROC in pixel-level performance and from 91.5% AU- ROC to 80.7% AUROC in image-level performance. This performance decline indicates
16

---

Published as a conference paper at ICLR 2024
Table Study on the effect of each module. Ti: DPAM, Th: Object-agnostic prompt learning. T3: Textual prompt tuning, and T4: Integration of multi-scale local visual feature.
MVIce
Image-lever
Pixcl-Jovel
Image
(91.),
[91.5 96.23
(95.5, (95.5,87.0) 87.0)
(82.1, 83.4)
(87.9. 80.0)
(80.7. 89.5)
(91.9. 84.9)
(73.0. 77.7)
(81.3,
089.5.
(9066 80.5)
(90.4. 95.3)
(94.5, 85.6)
(81.6. 85.1)
(90.0) 81.1)
(91.0, 96.1)
095.2. 86.0)
(81.9, 85.2)
Table 8: The study of computation overhead among baselines.
Methuds
per opoch
Triining time
FPS
Pisel-lovel
Perfommunce (VisA)
CLIP
13:25
(40.6 14.8)
(66-4, 71.55
WisCLIP
1.20
(79.6, 56.8)
[78.1. 81.2)
VAND
13.56
9.23
(94.2. 86.8)
(78.0, 81.4)
CoOp
12:75
(24.2. 3.8)
(62.8,
AnomlyCLIP
13:71 min
8.92
(95.5, 87.00
[82.1, 85.4)
Anemaly without DPAM
12.98 min
10.21
(91.9,84.9)
(73.0, 77.7)
the importance of DPAM, which enhances local visual semantics by modifying the atten- tion mechanism. However, the decrease in performance at the image level is more pro- nounced than that at the pixel level. This discrepancy is attributed the fact that the total loss places greater emphasis on local context optimization, driven by a larger local loss compared to the case with DPAM.
2. The effectiveness of object-agnostic prompt learning (T2). Excluding object-agnostic prompt learning makes AnomalyCLIP suffer from the huge performance gap (i.e., 91.1% AUROC to 84.3% AUROC in pixel-level and 91.5% AUROC to 65.6% AUROC in image- level). This performance decline illustrates that the object-agnostic text prompt template plays significant role in improving the performance of AnomalyCLIP at both pixel and image levels.
3. The effectiveness of textual prompt tuning (T'). When removing textual prompt tuning, the performance of AnomalyCLIP declines from 91.1% AUROC to 90.6% AUROC in pixel- level performance and from 91.5% AUROC to 90.4% AUROC in image-level performance. This demonstrates the importance of adapting original textual space by adding learnable textual tokens in the text encoder.
4. The effectiveness of the integration of multi-layer local visual semantics (Ti). When re- moving multi-layer local visual semantics, the outcomes reveal decrease from 91.1% AUROC to 90.0% AUROC in pixel-level performance and from 91.5% AUROC to 91.0% AUROC in image-level performance. This performance decline indicates the importance of incorporating multi-layer local visual semantics.
Study of computation overhead In addition to performance, computation overhead is also an important metric to evaluate the model. Therefore, we assess the time taken during training (train- ing time per epoch) and the inference speed (frames per second, FPS). For a fair comparison, all experiments are conducted in a single 3090 NVIDIA RTX 3090 24GB GPU, and the GPU is kept free before evaluation. In Table g AnomalyCLIP takes 13.71 min per epoch on MVTec AD (The total number of samples is 1725) and only requires a total of 15 epochs for the whole fine-tuning. Once AnomalyCLIP finishes fine-tuning, AnomalyCLIP can be applied to different datasets and do- mains without additional training. We also compare AnomalyCLiP with other baselines that need auxiliary data (i.e., CoOp and VAND). The minimum training time per epoch is 12.25 min of CoOp, and hence the training time taken is similar for fine-tuning methods. As for inference speed, CLIP achieves the 13.23 FPS. However, it suffers from weak detection performance. Although WinCLIP achieves better performance, WinCLIP has only 1.2 FPS because it needs multiple forward image patches to derive the segmentation. AnomalyCLIP outperforms WinCLIP and obtains 8.92 FPS.
We also evaluated the computation overhead of DPAM separately. In Table 8, without DPAM, AnomalyCLin takes 12.98 min to train per epoch. Compared to the 13.71 min for Anomaly with DPAM, we observe that introducing DPAM does not significantly increase the time complexity.
17

---

Published as a conference paper at ICLR 2024
(a) Successful case of de-(b) Failure case detect-(c) Successful case of de-(d) Failure case of detect- ing color stain
tecting scratch-like defects.ing scratch-like defects. tecting color stain.
Figure 7: Analysis for successful and failure cases.
Table 9: Study on the effect of shared and unshared learnable word embeddings.
MV AD
VisA
Pisck-level
Image-Jevel
Pixel-evel
Imago-tevel
Shared
(901)
(909, 95.2
(93.0 56.4)
(1.4
Unshazed
0111 81.4)
PL5,96.3)
(95,5,
(8.41.
This is attributed to the fact that DPAM only creates the two paths during the computation of the attention map and is frozen during fine-tuning, thereby avoiding the computationally expensive pro- cess of gradient computation. Meanwhile, DPAM also does not result in large computation overhead during inference: AnomalyCL.IP w/ DPAM gets 8.92 FPS vs. 10.21 FPS for w/o using DPAM.
Discussion of successful and failure case Although AnomalyCLIP achieves superior results in ZSAD, we find that there still exist some potential failures. In Fig. 7a| AnomalyCLIP accurately detects scratch-like patterns on the product, even when they typically appear in the texture. How- ever, false detection occurs when scratch-like patterns are situated in the background, as depicted in Fig. 7b Mcanwhile, we also show the color stain pattern. As shown Fig. 7c AnomalyCLIP suc- cessfully detects the color stain, which exhibits subtle visual differences from the detected entities. However, AnomalyCL.P may face challenges when the normal region displays patterns that are distinguishable to the naked eye from anomalies. For instance, in skin cancer detection, the normal regions falsely detected as anomalies are actually visually similar to the disease region in Fig. [7c] Also, the stain interference in the background is also problem. These failure cases illustrate the importance of mitigating background interference and achieving fine-grained discrimination, espe- cially in cases of visually similar abnormalities. Exploring these challenges for enhancing ZSAD is a valuable direction for future research.
Study on the effect of shared and unshared learnable word embeddings As presented in Ta- ble 9 when sharing the learnable word embeddings of 9n. and Ja. AnomalyCLIP achieves 90.5% AUROC pixel level and 90.9% in image-level on MVTee AD and 95.0% AUROC in pixel level and 81.5% AUROC in image level on VisA. The results show that AnomalyCL.IP without sharing also works well for ZSAD and the efficiency of our object-agnostic prompt learning. However, the shared prompt performs slightly worse than the unshared prompts (used in the original paper). The performance decrease is AUROU and AUROC in image level on MVTec AD and Visa, and 4.5%AUROC and AURO in pixel level. We believe that the separate learning for these two prompts helps discriminate the generic normality and abnormality because when we share the parameters of V; and W,, the learned semantics of normal and anomaly may be confused.
Study on the effect of local visual features Here, we examine the impact of various types of local visual semantics. We explore two ensemble methods, namely AnomalyCLIP Pe..eenmeeeel and CainaalyLIPeesemb..r
involving the ensemble of Q-Q, K-K, and V-V and the ensemble of Q- K, Q-Q. K-K, and V-V, respectively. In addition to Q-K and V-V features, we average the logit output of different features for the ensemble. As shown in Table I0 AnomalyCLIP CoomalyCLIeeneemttee 1eeembbel shows performance improvement by leveraging the advantages of three DPAM features. How- ever, while AnomalyCI.IP Cn.malyy.PPossembitee ensemble2 outperforms the Q-K feature version, it experiences perfor-
18

---

Published as a conference paper at ICLR 2024
Table 10: Study on the effect of local visual features.
Module
Pixei-level
MV AD
Image- level
Plael-Jevel
VisA
Image- level
(87.9,
(80.7.
(75.0,
V.
(91.1.8144)
(91.5, 96.22 2)
(95.5. 0)
(82.1,85 85.4)
Q-Q. K-K, V-V (ensemble)
(91.2,833
(91.4 96. 1)
(95.8, 87 6)
(82.3,85,7)
Q-K, Q-Q K-K, V.(essemlle)
(90.7, 80.5)
(9088 96.0)
(94.9, 66
(80.7, 83.7)
mance decrease compared to V-V from 91.1% to 90.7%AUROC and 91.5%ARROC to 90.8%AUROC on MVTec AD. There is also a decline from 95.5%AUROC to 94.9%AUROC and 82.1% AUROCC to 80.7%AUROC on VisA. The decline in performance upon adding Q-K features to AnomalyCLIP ensemmes suggests that the Q-K feature fails to provide valid local visual semantics to facilitate ZSAD. Note that the original CLIP exploits Q-K features and gets the weak segmentation performance. The seemingly good pixel-level performance of Q-K in AnomalyCLIP is attributed local optimization, where the object-agnostic prompt helps alleviate the disrupted local visual semantics of Q-K.
Table 11: Ablation on the effect of the Focal and Dice loss.
MVTo: AD
VisA
Focal loss
Dice loss
Pisel-lovel
Image-level
Pixci-level
Imsae--evel
(80.3,
(89.9, 95.4)
(80.0 78.1)
[82.2, 94.99
(87.2, 78.0)
(89.5, 95.2)
(90.1 79.8)
(81.8, 85.21
(96.6, T8.1)
(91.0, 96.0)
(94.9, 86.13
(81.2, 84.6)
(9111,834)
(91,5, 96.2)
(95.5, 87.00
(82.1, 85.4)
Focal and Dice loss ablation Focal and Dice loss play a crucial role in optimizing local context. They are introduced to empower object-agnostic text prompts to focus on fine-grained, local ab- normal regions from intermediate layers of the visual encoder. As mentioned in Section 3.2, Focal loss addresses the imbalance between anomaly and normal pixels, typically caused by the smaller size of anomalous regions. Mcanwhile, Dice loss aims to precisely constrain the anomaly boundary by measuring the overlap between the predicted segmentation (S,,/S,) and the ground truth mask. To provide a more comprehensive analysis, we have included an ablation study on Focal and Dice loss in Table [ Compared to scenarios without local context optimization, Dice loss improves the pixel-level and image-level performance from AUROC to 87.2% AUROC and AUROCAAUOO to 90.1 % AUROC in pixel level on MV'Tc AD and VisA. Focal loss also brings the performance gain of 10.3%AROOC and 8.3 AURO. Combining Focal and Dice loss, AnomalyCLIP achieves the best results (i.e., 91. AUROOC and 95.5 AURO). Note that the global context optimization is always used during the ablation, since we need at least one loss function to drive the optimization.
Table 12: Comparison of 7.SD performance between AnomalyCL.P and SOTA full-shot methods. The best performance is highlighted in red, and the second- best is highlighted in blue.
Task
Calegory
Dañasets
[C]
AncmulyCLIP
PatchCore
RD4AD
&testtne
MVTec AD
(91.5, 96.2)
99.7)
ViLA
(82.1, 85
(94 95.9)
(95.3,
Image level (AUROC, AP)
Obj
MPDD
(77.0, 82.0)
(91.1, 96.3)
(91.6, 93.99
BTAD
(88.3, K7
(03.2. 9%
(93.8, 96 8)
90.00
Texture
DAGM
10
(97.5, 92.3)
(92.7. 81.3)
(92.9, 79.
Uby dooxture
MVTec AD
15
191. 81.4)
(95.1, 42.8)
(97.8, 53.00
VisA
(95.5,
92.2)
Pixel-level (AUROC,, PRO)
Obj
MPDD
196.5, 88.71
94.99
(98.4,
BTAD
(94.2. 74.81
(97 4, 74.4)
(97.5 5, 75 1
SDD
(90.6, 67
(87.9) 46.3)
(92.2, 72.0)
Texture
DAGM
87.99)
(96.8,
Comparison with SOTA full-shot methods In this section, we are interested in the performance gap between AnomalyCLIP and the recently published SOTA full-shot methods, such as Patch- Core (Roth et al. 2022) and RD4AD (Deng & Li (022). Since some datasets do not provide normal training data, conduct experiments on six public datasets. As shown in Table I2 AnomalyCLIP achieves comparable anomaly detection and segmentation performance compared to PatchCore and RD4AD, and it even outperforms them in some datasets. This illustrates that the generic prompt
19

---

Published as a conference paper at ICLR 2024
100
100
85.0
MVlec
95
NVlec VISA
MV/eec
MVTTec
82.5
90
82.5
80.0
85
60.0
85
16
12
on the length of learnable text prompt
Ablation depth text prompt
(a)
(b)
95
95
85.0
MVToc
MVTbc
90
MVTac
VIsA
VisA
MVTec
VisA
60.0
85
Visi
Ablation on the longth of learnable token embodding
Ablation on used layer of visial encodor N
(c)
(d)
Figure 8: Hyparameter analysis. (a)L' ablation. (b) M ablation. (c)L ablation (d)N ablation. Pixel/image-lvel (AUPRO, AP) performances are shown on the left and right sides each subplot, respectively.
embeddings empower AnomalyCLIIP to effectively capture the normality and abnormality so that AnomalyCLIP can surpass the performance boundary decided by the training data.
Refinement of the textual space A representative embedding is not only decided by the well- designed text prompt, also depends on the appropriate textual space. During fine-tuning, randomly initialized learnable token embeddings are introduced in the text encoder to refine the textual space for the adaption to AD. To control the degree of refining the textual space, we choose to insert the learnable token embeddings into the text encoder from its bottom to the top layer. In particular, the trainable and original tokens are denoted as t',, and fm respectively, where m represents the layer of the text encoder. To integrate the original textual representations, for the layer m. concatenate th, and bm along the dimension of the channel and then forward them into Tm, to get I'mt and 4m+1- Due to the self-attention mechanism, the output of 4m+11 contains the information of Imm In order to provide adequate calibration, we discard the obtained r' and initialize new learnable token embeddings t Through this operation, t'm+1 further refines textual representations of the layer m + 1. We repeat this operation until we reach the designated layer M'. This procedure is given by:
[m--1: Im+1] = Tm (I'm (1'mmstng (m]) 1 mm42 = Tm+ 1 (fmo41 (mm+l)))
(3)
tM (tMr),
where the operator [ represents the concatenation along the channel.
Hyparameter analysis We study the length of learnable text prompts E, depth of learnable token embeddings M, length of learnable token embeddings M, and number of used layers in visual encoder N. As shown in Fig. 8b we observe that the detection and segmentation performance initially improves with increase in the value of E. However, within the range of lengths from 12 to 16, we notice a decline in performance, which suggests that excessively long learnable text prompts could involve redundant information. Therefore, an appropriate value for E, such as beneficial to accurate learning of object-agnostic text prompts. Besides, we also investigate the depth of the attached learnable token embeddings Fig. 8b The degree of refining of the initial text space becomes more pronounced as the depth increases, enabling more discriminative textual embeddings for normal and anomaly. However, the performance drops when the refinement is excessive and impairs the generalization of AnomlayCLIP, seen in the case when M equals 9. After selecting the depth, we proceed to investigate the influence of the length of learnable token embeddings. As illustrated in Fig. Sc. we find that the length of token embeddings also involves similar tradeoff between the model generalization and calibration of textual space in Fig. &d AnomalyCLIP achieves the overall performance gain when we provide the most local visual semantics (N 4).
12, is
20

---

Published as a conference paper at ICLR 2024
Table 13: Ablation on the robustness of the abnormality-related token in our prompt template on industrial defect datasets.
Category Otg &textune
Datesets
maomalons
flawed
defective
blemished
MVlle
(91.5, 96.2)
(91.4, 96.2)
(91.3] 96.2)
(91.4, 96.2)
(91.5.96.2)
VisA
(22.1.8.44 85.4)
(30.7. $4.5)
(80.7. 81.5)
(80.9. 81.61
(80.7.44) 84.5)
Image-tevel (AUROC, AP)
MPDD
(77.082.0)
(78.0, 83.9)
(779, 83.6)
(77.8. 83.5
(78.6, 84.1)
87.4)
SDD
(81.7 80,0)
(82.3, 76.3)
(82.6. 76.8)
(82.8. 71.2)
(82.7. (82.7.77.0)
Texture
DTD Synthetic
DAGM
(97.5, 92.5)
97.00
(91.7, 92.60
(93.2)
(97.5, 92.4)
(97.5, 92.3)
(93.4,
(97.5, 92.4)
(93.5,9%)
Ciy &texture
MVTec
01.4,
(91.0, 81.4)
(90.7,81) 81.4)
(91.0, 61.7)
(90.9.81.2)
VisA
(95.5 87.0)
(95.5, 86.5)
(95.5. 86.5)
(95.5, (6-2)
(95.6, 86.5)
Pisel-ievel (AUROC, PRO)
MPDD
(96.5, 88.7)
(96.6, 58.
(96.7, 89.0)
(96.7, 89.2)
(96.6 88.8)
(96.3.3377)
67.8)
(89.5, 66.
(89.5, 64-8)
(89.6,
Texture
DTD Symiheiiiic
DAGM
(95.6. 91.0)
(97.9. 92.3)
(97.9. 92.3)
(95.6, 91.2)
(95.6. 91.3)
(97.9, 92.1)
(97.9, 92.5)
(95.5, 90.9)
(95.6. 90.9)
(97.9 92.2)
Image-level MUROC AUROC
Reei-level AUROC
AUPRO
An AnamalyCLPP.
AnomalyCa AnomalyCLPPr
AnomstyCUPP AnomatyCUPP
Anoma YCLIP AnomalyCLIP
MVRec AD
VIA
MVTe: AD
VBA
MV/e: AD
ViA
MVTec AD
VISA
Figure 9: Object ablation.
Prompt template ablation Here, study the robustness of AnomalyCLIP prior anomaly semantics in the object-agnostic text prompt template. We replace damaged in the object-agnostic text prompt with other words having similar anomaly semantics, such anomal nous, flawed, defecting ve. blemisiie shed. The results are presented in Table and Table [4 The steady results indicate that AnomalyCLIP not sensitive the prior anomaly semantics introduced by the object- agnostic text prompt template.
Object ablation To investigate what the object-agnostic text prompts have learned, we replace ob ject in object-agnostic text prompts with specific target [cls], resulting in AnomalyCLIP In Fig. 9 AnomalyCLIP,, still performs well in ZSAD, even as we block out the object seman- ties during fine-tuning. This suggests that the knowledge learned by object-agnostic text prompts is the underlying anomaly patterns, allowing them to provide discriminative textual embeddings even when specific object semantics incorporated. Furthermore, compared AnomalyCLip AnomalyCLIP,e shows a performance decay, which can be attributed to the inclusion of redun- dant/noisy object semantics. These results once again demonstrate the generalization ability of object-agnostic prompt learning.
Table 14: Ablation on the robustness of the abnormality-related token in our prompt template on medical image datasets.
Tisk
Category
Datisets
damaged
anomaloos
fawed
defective
Nemishel
HeadCT
(93.1,
91.5)
Image-Aevel (AUROC, AP)
Brain
BrainMRI
(90.3, 92.2)
(87.8) 90.4)
(87.7. 90.0)
(88.3, 90.5)
(88.6. 90.7)
B-35H
(94.6, 94.7)
(93.1. 93.0)
(92.9. 92.8)
(93.1.93.0)
(93.2. 93.1)
COVID-19
99.0)
ISIC
(89.7.75)
(90.1.
(90.1. 30.1)
(50.4, 81.0)
(902. 30.6)
CVC ColonDU
(81.9. 71.3)
(82.2. 71.51
(82.3, 71.6)
(82.1. 71.1)
(82.2, 71.5)
Pisel-level (AURUC, PRO)
Colon
CVC ClimiDDG
(82.9. 67.88
(83.0.
(83.1. 68.4)
(82.9, 67.9)
(83.1. 68.2)
Kyasur
(79.4,
Endo
(84.1. 63.6)
(84.3, 63.5)
(84.2. 63.5)
(84.2, 62.91
(81.3, 63.4)
Thyroid
TN3K
(81.5,50.4)
(81.5,
(61.3, 50.9)
(81.3, 50.3)
(1,16.5.11)
21

---

Published as a conference paper at ICLR 2024
E VISUALIZATION
Similarity score between textual and visual embeddings. We present visualizations of the sim- ilarity scores generated by both CLIP and AnomalyCLIP. These visualizations aim to provide an intuitive illustration of the effective adaptation made by AnomalyCLIP in comparison to CLIP. As shown in Fig. I0 and and Fig. I1 we present the similarity score of CLIP on MVTec AD and VisA. The normal and anomaly scores are severely overlapped. Further, the range of scores is centered at 0.5. These show that the textual and visual space CLIP originally aligned for object semantics are desired for ASAD. Also, we visualize the similarity scores of AnomalyCI.I in Fig. 12 and Fig. [3] Compared to CLIP, there is significant overlap between the scores assigned to normal and anomaly instances, and at the same time, the score range is considerably wider. These results indicate that AnomalyCLIP achieves a significant improvement adapting CLIP to ZSAD.
Anomaly map for different datasets. In addition to the similarity sification, we also visualize the anomaly score maps to present the strong anomaly segmentation ability of AnomalyCLIP. Specifically, we visualize the industrial object class: hazelnut, pill, and screw from MVTec AD; candle, chewinggum, capsule, cashew, peb, and pip fryum from Visa; bracket, metal plate, and tube from MPDD. We also visualize the industrial texture: grid, leather, carpet, tile, wood, and zipper. In addition, we visualize the segmentation in medical domain across photography, endoscopy, and radiology images: skin cancer detection from ISIC; thyroid nodule detection from TN3K; colon polyp detection from Kvasir; brain tumor detection from Br35H.
for anomaly clas-
carpet
bottle
hazelnut
leather
cable
Norma
Normal
Normal
Normal
Normal
Anomaly
Anomaly
Anomaly
Anomaly
Anomaly
1.00.0
1.00.0
capsule
grid
pill
transistor
metal _ut
Nomal Anomaly
Normal Anomaly
Normal Anomaly
Normal Anomaly
Normal Anomaly
0.0
0.5 screw
1.00.0
0.5 toothbrush
1.00.0
0.5 zipper
1.00.0
0.5 tile
1.0.00
0.5 wood
1,0
Nomal Aremaly
Norml Anomalyy
Normal Anomaly
Nernal Anomaly
Normal Anomaly
0.0
0.5
1.00.0
0.5
1.00.0
0.5
1.00.0
0.5
1.00.0
0.5
1.0
Figure 10: Similarity scores CLIP on MVTec AD. Each sub-figure represents the visualization of one object.
FINE-GRAINED ZS AD PERFORMANCE
In this section, we present the fine-grained data subset-level ZSAD performance details.
22

---

Published as a conference paper at ICLR 2024
Table 15: Fine-grained data-subset- wise performance comparison (AUROC) for anomaly segmen- tation on MVTec AD.
Object name
CLIP
CLIP-AC
WinCLIP
VAND
CoOp
AnomalyCLIP
Carper
11.5
95.4
1884
Hazelnut
25.2
34.0
94.3
96.1
30.2
97.1
Leather
9.9
5.6
96.7
99.1
11.7
98.6
Cable
37.4
72.3
Capsule
92.0
Grid
8.7
11.9
82.2
95.8
7.8
97.3
Pill
55.8
60.5
803
76.2
46.5
92
Tranistor
Scee
90.1
76.4
89.5
97.8
17.0
97.5
Tochhroch
36.3
35.0
86.9
95.8
64.9
91.9
Zapper
77.6
92.7
Weed
45.7
42.4
93.4
95.8
31.4
96.5
Mean
38.4
36.2
85.1
87.5
33.3
Table 16: Fine-graina data-subset-wise performance comparison (PRO) for anomaly segmentation on MVTec AD.
Object
Colp
AnomalyCLiP
Carpet
2.9
19
64.1
48.5
0.5
90.1
Boule
1.4
76.4
45.6
4.5
80.9
Leather
0.2
91.1
72.4
188
Cible
7.3
6.9
42.9
25.7
12.2
64.4
Capsule
13.2
51.3
5.7
57.2
6.0
63,0
65.4
3.2
88.2
Transistor
15.3
11.2
43.4
21.3
9.3
$8.1
Metal.ao
2.9
10.3
31.8
38.
7.0
71.0
Tootbrush
5.8
67.7
$8.5
16.6
83.5
Zipper
17.7
15.2
71.7
10.7
11.6
65.3
10.1
Mean
11.3
11.6
64.5
44.0
6.7
B14
Table 17: Fine-grained data-subset-wise performance comparison (AUROC) for anomaly classifica- tion on MVTec AD.
Object
CLIP
CLIP-AC
WinCLIP
VAND
CoCp
AnomalyCLIP
Carpet
180,0
99.5
Bottee
45.9
46.
99.2
92.0
87.7
89
Hazelnot
88.7
91.1
93.9
89.6
93.5
97.2
Leather
99.4
100.0
88.4
Capsule
71.4
68.8
72.9
79.9
81.1
89.9
Grid
72,5
63.7
98.8
86.3
94.7
97.0
Metal nut
62.8
63.4
97.1
68.4
85.3
93.6
Screw
78.2
66.7
83.3
84.9
88.9
$1.1
Toothhruch
Zipper
36.1
91.5
89.6
Tile
88.5
89.0
100.0
99.9
99.7
1000
Wood
94
99.4
99.0
97.7
96.8
23

---

Published as a conference paper at ICLR 2024
Table 18: Fine- cene-grained satta-subset-wisee performance comparison (AP) on for anomaly classification
MVTèc AD.
Objeet name
CIJIP
WinCTIP
VAND
CoOp
AnomalyCTJJP
Carpet
98.8
97.8
[00,]
99.5
100.0
100.0
Bontle
78.9
96.4
Leather
99.8
99.8
100.0
99.9
100.0
99.9
Cable
70.8
64.3
91.2
93.1
09.4
$1.4
Capsule
92.1
91.5
95.5
95.7
Pill
93.4
93.6
95.7
96.0
94.2
95.4
Transistor
48.1
29.9
87.1
77.5
90.2
90.6
Metal
96.3
93.5
Toothrrsh
90.7
96.0
95.5
71.5
90.4
93.7
Zipper
87.4
73.9
97.5
97.1
99.7
99.6
100.0
1000
98.3
99.7
Micm
87.6
85.4
96.5
93.5
94.8
96.2
Table 19: Fine-grained data-subsett-wse performance comparison (AUROC) for anomaly segmen- tation on VisA.
Object name
CLJIP
CLIP-AC
WinCLIP
VAND
CoOp
AnomalyCTIIP
Candle
33.6
50.0
88.9
97.8
16.3
98.8
56.8
97.5
Chowinggum
43.0
56.5
93.3
99.5
3.4
99.3
Fryum
45.6
62.7
88.5
92.0
21.7
94.6
Macaronil
20.3
98.8
36.8
Macoron12
97.8
27.5
Pcbl
57.8
51.6
61.2
92.7
19.8
94.1
Pch2
34.7
38.4
71.6
89.7
22,9
92.4
88.4
18.0
94.6
Pipe Fyym
58.7
44.7
75.4
96.0
29.2
98.2
Mean
16.6
47.3
79.5
98.2
21.2
953
Table 20: Fine-grained data-subset-wise performance comparison (PRO) for anomaly segmentation on VisA.
Object name
CLIP
CLIP-AC
WinCLIP
VAND
CoOp
AnomalyCLIP
Candie
Caponles
15.8
35.3
86.7
Coshew
9.6
10.9
76.4
91.7
1.7
91.6
Chowinggum
17.8
302
7044
87.3
0.1
91.2
93.2
18.1
Mocaroni2
20.9
1844
21A4
82.3
2.7
84.2
Pchl
11.7
12.5
26.3
87.5
0.1
$1.7
77.8
0.0
Pebs
17.1
20.3
80.4
86.8
0.0
91.3
Fiyu.
16.7
82.3
90.9
0.6
96.8
Table 21: Fine-grained data-subset-wise performance comparissn (AUROC) for anomaly classifica- tion on VisA.
Object
CLIP
CLIP-AC
WinCLIP
VAND
AnomalyCLIP
Candle
37.9
95.4
83.8
16.2
Cupoales
69.7
75.3
85.0
61.2
77.2
$1.5
Coshew
69.1
72.7
92.1
87.3
75.7
76,
Chewinggum
Fryum
67.2
80.3
94.3
80.0
Macimonil
64.4
67.4
76.2
71.6
$3.6
K7.2
Macamoni2
65
65.7
63.7
66.5
66.5
73.4
Peb2
62.6
51.2
71.8
44.6
62.2
Peb3
$2.2
49.0
73,4
66.8
54.4
62.7
Pchs
87.7
8000
79.6
95.0
66.0
93.9
Pipe.fryum
Mean
66.4
65.0
78.1
78.0
62.
82.1
24

---

Published as a conference paper at ICLR 2024
Table 22: Fine- sine-grained data-subsct- performance comparison (AP) for anomaly classification on VisA.
Object name
CLIP
WiRCLIP
VAND
Col3p
AntomalC[iIP
Candie
42.9
50.3
95.5
86.9
52.9
51.1
Capoules
$1.0
81.3
90.9
74.3
85.3
88.7
Chewinggum
90.4
90.2
98.5
98.4
93.1
98.9
Fryum
82.0
76.6
90.1
97.2
90.2
96.8
Macaronil
56.8
58.7
75.8
70.9
52.3
86.0
Pebl
56.9
48.4
78.4
57.2
36,0
87.0
Pch2
63.22
59.5
49.2
73.8
47.3
64.3
Pcb3
53.0
76.5
70.7
54.8
70.0
Plpc.fryum
94.6
95.7
82.5
94.8
89.7
96.3
Moan
71.5
70.1
81.2
81.4
68.1
85.4
Table 23: Fine-grained data-subset-wise performance comparison (AUROC) for anomaly segmen- tation on MPDD.
Objeet name
CLIP
CLIP-AC
WinCLIP
VAND
Colp
AnomalyCLIP
Brackethisc
85.3
86.4
57.8
96.3
9.3
95.7
writk
83.5
77.4
79.5
99.8
Connector
56.5
$2.9
79.0
90.6
7.6
97.2
MetaLplate
64.3
52.5
93.8
99.1
Mean
62.1
38.7
76.4
94.1
13.4
90.3
Table Fine-grained data-subset-wiss performance comparison (PRO) for anomaly segmentation on MPDD.
Object
CLIP-AC
AnomalyCLip
Beacket blnck
62.6
58.9
45
89.7
1.5
K5.2
Bracket hrown
2.8
4.0
25.0
70.3
0.4
77.7
Bracket white
41.6
Mesal plare
31.5
27.0
78.2
74.5
0.2
86.9
Tubes
30.4
22.9
44.7
96.9
11.5
93.6
Table 25: Fine-graines data-subset-win performance comparison (AUROC) for anomaly classifica- tion on MPDD.
Object name
CLIP
CLIPPAC
WinCLIIP
VAND
CoOp
AnomulyCI.
Brackett.boon
50.9
57.9
48.5
64.00
43.9
62.22
Bracket white
45.4
42.6
40.2
79.5
43.9
64.9
Metal.latee
Tubes
87.3
72.8
78.7
95.9
83.4
95.5
Mean
31.3
56.
63.5
73.0
35.1
77.0
Table 26: Fine-grained data-subset-wise performance comparison (AP) for anomaly classification on MPDD.
Objeet name
CLIP
CLIP-AC
WinCLIP
VAND
CoOp
AsoomalCCLiP
Hiack
25.6
11.7
30.0
white
51.2
47.3
45.1
57.5
68.5
Connector
62.2
61.4
61.3
71.8
26.5
76.8
MeaaLplate
78.5
97.5
78.3
92.0
91.7
Moan
65.4
66.0
69.9
80.2
64.2
83.0
25

---

Published as a conference paper at ICLR 2024
candle
capsules
cashew
chewinggum
Norma Anomaly
Norma Anomayy
Nermal Anomalyy
Nermal Anomaly
0.00 0.25 0.50 0.75 1.000.00
0.25 0.50 0.75 1.000.00
0.25 0.50 0.75 1.00
0.25
0.50 pcb1
0.75
1.00
fryum
macaroni1
macaroni2
Normal Anomaly
Nermal
Nerma
Nermal
0.00 0.25
0.75 pcb2
0.50
1.000.00
0.25
0.50 0.75 1.00
0.25
0.50
0.75
1.00
0.00
0.25 0.50 0.75 1.00
pcb3
pcb4
pipe fryum
Normal
Nermal
Nermal
Normal Aremaly
0.00
0.25
0.50
0.75
1.000.00
0.25 0.50
0.75 1.000.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50 0.75 1.00
Figure 1I: Similarity scores of CLIP on VisA. Each sub-figure represents the visualization of one object.
carpet
bottle
hazelnut
leather
cable
Normal Anomaly
Anomaly
1.0
capsule
grid
pill
transistor
metal _ut
Normal Anomaly
Normal Anomaly
Normal Anomally
Normal Anomaly
Nermal Acomaly
0.0
0.5
0.0
0.5
1.000.
0.5 zipper
1.0
0.5 tile
1.0
0.5 wood
1.0
screw
toothbrush
Anomaly
Anomaly
Anomaly
Anomaly
Anomaly
0.0
0.5
1.0000
0.5
1.0.0
0.5
1.0 0.0
0.5
0.0
0.5
1.0
Figure 12: Similarity scores of AnomalyCLIP on MVTec AD. Each bub-figure represents the visu- alization of one object.
26

---

Published as a conference paper at ICLR 2024
candle
capsules
cashew
chewinggum
Normal
Normal
Normal Anomaly
Normal Anomaly
0.0
0.2
0.4
0.6
0.8
1.00.0
0.2
0.4
0.6
0.8 1.00.0
0.2
0.4
0.6
0.8
1.00.0
0.2
0.4
0.6
0.8
1.0
fryum
macaroni1
macaroni2
pcb1
Normal Ancemaly
Normal Anomaly
Normal Anomalyy
Ancomalyy
0.2
0.8
1.00.0
0.4
1.00.0
0.2
0.8
pcb2
pcb3
pcb4
pipe_fryun fryum
Normal Anomaaly
Normal Anomaly
Normal Anomally
Nommal Anomaly
0.0
0.2
0.4
0.6
0.8
1.00.0
0.2
0.4
0.6
0.8
1.00.0
0.2
0.4
0.6
0.8
1.00.0
0.2
0.4
0.6
0.8
1.0
Figure 13: Similarity scores of AnomalyCLIP on VisA. Each sub-figure represents the visualization of one object.
Figure 14: Anomaly score maps for the data subset, hazelnut, in MVTec AD. The first row rep- resents input, and we circle the anomaly regions in the second row. The last row presents the segmentation results from AnomalyCI.IP.
Figure 15: Anomaly score maps for the data subset, pill, in MVTec AD. The first represents the input, and we circle the anomaly regions in the second row. The last row presents the segmentation results from AnomalyCLIP.
27

---

Published as a conference paper at ICLR 2024
Figure 16: Anomaly score maps for the data subset, metal nut, in MVTec AD. The first row rep- resents the input, and we circle the anomaly regions in the second row. The last row presents the segmentation results from AnomalyCLIP.
Figure 17: Anomaly score maps for the data subset, capsule, in MVTec AD. The first row represents the input, and we circle the anomaly regions in the second row. The last row presents the segmenta- tion results from AnomalyCLIP.
Figure 18: Anomaly score maps for the data subset, screw, in MV'ee AD. The first row represents the input, and we circle the anomaly regions in the second row. The last row presents the segmenta- tion results from AnomalyCLIP.
Figure 19: Anomaly score maps for the data subset candle. The first row represents the input, and we circle the anomaly regions the second row. The last row presents the segmentation results from AnomalyCLIP.
28

---

Published as a conference paper at ICLR 2024
Figure 20: Anomaly score maps for the data subset chewinggum. The first row represents the input, and we circle the anomaly regions the second row. The last row presents the segmentation results from AnomalyCLIP.
Figure 21: Anomaly score maps for the data subset capusle. The first row represents the input, and we circle the anomaly regions the second row. The last row presents the segmentation results from AnomalyCLIP.
Figure 22: Anomaly score maps for the data subset cashew. The first row represents the input, and we circle the anomaly regions the second row. The last row presents the segmentation results from AnomalyCLIP.
Figure 23: Anomaly score maps for the data subset peb. The first row represents the input, and we circle the anomaly regions in the second row. last row presents the segmentation results from AnomalyCLIP.
29

---

Published as a conference paper at ICLR 2024
Figure 24: Anomaly score maps for the data subset pip fryum. The first row represents the input, and we circle the anomaly regions second row. The last row presents the segmentation results from AnomalyCLIP.
Figure 25: Similarity scores for the data subset bracket. The first row represents the input, and we circle the anomaly regions in the second row. The last row presents the segmentation results from AnomalyCLIP.
Figure 26: Anomaly score maps for the data subset metal plate. The first row represents the input, and we circle the anomaly regions the second row. The last row presents the segmentation results from AnomalyLLIP.
4
47
Figure 27: Anomaly score maps for the data subset tube. The first row represents the input, and we circle the anomaly regions in the second row. The last row presents the segmentation results from AnomalyCLIP.
30

---

Published as a conference paper at ICLR 2024
Figure 28: Anomaly score maps for the data subset grid. The first row represents the input, and circle anomaly regions in the second row. The last row presents the segmentation results from AnomalyCLIP.
Figure 29: Anomaly score maps for the data subset leather. The first row represents the input, and we circle the anomaly regions the second row. The last row presents the segmentation results from AnomalyCLIP.
Figure 30: Anomaly score maps for the data subset carpet. The first row represents the input, and we circle the anomaly regions in the second row. The last row presents the segmentation results from AnomalyCLIP.
Figure 31: Anomaly score maps for the data subset tile. The first row represents the input, and we circle the anomaly regions in the second row. last row presents the segmentation results from AnomalyCLIP.
31

---

Published as a conference paper at ICLR 2024
Figure 32: Anomaly score maps for the data subset wood. The first row represents the input, and we circle the anomaly regions in the second row. The last row presents the segmentation results from AnomalyCLIP.
Figure 33: Anomaly score maps for the data subset zipper. The first row represents the input, and we circle the anomaly regions in the second row. The last row presents the segmentation results from AnomalyCI.IP.
Figure 34: Similarity scores for the data subset skin.
Figure 35: Anomaly score maps for the data subset thyroid. The first row represents the input, and we circle the anomaly regions the second row. The last row presents the segmentation results from AnomalyCLIP.
32

---

Published as a conference paper at ICLR 2024
Figure 36: Anomaly score maps for the data subset colon. The first row represents the input, and we circle the anomaly regions in the second row. The last row presents the segmentation results from AnomalyCLIP.
Figure 37: Anomaly score maps for the data subset brain. The first row represents the input, and we circle the anomaly regions in the second row. The last row presents the segmentation results from AnomalyCI.II.
33


Reviews:
[{"review_id": "wQfLtPFbAh", "reviewer": "Reviewer_NCKk", "summary": "This paper propose a zero-shot anomaly detection (ZSAD) approach using CLIP, named AnomalyCLIP. Different from the previous ZSAD methods, AnomalyCLIP attempts to learn object-agnostic text prompts of normality and abnormality to segment abnormal part. To reach this, the authors leverages textual prompt tunning and DPAM technologies. Extensive evaluations conducted on 17 publicly datasets demonstrates the effectiveness of the proposed approach.", "questions": "It is important to discuss why the \"damaged [cls]\" is suitable for more diverse anomaly detection scenarios.\n\nExplanations on the effectiveness of DPAM and computation overhead are necessary.", "limitations": "", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The empirical evaluations are extensive covering 17 diverse benchmarks.\n\n2. The proposed approach attempts to learn class-agnostic prompts which seems to contradict the pretrained CLIP that is mostly used to classify semantic objects. The authors successfully mitigate this issue by several technical modules, e.g. object-agnostic text prompt design and DPAM layer.", "weaknesses": "1. The prompt template might be too restrictive. \"damaged [cls]\" may not well represent all types of anomalies. For example, if a component is missing or applying the method to other domains than defect identification the proposed prompt template may not work well.\n\n2. According to Figure 2, the pipeline needs to feed the same images into two visual encoders, this would introduce additional computation overhead.\n\n3. The proposed DPAM layer lacks a theoretical basis. It is not clear why replacing Q-K attention map with V-V attention map without updating any visual encoder's parameters into CLIP would improve the performance.\n\n4. The paper has not discussed why the proposed approach is able to learning class-agnostic feature from the pretrained semantic CLIP as the CLIP is trained to be sensitive to semantic classes.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "B2vrJZslkM", "reviewer": "Reviewer_xREe", "summary": "In the paper, the authors address Zero-shot anomaly detection (ZSAD) where training data is unavailable. They introduce AnomalyCLIP, which improves CLIP's generalization for ZSAD by using object-agnostic prompt learning. This allows for generic recognition of normality and abnormality across diverse images. AnomalyCLIP also incorporates both global and local anomaly contexts for optimization. Tested on 17 datasets, AnomalyCLIP demonstrates superior ZSAD performance.", "questions": "See the weaknesses.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper introducing AnomalyCLIP for Zero-shot anomaly detection (ZSAD) stands out for its originality, exemplified by its novel approach of using object-agnostic text prompts for anomaly detection, a creative departure from traditional methods. The quality of the work is underscored by its robust methodology and extensive validation across 17 diverse datasets. The authors effectively communicate their ideas with clarity, making the complex concepts accessible. Significantly, AnomalyCLIP's ability to generalize across various domains without needing prompt redesign, particularly in industrial and medical settings, marks it as a notable advancement in the field of anomaly detection.", "weaknesses": "A weakness in the paper is the unexplained initial use of the term \"glocal.\" Clarifying this key term when first mentioned would improve understanding and clarity.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ra8HO7eYIk", "reviewer": "Reviewer_w1Ea", "summary": "The paper proposes a new method named AnomalyCLIP which deal with the Zero-shot anomaly detection problem. It leverages the CLIP prompt learning techniques to capture the normal/abnormal information within an image regardless of its foreground objects. The experiment results show the effectiveness of their method.", "questions": "See Weakness section.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1.\tLeveraging pre-trained model like CLIP to address anomaly detection is a good direction and an interesting topic.\n2.\tThe paper shows comprehensive and detailed experiments and results which outperforms other baselines.", "weaknesses": "1.\tThe idea and some techniques, like glocal context optimization, are similar to a concurrent work[1]. The author may compare with highly related works.\n[1] Gu, Zhaopeng, et al. \"AnomalyGPT: Detecting Industrial Anomalies using Large Vision-Language Models.\" arXiv preprint arXiv:2308.15366 (2023).\n2.\tThe DPAM strategy is confusing. The author claims that the Q-Q, K-K, V-V self-attention suffers from different issues, and V-V self-attention derives the best result. However, these three variants are very similar in the DPAM mechanism. It is better to explain it in the paper and show the results of the original Q-K attention.\n3.\tI doubt the contribution of some modules. The module ablation results indicate that the T2 object-agnostic text prompts module contributes the most, similar to CoOp method. However, the author employed a different prompt for the CoOp experiment. For instance, the text prompt template for normality is defined as [V1][V2]...[VN][normal][cls], while that for abnormality is [V1][V2]...[VN][anomalous][cls]. This variation in prompt settings doesn't allow for a fair comparison.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Ln4gSCZ16b", "reviewer": "Reviewer_GzuZ", "summary": "The paper proposes a zero-shot anomaly detection/segmentation method utilizing CLIP and prompt learning. The authors observe that CLIP-like VLMs learn class semantic features of the object and pay less attention to normality/abnormality, e.g., associated with fine-grained local features. They assume that the normality/abnormality features can be object-agnostic. For adapting the pretrained CLIP to focus on normality and abnormality features, they proposed to learn two text prompts for each respectively. To learn these generic prompts, they utilize both global optimization, i.e., maximizing similarity between image embeddings and text embeddings, and local optimization, i.e., maximizing similarity between image-patch embeddings and text embeddings. They also show that adding more learnable tokens to the text encoder and image encoder of CLIP can improve results, and using V-V attention can help in refining the local features. The paper provides a comprehensive empirical evaluation of 17 real-world datasets showcasing the effectiveness and advantages of the proposed method over existing zero-shot anomaly detection baselines.", "questions": "1. Why is learning two text prompts necessary? Do you have comparisons to $g_n = [V_1][V_2]...[V_E][object]$ and $g_a = [V_1][V_2]...[V_E][damaged][object]$?\n1. How is the performance of the variant with all components except DPAM?", "limitations": "", "rating": 6, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. The paper brings the principle of prompt learning for CLIP-like VLMs in zero-shot anomaly detection. \n2. The proposed method achieves remarkable zero-shot anomaly detection performance on both industrial anomaly detection datasets and medical datasets.\n3. The paper provides a large body of ablation studies for each involved design component individually.", "weaknesses": "1.  My largest worry is the paper relies on an optimistic assumption of abnormality, namely, the abnormality is object-agnostic.  The assumption might hold in some cases. The learned model is able to detect the abnormality that has similar abnormal patterns to the features in the auxiliary training set, e.g., detecting the defects of a manufactured part. However, the assumption can also fail in some cases. For example, the model may misclassify a product with texture patterns similar to scratches as an abnormal one, even though the scratch-like textures are normal texture patterns of the product.  It would be better to explicitly discuss the successful cases and the potential failures of the proposed method.\n\n2. Minor comment: $S_n$ and $S_a$ are first-time used in $L_{local}$ on page 5, but are clearly defined on page 6. Having the definitions and equations of $S_n$ and $S_a$ before $L_{local}$ would be good.\n\n3. The discussion of some related work about zero-shot anomaly detection, e.g., [1,2,3], is missing.\n\n[1] Philipp Liznerski, Lukas Ruff, Robert A Vandermeulen, Billy Joe Franks, Klaus Robert Muller, and Marius Kloft. Exposing outlier exposure: What can be learned from few, one, and zero outlier images. Transactions on Machine Learning Research, 2022.\n\n[2] Sepideh Esmaeilpour, Bing Liu, Eric Robertson, and Lei Shu. Zero-shot out-of-distribution detection based on the pretrained model clip. In Proceedings of the AAAI conference on artificial intelligence, 2022\n\n[3] Aodong Li, Chen Qiu, Marius Kloft, Padhraic Smyth, Maja Rudolph, Stephan Mandt. Zero-Shot Batch-Level Anomaly Detection. Thirty-seventh Conference on Neural Information Processing Systems, 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "YXXoHHc3su", "reviewer": "Reviewer_vcXy", "summary": "This paper targets ZSAD leveraging visual language pretrained models. To learn object-agnostic information, this paper proposes to use the well-known text prompt technique, allowing the model to focus on the abnormal image regions rather than the object semantics. Experiments are conducted on several anomaly detection benchmarks.", "questions": "See the weakness.", "limitations": "", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.\tThe contribution is clear, with well-organized paper architecture.\n2.\tThe motivation sounds reasonable.\n3.\tExperiments show the effectiveness of the proposed method.", "weaknesses": "1. The motivation behind DPAM appears to be relatively weak. It is not entirely clear why employing V-V self-attention would preserve more localized information compared to the original Q-K attention. Relying solely on a single instance shown in Figure 3 may not furnish sufficient evidence to support this assertion. Furthermore, the rationale for treating these features separately remains unclear. What might be the outcome if we were to combine Q-Q, K-K, V-V, and Q-K features into an ensemble?\n\n2. The ablation studies lack clarity regarding the specific details of each component, making it challenging to assess the actual contributions of the proposed method:\n(i) The experiments in Table 4 involve the incremental addition of modules (T1-T4), but a more informative approach would be to examine the performance when each of them is removed. For instance, it is unclear how DPAM (T1) performs when used in conjunction with both T2, T3, and T4. So an importment setting should be removing T1 but keeping using T2-T4. Additionally, the increase in AUC from 47.9 to 54.8 by introducing T1 does not signify a significant improvement (both values are close to random guessing) for a two-class classification problem.\n(ii) In Table 5, it remains unclear how the models are optimized when both the global and local losses are absent.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "rnuWP8LMjM", "reviewer": "Reviewer_5wSC", "summary": "This paper proposes a zero-shot anomaly detection framework using CLIP and object-agnostic text prompts with learnable prompt tokens and a modified attention mechanism.", "questions": "Besides all of the other modifications such as DPAM, why are object-agnostic prompts better than object-aware prompts for a specific object class?\n\nIs it true that hidden layer visual embeddings are only used for the segmentation map, while the final output embedding is only used for the image-level score?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "The paper is well written and easy to follow overall\n\nExperiments are comprehensive, with many datasets and evaluation metrics, as well as the ablation study\n\nVLMs for anomaly detection is relatively under-explored and prompt learning is especially new amongst AD literature.", "weaknesses": "Little/no discussion about the additional cost of training\n\nNo comparison of different ViT backbones of CLIP\n\nFigure 6 feels confusing and unnecessary\n\nIt would be better to provide more intuition why the Focal and Dice loss are necessary\n\nFigure 2 caption should be more explanatory", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)