MuSc: Zero-Shot Industrial Anomaly Classification and Segmentation with Mutual Scoring of the Unlabeled Images
Authors: Xurui Li
Ziming Huang
Feng Xue
Yu Zhou
Venue: ICLR 2024 poster (2024)
Keywords: zero-shot anomaly detection;  Industrial Informatics；
Primary area: general machine learning (i.e., none of the above)

Abstract: This paper studies zero-shot anomaly classification (AC) and segmentation (AS) in industrial vision.
We reveal that the abundant normal and abnormal cues implicit in unlabeled test images can be exploited for anomaly determination, which is ignored by prior methods.
Our key observation is that for the industrial product images, the normal image patches could find a relatively large number of similar patches in other unlabeled images,
while the abnormal ones only have a few similar patches.
We leverage such a discriminative characteristic to design a novel zero-shot AC/AS method by Mutual Scoring (MuSc) of the unlabeled images,  
which does not need any training or prompts.
Specifically, we perform Local Neighborhood Aggregation with Multiple Degrees (LNAMD) to obtain the patch features that are capable of representing anomalies in varying sizes.
Then we propose the Mutual Scoring Mechanism (MSM) to leverage the unlabeled test images to assign the anomaly score to each other. 
Furthermore, we present an optimization approach named Re-scoring with Constrained Image-level Neighborhood (RsCIN) for image-level anomaly classification to suppress the false positives caused by noises in normal images.
The superior performance on the challenging MVTec AD and VisA datasets demonstrates the effectiveness of our approach. 
Compared with the state-of-the-art zero-shot approaches, 
MuSc achieves a $\textbf{21.1}$% PRO absolute gain (from 72.7\% to 93.8\%) on MVTec AD, a $\textbf{19.4}$% pixel-AP gain and a $\textbf{14.7}$% pixel-AUROC gain on VisA.
In addition, our zero-shot approach outperforms most of the few-shot approaches and is comparable to some one-class methods.
Code is available at https://github.com/xrli-U/MuSc.

Published as a conference paper at ICLR 2024
MUSc: ZERO-SHOT INDUSTRIAL ANOMALY CLASSI- FICATION AND SEGMENTATION WITH MUTUAL SCOR- ING OF THE UNLABELED IMAGES
Xurui Li Ziming Huang"., Feng Xue", Yu Zhou
School of Electronic Information and Communications, Huazhong University of Science and Technology
2 Artificial Intelligence Research Institute, Wuhan JingCe Electronic Group Co.,LTD
3 Department of Information Engineering and Computer Science, University of Trento
[xr1i plus, zmhuang, yuzhou juzhou)ehust. @hust edu. en, feng. xue@unitn.
ABSTRACT
This paper studies zero-shot anomaly classification (AC) and segmentation (AS) in industrial vision. We reveal that the abundant normal and abnormal cues im- plicit in unlabeled test images can be exploited for anomaly determination, which is ignored by prior methods. Our key observation is that for the industrial product images, the normal image patches could find a relatively large number of similar patches in other unlabeled images, while the abnormal ones only have a few sim- ilar patches. We leverage such a discriminative characteristic to design a novel zero-shot AC/AS method by Mutual Scoring (MuSe) of the unlabeled images, which does not need any training or prompts. Specifically, we perform Local Neighborhood Aggregation with Multiple Degrees (LNAMD) to obtain the patch features that are capable of representing anomalies in varying sizes. Then we propose the Mutual Scoring Mechanism (MSM) to leverage the unlabeled test images assign the anomaly score to each other. Furthermore, we present an optimization approach named Re-scoring with Constrained Image-level Neigh- borhood (RsCIN) for image-level anomaly classification to suppress the false pos- itives caused by noises in normal images. The superior performance on the chal- lenging MVTec AD and VisA datasets demonstrates the effectiveness of our ap- proach. Compared with the state-of-the-arr zero-shot approaches, MuSc achieves 21.1 % PRO absolute gain (from 72.7% to 93.8% on MVTec AD, 19.4% % pixel-AP gain and a 14.7% piXel-AUROC gain on VisA. In addition, our zero- shot approach outperforms most of the few-shot approaches and is comparable some one-class methods. Code is available at https://iihub.com/rrii U/MuSe
INTRODUCTION
Industrial anomaly classification (AC) and anomaly segmentation (AS) are fundamental tasks in computer vision with diverse applications. Real-world anomalies may appear on a wide range of objects and textures. The high diversity of anomalies makes the AC/AS task challenging.
Many excellent unsupervised AC/AS approaches, e.g., (Zavrtanik et al. 2022 Zhang et al. 2023 Zavrtanik et al. 2021 Liu et 2023), have been proposed. They typically rely on the whole normal dataset for training. thus called as full-shot methods. In contrast, RegAD (Huang et al., 2022 and GraphCore (Xie et al., (203) leverage a few normal images only, known few-shot AC/AS setting, and also achieve promising accuracy. All the methods above can be categorized as one-class AC/AS approaches, and share a similar pipeline as shown in Fig. Iīa.. Recently, WinCLIP (Jeong et al., 2023) and APRIL-GAN (Chen et al.] 2023) propose zero-shot AC/AS, which breaks new ground by using text prompts for anomaly measurement, as shown in Fig. [b).
In these existing approaches, the test images are compared to the additionally introduced informa- tion, such as normal training images or text prompts. However, rich amount of normal information
Contributed Equally, Corresponding Authors.

---

Published as a conference paper at ICLR 2024
Classifiation
Closs ficetion
Chestfiattio
optimizaton
Un.abeled Best image
Festurs Campurisu
Patchslevel snumaly
token
tàs tnkens
GT
Image enovler
Image enosker
Text encucker
Image exeoder
Test Set
Narual Set.
Set
Text Pronnt
Teat Sei
One-olass AC and AS based
(b) Zero-shol AC aS
Normal image
Memory Bank
based on CLIP
(c) (ours)
Figure 1: (a) One-class AC and AS hased on a memory bank, which requires many normal reference
images. (b) Zero-shot AC and AS based CLIP, which relies additional text prompts. (c) Our
MuSc only leverages the unlabeled test images from patch-level and image-level.
implicit in the unlabeled test images is rarely used. (According to statistics, the normal pixels ac- counted for 97.26% in the test images of MVTec AD dataset and 99.45% VisA datasel). Besides, due to the randomness and unpredictability of the anomalies, the abnormal regions may be dissimi- lar from each other, even though they belong to the same anomaly type. Therefore, both normal and abnormal cues in unlabeled test images can be exploited for anomaly determination.
Based on these observations, we reveal that the anomaly can detected by comparing with other unlabeled test images, and hence we design novel zero-shot AC/AS approach, named MuSc, which does not need any normal training images or prompts. As shown in Fig. (c), approach consists of patch-level representation extractor, patch-level anomaly scoring ap- proach, and an image-level optimization. First, we perform Local Neighborhood Aggregation with Multiple Degrees (LNAMD) the patch token express each image patch, which is capable of modeling anomalies in varying sizes and is suitable for AC and AS tasks. Next, we introduce Mu- tual Scoring Mechanism (MSM), which leverages the unlabeled test images assign the anomaly scores to each other. Such simple mechanism generates a high-quality anomaly score for each image region. can be credited the observation that the normal image patches could find large number of similar patches in other unlabeled images while the abnormal ones only have few sim- ilar patches. Furthermore, we explore the relationship between image-level features and propose the Re-scoring with Constrained Image-level Neighborhood (RsCIN) optimize the anomaly clas- sification. Extensive experimental evaluations on MVTec AD and VisA datasets demonstrate the effectiveness of approach. Compared with the state-of-the-as zero-shot AC/AS approaches, achieve +6.0% AUROC classification metrics on MVTec AD and +14.7% % increase on VisA. For anomaly segmentation, we obtain +21.1 % PRO and +21.9% % AP gains MVTee AD and +19.4% AP gains on VisA. We further emphasize that our zero-shot approach outperforms most existing few-shot methods in 4-shot case and is comparable several full-shot methods.
In summary, the major contributions our work are:
To the best of our knowledge, we propose the first method that only uses the unlabeled test images for industrial anomaly classification and segmentation.
We reveal the potential capability of normal and abnormal patches contained in unlabeled images. It inspires
propose brand new AC/AS method based mutual scoring of unlabeled images, which is new mechanism for the community.
Our approach significantly surpasses the existing zero-shot AC and AS approaches. In addition, approach outperforms most of the few-shot approaches and is comparable to some full-shot methods.
RELATED WORKS
Vision transformer. Vision transformer (ViT) (Dasovitskiy et al. 2020) is widely used in multi- modal models and large vision models such CLIP (Radford al. 2021 and DINO (Caron al.
2

---

Published as a conference paper at ICLR 2024
Festure extraction and uptimization
LNAMD
VIT
LNAMD
LNAMD
CLS Ioken
Test set Drest
Anomaly
Anomaly Sepmentation
MSM
MSM
MSM
MAX
RaCTN
Add
Image-tevel fecturre
lateh-level feature
Aggregated patch with different
Histogram of anomuly scores for AC
MuSc
Mutual Scoring Mechanism
Figure 2:
Our MuSc architecture. It consists of three parts: feature extraction and optimization
(Section 3.1). MSM to obtain anomaly segmentation results (Section 3.2 and RsCIN to optimize classification results (Section 3.3).
These models produce high-quality patch-level and image-level features. We find that the patch-level features have the limitation detecting industrial anomalies of varying sizes. Swin transformer (Liu al. 202T proposes the varied-size window attention calculate attention inside local windows of varying sizes, which can detect diverse size objects. We propose a simple method LNAMD to optimize the patch-level features that it can detect varying size anomalies.
2021
Industrial anomaly classification and segmentation. In the industrial vision field, previous works explore one-class AC/AS, such PatchCore (Roth al. 2022), PaDiM (Defard al. 202 l) and AST (Rudolph et al. (2023). which require lot of normal reference images. Recently, some zero- shot methods have worked well. Most of them rely on text prompts for the image-tex. text aligmment such as WinCLIP (Jeong et al. (023). Variance Vigilance Vanguard (Baugh et al. (023) and SAA (Cao et al. 2023). the few-shot setting, RegAD (Huang al. 2022) and GraphCore (Xie et al,, 2023) focus on data augmentation to estimate the normal feature distribution by few reference normal images. These methods ignore the rich information the unlabeled images. In the medical image analysis field, some methods like DDAD (Cai al., 2023) and SRR (Yoon et al. (021) integrate the unlabeled images into the normal image set for model training, which also needs to select set of normal images. In the zero-shot AC/AS field, the method (Aota al. (203) explores
the relationship between the patches inside one test texture image. ACR (Li et al. 2023) proposes adaptation strategy without human involvement and outperforms other methods. We propose the first zero-shot AC/AS method that only uses the unlabeled images.
Manifold learning. In high-dimensional manifolds, only local space maintains the nature of Eu- clidean space, so it is inaccurate to calculate the distance directly in high-dimensional manifolds. The idea of manifold learning methods (Zhou al., 2003 Wang & 2012) is construct new embedding space where the distance metric matches the manifold structure. We find that the image-level features satisfy the conditions for higher dimensional manifolds, and hence we design the RsCIN based on manifold learning to optimize the pixel-level anomaly classification results.
METHOD
As illustrated in Fig. our approach is directly designed on the unlabeled test images. Firstly, we extract the ViT feature for each unlabeled image and perform Local Neighborhood Aggregation with Multiple Degrees (LNAMD) patch tokens to represent abnormal regions with varying sizes, detailed in Section 3.1 Secondly, we propose a Mutual Scoring Mechanism (MSM) in Section 3.2 in which the unlabeled images assign the anomaly scores each other, resulting in lexel-level anomaly classification and segmentation. Thirdly, we leverage Re-scoring with Constrained Image- level Neighborhood (RsCIN) to optimize the pixel-level anomaly classification in Section 3.3
3

---

Published as a conference paper at ICLR 2024
add
1x1
add
3x3
5x5
RGB image
Ground truth
3.3.5
Figure 3: The visualization of anomaly segmentation results with different aggregation degrees r.
3.1 LOCAL NEIGHBORHOOD AGGREGATION WITH MULTIPLE DEGREES
Given the unlabeled images D,, { N.. We extract the ViT feature (Dosovitskiy et al. 2020) of 4, To increase the abnormal modeling ability with varying sizes, we perform local neighborhood aggregation with multiple degrees on the patch tokens of ViT. Specifically, we define the patch tokens F; E IR MCC of 4, where M is the patch number, then reshape F; to M VM C. For each patch token, we follow (Rohh et al. 2022) use an adaptive average pooling operation in its x r neighhorhood to obtain the aggregated patch token FT. E RVMM //MMx/MXX with different aggregation degree E {1,3,5), shown the left of Fig. 3 and reshape it back to M x C. Therefore each aggregated patch token is denoted as Pa. E glxC. where m E [1, M),. we divide the layers of ViT into L stages and E {1. 2, .... L} indicates the stage I of VIT. We will integrate multiple aggregation degrees and stages of ViT in the next subsection.
Compared with using one patch size, the patch token with multiple aggregation degrees is suitable for representing anomalies with varying sizes, leading to high-quality anomaly scores even with simple distance measurement. As the industrial images illustrated in Fig. When 1, we do not perform LNAMD. The defect has higher anomaly score in (a), which indicates that smaller is suitable for detecting small anomalies, while in (c), the large defect has several local false positives and false negatives, indicated by the black dotted box. since the smaller regions cannot fully express the entire defect. When 5, the false positives and false negatives (c), demonstrating that the aggregated feature is suited to detect the larger defects, while in (a), the small defect is smoothed and has lower score. Therefore, using all aggregated patch tokens with different degrees achieves a good balance in detecting the anomaly with a wide range of sizes, which the common case in real industrial scenes, as shown in the right side of Fig.
solved in
3.2 MUTUAL SCORING MECHANISM OF THE UNLABELED IMAGES
In this subsection, propose the mutual scoring mechanism to obtain the high-quality patch-level anomaly score by only using the unlabeled images in Du which consists of three steps. In the first step, leverage each image in (T) assign anomaly score for each aggregated patch token of a test image I; in stage as follows,
m.r 0 (1₂) = min IP am.
.0.7 P
(1)
where is the aggregation degree and (Iz) indicates that image I E (Da is employed for scoring. If the patch token mm." of I is similar to any patch token and 1y, the im- age 1, assigns a small anomaly score to mm. Then we define the scoring vector A. m.r m.
m. (41), a, m.y (12), a, (IN-1)) of each patch for each / and according to {Da M4). The statistical histogram of A of all the normal patches in D, given in Fig. 4(a), which indicates that most of the normal patches could find some similar patches in D.,. While (b) gives the histogram
4

---

Published as a conference paper at ICLR 2024
164
1c4
35
normal abnormal overlap
normal
10-
abmormal
Quantity
25-
uverlap
20-
0.0
6.2
0.4
0.6
1.0
0.0
0.2
0.4
06
0.8
1.0
anomaly scores of all normal putches
(b) anomaly scores of all abnormal patches
20 1e2
20 1w2
Quantity
10-
10
0.8 1.0
(0) whole value interval average
(d) the minimum valuu interval average
Figure 4: Top: The histograms of A,, of all normal patches (a) and abnormal patches (b). Bottom:
The histograms of ", of normal patches and abnormal patches with the whole value interval (c) and the minimum X% value interval (d) average. The blue and orange parts indicate the anomaly scores of normal patches and abnormal patches respectively under r = 3 and = 2 setting.
of abnormal patches, which shows that most abnormal patches only have a few similar patches in D.. indicating in the left corner of Fig. (b), and they are dissimilar most of the normal and other abnormal patches in (b).
Based on the above analysis, we claim that most of the unlabeled images in D, could assign higher score for the anomaly patch and lower score for the normal ones. Hence, directly performing an average operation
A, is capable of distinguishing the abnormal patches from the normal patches, shown in Fig. [fc.. For the overlap in (c), we notice that it is caused by some normal patches with appearance variations different normal images, which are assigned higher scores by the dissimilar patches. To reduce these dissimilar patches in scoring, the second step of MSM is to perform Interval Average (IA) operation the minimum X% of !.
1
Σ
ai (I&)
KE[L,K]
where I is the images in the minimum X% value interval and is the number of these images. Such design uses small number of images score each image patch, and suppresses the afore- mentioned issue effectively. As shown in Fig. 4d. the anomaly score distribution of normal and abnormal patches has smaller overlap in the score interval of [0.2, 0.4] compared with (c).
Aiming to achieve the fine-grained anomaly score for each image patch, the third step of MSM performs Eq. and Eq. on each combination of stage and aggregation degree r, respectively, and m."" to achieve the patch-level anomaly score as follows,
compute the average of all (.
1
Σ
a
m
IE(I....L.) (1(1.3.5)
The patch-level anomaly score vector of is defined A; |a ..., 2 Then reshape the original resolution of the input image as
A, Mx1 VM VM and upsample approaches (Roth et al., 2022). we extract the the anomaly segmentation result. Following existing maximum score 6; max(a!... ..., a of image I; as its pixel-level anomaly classification score and the AC score vector of the images in Da denoted C [c1, ... cv] See Appendix A. for more details.
3.3 CLASSIFICATION RE-SCORING WITH CONSTRAINED IMAGE-LEVER NEIGHBORHHOOD
We observe that the pixel-level AC score in C is sensitive to the local noises, e.g., in Fig. 5 i), normal image that contains several small noises has a higher AC score e 0.129, while the pure normal image (ii) has a lower score c = 0.079, when we calculate the class token distance between images (i) and (ii), they almost the same with similarity equal to 0.987, which reveals that the
5

---

Published as a conference paper at ICLR 2024
normal abnormal overlap
normal abnormal
Quantity
overlap
0.0
02
04
06
0.8
1.0
0.0
0.2
0.4
06 with RsCIN
0.8
(a) without RsCIN
Normal
0.987
0.989
Abnormal
iii
iv
0.129
0,079
0.277
0,054
Histogram of anomaly classification scores of unlabeled test images before (a) and after (b) using RsCIN, with blue representing normal images and orange representing abnormal images. Bottom: A normal and abnormal example of RsCIN.
Figure 5: Top:
class token distance is less sensitive to local noises, and two images that have a higher class token similarity should have an approximate AC score. A similar situation be observed in an abnormal image (iv), where the image contains low contrast abnormal area and has lower AC score in (iv), while has a higher similar the abnormal image (iii).
Motivated by such an observation, we propose the re-scoring with constrained image-level neighbor- hood to optimize the anomaly classification by exploiting the relationship between the image-level features of images in D,. Specifically, given the class token F, E R xC of the last layer of VIT, define an bllge-weighter graph G (V, W), the vertexes V denotes the images D.. and the edge weight W is defined as the similarity of the class tokens F; of I;, i.e. W.j = F; Fj. where means dot product operation. Therefore, the manifold learning approaches, e.g., SSO (Jiang et al.) 2011). be employed for optimizing C by utilizing the token similarity W.
However, due to the high smoothness C, it may be influenced by a large number of images, as the experimental illustration in Sec. 4.2 excessive utilization of the images in D, may reduce the AC accuracy. Therefore, we propose a Multi-window Mask Operation (MMO) to constrain the image number, which makes each image only influenced by small number of neighborhood images. To be specific, we design a binary window mask matrix Mj E N as,
Me(i,j) = 1, if I (
(4)
O, otherwise,
Nr indicates the k-nearest neighbor. We use multiple to form the multi window mask set M (Mey.., MEg J. where K is the number of window masks. Hence the AC score updated as,
Č= Σ (D (D' @ W)C) + C)/(K +1)
M.EM
where Ć Nx1 is the optimized AC score vector, is diagonal matrix with Dii, Σjn Mi.W(i,j) and @ means element-wise multiplication. Eq. 5 utilizes the weighted AC scores of multiple neighbor images of I; to refine C. Therefore I, is normal only if itself and its nearest neighbor images all have small AC score. As shown in Fig. [ the AC scores of normal and abnormal images in C have large overlap in (a), while (b), the overlap is reduced in Č. See Appendix 1.2 for more details. Meanwhile, we experimentally prove that the proposed image- level AC optimization method can be further employed improve the AC accuracy of existing approaches in Appendix A.2.4
4 EXPERIMENTS
Implementation Details. In this paper, we use ViT-L/14-336 pre-trained by OpenAl (Radford) al. 2021) backbone. contains 24 layers, which are divided into stages. Each stage is

---

Published as a conference paper at ICLR 2024
Table 1: Quantitative comparisons on the MVTee AD and VisA datasets. We compare MuSc with some state-of-the-art zero-shot and few-shot methods. Bold indicates the best performance, while underlined denotes the second-best result. All metrics in %.
Dataset Method
Fl Fl-mx-cls
AP-cls
AUROC-seam maxx-segm AP-segm PRO-segm
WinCLIP [Jeong et al.
2023
0-shot
91.8
92.9
96.5
85.1
31.7
64.6
MVTèc AD APRIL-GAN (Chen
2023
O-shor
86.
40.8
ACR (Li 2023
0-shot
85,8
91.3
92.9
92.5
44.2
38,9
72.7
MuSe (ours)
0-shot 97.8(+6.0) 97.5(+4.6)
99.1(+2.6)
97.3(+4.8)
62.6( 18.4)
WinCLIP (Jeon et al.
2023
0-shot
78.1
79.0
81.2
79.6
14.8
56.8
APRII-AAN (Chen
MuSc (ours)
0-shor
98.8+44.6
48.8(+16.5)
45.1(+19.4)
92.7(+5.9)
RegAD [[ang al.
88.0
PatchCore Roth of 2022
4-shor
88. 8..8:12.6
92.6:::.6
94.5=115 1.5
94.3::0.55
$5.0:1199
84.3 1.6
WinCLIP 1oog 0f 20055
4-shor
95.2++11.2
04.74-0.
97.3=006
96.2400.3
89,5+
89.0:20.8
MVTec AD APRII-GAN (Chien al. 2023
4-shot
92.8:10.2
92.8:00.1
96.320.1
95.9:10.00
56.911.11
54.51002
91.010.11
GrahhCore Xic or 2023
4-shot 92.9
97.4
MuSc (ours)
PatchCore Roth et al.
2022
4-shot 85.3:42211
84.3::1.3
87.5=2211
96.8-100.3
43.944..1
84.91:144
WinCLIP Jeong
4-shot 87.3.11 1.8
84.2.1
88.8:1
97.2.10.2
3.00.13.0
87,6
APRIL-GAN (Chnn al
2023
4-shot
92.6+-04
88 4:5.
94,5-0.3
96.2:0000
40.0:20.4
32.2:4:011
90.2:10.1
MuSc (ours)
92.7
Image
GT
APRIL- GAN(O)
APRIL-
Ours(0)
Image
GT
APRIL- GAN(O)
APRIL- GAN(4)
Ours(0)
bottle
pcb1
metalnut
cashew
pipe fryum
pil
MD
(b) VisA
Figure 6: Visualization of anomaly segmentation results on MV'Te AD and VisA benchmarks.
composed layers. We extract the patch tokens output from each stage for our method and use the linearly projected class token of the last layer for classification optimization. All the unlabeled test images are scaled resolution of 518x518 and then fed into the backbone. the MSM module, we use the minimum 30% anomaly scores. In the RsCIN module, we use the multi-window setting with 2, 3 on MVTec AD and 8, 9 on VisA.
Datasets. To study Industrial anomaly classification and segmentation performance, we conduct experiments on industrial image datasets MVTec AD (Bergmann et al.) 2019) and VisA (Zou et al.) 2022). MVTee AD has high-resolution (from 7002 to 1024²) RGB images of 10 object categories and 5 texture categories. VisA comprises high-resolution (0000x1500 RGB images covering 12 objects in 3 domains. Both benchmarks contain normal and anomaly images in the test images.
Evaluation Metrics. For classification metrics, we report metrics: the Area Under Receiver Operator Characteristic curve (AUROC), Average Precision (AP), and F1-score at optimal threshold (F1-max). For segmentation metrics, we report 4 metrics: pixel-wise AUROC, pixel-wise F1-max, pixel-wise AP, and Per-Region Overlap (PRO).
Baselines. We compare our method with some coite-of-thhearr zero/few-shot approaches, e.g. Win- CLIP (Jeong al. 2023), APRIL-GAN (Chen et al. (202)), RegAD (Huang al. 2022). PatchCore (Roth et al. 2022) GraphCore (Xie et al. 2023) and ACR Li et al. (202)). Among these methods. place for the Zero-shot Track of the CVPR 2023 VAND Workshop Challenge, which uses an additional labeled auxiliary dataset for training. RegAD and ACR use other categories of the dataset when training. The unmeasured metries in these papers are reproduced by using the official implementation code. In addition, we also compare our method with some full-shot methods.
APRIL-GAN
4.1 QUANTITATIVE AND QUALITATIVE RESULTS
Comparison with zero/few-shot methods. In Tab. [ we compare MuSe with some state-of-the art zero-shot methods MVTee AD and VisA datasets. On MVTeec AD, MuSe achieves 21.1
7

---

Published as a conference paper at ICLR 2024
Table 2: Comparison with some existing many- shot methods in image-AUROC and pixel- AUROC (%) on MVTec AD.
Table 3: Ablation study of LNAMD with differ- ent aggregation degrees r Amage-AUROe and pixel-AUROC (%) on MVTec AD and VisA.
Methods
Setting
AC
AS
MVTec
VisA
CutPaste al. 2021
full-shot
95.2
88.3
AS
NSA [Schimee et a.a 2022
full-shot
97.2
96.3
(1)
96.8
94.6
93.0
97.9
IGD
(Chen et
full-shot
93.4
93.0
[3]
97.3
97.1
89.5
98.4
PatchCore (Koth et ml 2022
full-shot
99.6
98.2
(5)
94.5
97.6
82.2
97.6
RegAD
[Hhang el 2122
32-shot
94.6
96.9
(1.3)
98.1
96.3
93.7
98.6
GraphCore
8-shot
95.9
(3.5)
96.5
97.6
87.2
98.1
MuSc(ours)
0-shot
97.8
97.3
(1,3,5
97.3
98.7
Table 4: Ablation study of MSM with different sam- ple strategies in image-AU AUROC and pixel-AUROC (%) MVTec AD and VisA.
Table 5: Ablation study of RsCIN. We re- port anomaly classification performance in AUROC, Fl-max and AP on MVTec AD and VisA dataset. All metrics are in %. Bold indicates the best performance.
MVTe AD
VisA
setting
AC
AS
(a)min
96.8
94.1
91.0
98.4
Datasets
RoCIN
AUROC
Fl-max
AP
(bjmax
83.8
92.9
74.9
95.7
wl
97.4
96.6
98.7
[c]mean
95.0
96.8
89.6
98.4
MVTec AD
97.5
(d)30%
98.4
(e)30% mux
96.0
97,5
91.2
98.6
VisA
wlo
90
87.1
90.9
(9)30%
97.8
97.3
98.7
92.8
89.5
93.5
improvement on PRO and 21.9% improvement on AP than the second-best method in the zero-shot setting. On VisA, in classification and segmentation, method has more than 10% improvement compared with the second-best method. By comparing with some 4-shot methods, the classifica- tion and segmentation results of MuSc beyond most methods with four normal reference images, although we do not use any labeled images.
We show the industrial image visualization of anomaly maps of MuSc in Fig, 6 Compared with other zero/few-shot methods, our method can detect small defects and has fewer false positives, as sensitive the normal spots
shown in the pill and cashew. Other methods in zero/few-shot the pill and the tiny splinters on the cashew. To the large defects shown in bottle and pebl, our method can segment the whole regions. Some logical anomalies, such as the flip of metalnut and the proliferation of pipe-fryum, can be segmented entirely by our method.
Comparison with many-sho methods. We compare our MuSc with some many-shot methods on MVTec AD, shown in Tab. 2 MuSc is comparable to some full-shot methods, such as CutPaste (Li al. 202T). IGD (Chen al. 2022) and NSA (Schlüter al.) (022), which utilize all the normal reference images in the train set. Without using any labeled images, our method also outperforms RegAD with 32-shot and competitive with GraphCore with 8-shot.
4.2 ABLATION STUDY
The influence of the aggregation degree r. The aggregation degree controls the size of the local neighborhood to be aggregated in the LNAMD module. As reported in Tab. 3 we conduct the exper- iments with combinations of different aggregation degrees. The experimental results demonstrate that using all three aggregation degrees of r E {1, 3. 5} is better than other cases in the compre- hensive results of AC and AS. Moreover, due to the different datasets with varying anomaly sizes, smaller is suitable for segmenting small abnormal regions, which is common on VisA, while a larger r does well in segmenting large abnormal regions, which is common on MVTec AD. To different datasets, use three scales of aggregation degrees.
balance the representation
Discussion of the mutual scoring mechanism. Here, we modify Eq. in six different cases, shown in Tab. 4 In (a), (b), and (c), we remove the operation of selecting the minimum 30% value interval. (a) and (d) means replace the average operation with the minimum operation, and in (b) and (e) we replace it with the maximum operation. (f) is the optimal setting for method. We also test the effect of the selection percentage shown Fig. Selecting the minimum 30% can bring better comprehensive AC and AS results on the two datasets.
8

---

Published as a conference paper at ICLR 2024
Table 7: The effect of dataset size. We divide the dataset into parts, evaluate image-AUROC and pixel-AUROC(%) on MVTec AD and VisA.
Table 6: Per-image inference time and maxi- mum GPU memory cost by our MuSc. We di- vide the whole dataset into 8 subsets, measured at NVIDIA RTX 3090 GPU. We report the time and GPU memory cost with different
Datasets
AC
A$
Size of the subiets
97.8
97.3
42-161
MVTec AD
97.1
97.2
21-84
Time (ms)
GPU (MB)
96.7
97.3
14-56
998.8
7168
98.8
150-200
605.8
5666
VisA
92.5
98.7
75-100
513.5
5026
92.3
98.6
50-67
Influence of the proportion of minimum value interval
Influence window mask sizes
Ac
No Jac
305
73%
176
MV AD
VisA
MV AD
VisA
Figure The influence of the proportion of minimum value interval selection in MSM.
Figure 8: The influence of different window mask sizes MVTec AD and VisA.
Discussion of the re-scoring with constrained image-level neighborhood. In Tab. 5 we remove the re-scoring with constrained image-level neighborhood module to check our method for clas- sification. The results demonstrate the RsCIN module could bring 0.9% F1-max-score gains on MVTee AD and 2.8% AUROC gains on VisA. Meanwhile, for the multi-window mask operation of RsCIN, we replace the multi-window with the single-window k E (., and full-window = oo without the mask operation in Fig. g Our multi-window strategy outperforms the single- scale strategy both datasets. In addition, we insert our RsCIN module into some other different AC/AS methods, and the classification results have been improved, detailed in Appendix AA...4
Discussion of inference time and memory cost. The inference time and memory cost also what we are concerned about. our MuSc, we report the per-image inference time and maximum GPU memory cost in Tab. 6 where the memory cost is measured on the most numerous dataset which contains 200 images. When the number of test images increases, the inference time and memory cost continue increase, which may become limitation of our method.
To break through this limitation, we divide the entire test images into s subsets, calculate the anomaly scores on every subset, and finally compute the AC and AS metrics together. The AC/AS results are shown in Tab. 7 where 42- 167 means the maximum size of the divided subset is 167 and the minimum size 42. As the subset size decreases, the performance decreases by less than 1.1% for AC and 0.2% for AS, which means our method works well even with tiny datasets. The inference time and memory cost also decrease as the subset size decreases, shown in Tab. (6] Therefore, when the test set is large, we consider using our MuSc in its subsets separately. Additionally, we conduct a comprehensive efficiency comparison in Appendix A..4
Discussion of few-shot setting. Our MuSc is pure zero-shot method that does not require any labeled images. We design two extension methods in the few-shot setting based on our method when small number of normal reference images are provided, significant improvement in the results when a few images are added because our method already uses large amount of normal prior information in the unlabeled test images.
Appendix A.6) There is no
CONCLUSION
In this paper, we propose novel zero-shot industrial AC and AS framework named MuSc. We plore the normal and abnormal cues implicit in the unlabeled test images. First, we fuse local neigh- borhood aggregated patch tokens in different degrees. Then, we use mutual scoring mechanism for each image. Finally, we propose an image-level AC optimization method. Notably, our method surpasses the existing zero-shot approaches, even outperforms most of the few-shot approaches, and is comparable to some full-shot methods.

---

Published as a conference paper at ICLR 2024
6 ACKNOWLEDGMENTS
This work was supported by the National Natural Science Foundation of China under Grant No. 62176098. The computation is completed in the HPC Platform of Huazhong University of Science and Technology.
REFERENCES
Toshimichi Aota, Lloyd Teh Tzer Tong, and Takayuki Okatani. Zero-shot versus many-shot: Un- supervised texture anomaly detection. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 5564-5572, 2023.
Matthew Baugh, James Batten, Johanna P Müller, and Bernhard Kainz. Zero-shot anomaly detection with pre-trained segmentation models. arXiv preprint arXiv:2306.0922 09269, 2023.
Paul Bergmann, Michael Fauser, David Sattlegger, and Carsten Steger. Mvtec ad-a comprehen- sive real-world dataset for unsupervised anomaly detection. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 9592-9600, 2019.
Yu Cai, Hao Chen, Xin Yang, Yu Zhou, and Kwang- Ting Cheng. Dual-distribution discrepancy with self-supervised refinement for anomaly detection in medical images. Medical Image Analysis, 86: 102794, 2023.
Yunkang Cao, Xiaohao Xu, Chen Sun, Yuqi Cheng, Zongwei Du, Liang Gao, and Weiming Shen. Segment any anomaly without training via hybrid prompt regularization. arXiv preprint 10724 2023.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self- vision transformers. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 9650-9660) 2021.
Xuhai Chen, Yue Han, and Jiangning Zhang. A zero-/few-shot anomaly classification and segmen- tation method for evpr 2023 vand workshop challenge tracks 1&2: 1st place on zero-shot ad and 4th place on few-shot ad. arXiv preprint arXiv: 2305..382.. 2023.
Yuanhong Chen, Yu Tian, Guansong Pang, and Gustavo Carneiro. Deep one-class classification via interpolated gaussian descriptor. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pp. 383-392, 2022.
Niv Cohen and Yedid Hoshen. Sub-image anomaly detection with deep pyramid correspondences. arXiv preprint urXiv:2005.02357. 2020.
Thomas Defard, Aleksandr Setkov, Angelique Loesch, and Romaric Audigier. Padim: patch distri- bution modeling framework for anomaly detection and localization. International Conference on Pattern Recognition, pp. 475-489. Springer, 2021.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An im- age is worth 16x16 words: Transformers for image recognition at scale. In International Confer- ence on Learning Representations, 2020.
Chaoqin Huang, Haoyan Guan, Aofan Jiang, Ya Zhang, Michael Spratling, and Yan-Feng Wang- Registration based few-shot anomaly detection. In European Conference Computer Vision, pp. 303-319. Springer, 2022.
Jongheon Jeong, Yang Zou, Taewan Kim, Dongqing Zhang. Avinash Ravichandran. and Onkar Dabeer. Winelip: Zero-ffew-shot anomaly classification and segmentation. In Proceedings of the IEFF/CVF Conference Computer Vision and Pattern Recognition, pp. 19606-19616 1,, 2023.
Jiyaan Jiang, Bo Wang, and Zhuowen Tu, Unsupervised metric learning by self- smoothig operator. In 20/ International Conference Computer Vision, pp. 794-801. IEFF, 2011.
10

---

Published as a conference paper at ICLR 2024
Jiarui Lei, Xiaobo Hu, Yue Wang, and Dong Liu. Pyramidflow: High-resolution defect contrastive localization using pyramid normalizing flow. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14143-14152, 2023.
Aodong Li, Chen Qiu, Marius Kloft, Padhraic Smyth, Maja Rudolph, and Stephan Mandt. Zero-shot anomaly detection via batch normalization. In seventh Conference on Neural Information Processing Systems, 2023.
Chun-Liang Li, Kihyuk Sohn, Jinsung Yoon, and Tomas Pfister. Cutpaste: Self-supervised learning for anomaly detection and localization. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 9664-9674 2021.
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer. Hierarchical vision transformer using shifted windows. In Proceedings the IEEE/CVF international conference on computer vision, pp- 10012-10022, 2021.
Zhikang Liu, Yiming Zhou, Yuansheng Xu, and Zilei Wang. Simplenet: A simple network for image anomaly detection and localization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 20402-20411, 2023.
Pankaj Mishra, Riccardo Verk, Daniele Fornasier, Claudio Piciarelli, and Gian Luca Foresti. Vt-adl: A vision transformer network for image anomaly detection and localization. In 202/ IEEE 30th International Symposium on Industrial Electronics (ISIE), pp. 01-06. IEEE, 2021.
Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Mare Szafraniee, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint 2rX1v:230..07/93, 2023
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp- 8748-8763. PMLR, 2021.
Karsten Roth, Latha Pemula, Joaquin Zepeda, Bernhard Schölkopf, Thomas Brox, and Peter Gehler. Towards total recall in industrial anomaly detection. In Proceedings the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14318-14328, 2022.
Marco Rudolph, Tom Wehrbein, Bodo Rosenhahn, and Bastian Wandt. Asymmetric student-teacher networks for industrial anomaly detection. In Proceedings of the IEEÉ/CVF Winter Conference on Applications of Computer Vision, pp. 2592-2602, 2023.
Hannah M Schlüter, Jeremy Tan, Benjamin Hou, and Bernhard Kainz. Natural synthetic anomalies for self-supervised anomaly detection and localization. In European Conference on Computer Vision, pp. 479-489. Springer, 2022.
Bo Wang and Zhuowen Tu. Affinity learning via self-diffusion for image segmentation and cluster- ing. In 2012 IEEE conference on computer vision and patlern recognition, pp. 2312-2319. IEEE, 2012.
Guodong Wang, Shumin Han, Errui Ding, and Di Huang. Student-reacher feature pyramid matching for anomaly detection. arXiv preprinl arXiv:2103. 04250.457....... 2021.
Guoyang Xie, Jingbao Wang. Jiaqi Liu, Feng Zheng, and Yaochu Jin. Pushing the limits of few- shot anomaly detection in industry vision: Graphcore. In International Conference on Learning Representations (ICLR), 2023.
Jihun Yi and Sungroh Yoon. Patch svdd: Patch-level svdd for anomaly detection and segmentation. In Proceedings of the Asian conference on computer vision, 2020.
Jinsung Yoon, Kihyuk Sohn, Chun-Liang Li, Sercan O Arik, Chen- Yu Lee, and Tomas Pfis- ter. Self-supervise, refine, repeat: Improving unsupervised anomaly detection. arXiv preprint arXiv:2106.0611 15, 2021.
11

---

Published as a conference paper at ICLR 2024
Vitjan Zavrtanik, Matej Kristan, and Danijel Skočaj. Draem-a discriminatively trained reconstruc- tion embedding for surface anomaly detection. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 8330-83333 2021.
Vitjan Zavrtanik, Matej Kristan, and Danijel Skočaj. Dsr-a dual subspace re-projection network for surface anomaly detection. In European conference on computer vision, pp. 539-554. Springer, 2022.
Hui Zhang, Zuxuan Wu, Zheng Wang, Zhineng Chen, and Yu- Yu-Gang Jiang. Prototypical residual networks for anomaly detection and localization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 16281-16291, 2023.
Dengyong Zhou, Jason Weston, Arthur Gretton, Olivier Bousquet, and Bernhard Schölkopf. Rank- ing data manifolds. Advances in neural information processing systems, 16, 2003.
Yang Zou, Jongheon Jeong, Latha Pemula, Dongqing Zhang, and Onkar Dabeer. Spot-the-different self-supervised pre-training for anomaly detection and segmentation. In European Conference on Computer Vision, pp. 392 408. Springer, 2022.
12

---

Published as a conference paper at ICLR 2024
A
APPENDIX
A.1 METHOО DETAILS
A.1.1 MSM MODULE DETAILS
The intuition of MSM based on the observation of industrial product images, the normal image patches could find relatively large number of similar patches in other unlabeled images, while the abnormal ones only have few similar patches. In order to clearly explain the detailed theories of the MSM module, we divide it into three steps.
The first step is to score each patch of each image using the other unlaheled images. We perform LNAMD on the patch tokens of VīT, each aggregated patch token is denoted as Paj E IRI where
mm.5
m [1, .M indicates the index of the patch, is the aggregation degree and E {1, L) is the stage/ of VIT. For a given aggregation degree r and stage of ViT, if the patch token m of I; is am..
SnLF of Jj. the image 15 assigns small anomaly score to Pi. see Eq. similar to any patch token Pjá
measure the Euclidean distance between PL and all patches in I5. The min operator is used find the most similar patch in Ij and uses it to score mn. When P., represents a normal region, the most similar patch in Ij is usually a normal image patch, resulting in a relatively small anomaly score. While Pi,' patch in L,, resulting in a larger anomaly score.
Specifically,
sm.. represents an anomalous region, it is difficult to find similar
The second step is to selectively utilize the anomaly scores derived from unlabeled images. We pro- pose an Interval Average (IA) operation, which only utilizes the minimum X% of the scoring vector AMT as defined earlier. In Tab. compared to the common practice of averaging all images, using IA to select the lowest 30% scores could improve the image-AUROC by 2.8% and the PiXeI-AUROC by 0.5% on MVTec AD. Regarding the overlap in Fig. 4tc), we observe that it is caused by normal patches with appearance variations from different normal images. These dissimilar normal patches are assigned higher anomaly scores by other normal patches. Consequently, for patches representing a normal region, the IA operation can often find similar normal patches in some unlabeled images, thereby reducing the anomaly scores of normal patches. For abnormal patches, due to the random- ness and unpredictability of the anomalies, the IA operation cannot find similar patches in unlabeled images. It also maintains higher abnormal score for the abnormal patch, further widening the gap between normal and abnormal patches.
The third step involves combining anomaly scores from multiple degrees and stages. We calculate Eq. [ and Eq. 2 for each combination of stage 1 and aggregation degree respectively. Then we calculate the average of all a. to obtain the patch-level anomaly score using Eq.
A.1.2 RSCIN MODULE DETAILS
In order to clearly explain the detailed theory of RsCIN module, take it to optimize the pixel- level anomaly classification score e; of image I; as an example. We assume that in the Multi-window Mask Operation, k (ki, k2). where kn < k2. According the similarity matrix W. we define the k-nearest neighbor image I, as {fr,.. f.]. Then their similarities image are defined as {wi, 1, ..., W., which are obtained by Ms @ W in Eq. 5 The D-1 is used to make the sum of these similarities equal to 1. The results of the transformation over different k-nearest neighbors are defined as { ait k,k, and {w.,i.... 1,k }. The equations for calculating w and w are,
W1,1 + W1.2 + + Wi where w E (w
Wi.j
k2
Wi,j
where 3 E (wi di (
W1,1 + W12 + + wa 1.
where wit means the normalized similarity between image 7; and image 1ú the k-nearest neigh- bor, is E {k1, k2). The pixel-level anomaly classification scores of the k images {iI..., fe) are (ci..., ). Then using Eq. 5 to optimize only the anomaly classification score e; of image I can be rewritten as,
13

---

Published as a conference paper at ICLR 2024
Σ
(10 + -.. (x c2)((2+)
k2)
(1 + (x)
3
(k)
Gi
(10 c1 +... iiki Ck2 + 11 i Ck2)
3
3
G
+
1
C1 +... + with Cb + w +... + wi Cba + u i, 9k1 +... + ui Ck2)
3
3
G
(10
+ +0/1)a +... + +w0,) + w Jon, + t +1 +... + (x2 Tka)
3
3
1
CijC
3
where 2 represents the optimized anomaly classification score image 4, Way is the similarity between image I; and image under the multi-window setting,
t +
< h if
Wij
(8)
if k1 < j lzz
Eq. shows that é is affected by anomaly classification scores in multiple k-nearest neighbors. According to this equation, the value of & increases if image 1, has high-scoring k-neighbors (i.c., scores Ej E (cI...,ūūx ) are high), and vice versa.
A.2
ADDITIONAL ABLATION RESULTS
A.2.1 EFFECT OF DIFFERENT BACKBONES
Table 8: Comparison of MuSc performance image-AUROC and pixcl-AUROC (%) on MVTec AD and VisA benchmarks across different pre-training methods with various backbone architec- tures.
MVTec AD
VisA
Pre-training Method
Pre-training Dataset
Arch.
AC
AS
AC
AS
ViT-B-16
ImageNet- lk
94,9
97.8
88.2
97.8
DINO
ViT-B-8
ImageNet-ID lk
96.2
98.1
93.5
98.2
VIT-B-14
96.4
98.0
89.7
98.6 4
DINOv2
ViT-L-14
LVD-142M
97.1
98.1
90.5
98.4
VIT-B-32
WTT-400M
90.0
95.6
79.3
95.8
VIT-B-16
WIT-400M
94,8
96.9
85.7
97.6
CLIP
ViT-B- 16-pus-4-
LAION-00MM
97.1
89.3
ViT-L-14
WIT-400M
96.1
97.4
90.9
98.5
VYT-L-14-336
WIT-400M
97.8
97.3
92.8
98.8
Swin-B-4
ImageNet-22K
87.7
92.9
80.1
86.2
Swin-L-44
ImageNet-22K
87.6
90.9
76.7
82.1
We explore the effect of different pre-training methods with various backbone architectures in our method, such DINO (Caron et al. 2021). DINOv2 (Oquab et al. 2023), and CLIP (Radford al.) 2021). addition, we also conduct experiments on the swin transformer (Liu et al. 2021). which varied-size window attention to calculate attention inside local windows to detect objects with different sizes. ViT-Large contains 24 layers, we keep the same setup with our experiment setting to divide them into 4 stages, and each stage comprises 6 layers. ViT-Base contains 12 layers, also
14

---

Published as a conference paper at ICLR 2024
divide them evenly into 4 stages, and each stage comprises 3 layers. For the swin transformer, follow its own setting to divide the whole backbone into stages. Then we extract the patch tokens output from each stage for method and use the linearly projected class token of the last layer for classification optimization, except the swin transformer without the class token. All other settings remain the same those in the main text.
We show the image-AUROC and pixel-AUROC using different pre-training methods with various backbone architectures in Tab. 8 CLIP with ViT-L-14-336 achieves better comprehensive results in anomaly classification and segmentation on MVTTcc AD and VisA datasets. We notice that the larger model with high resolution and small patch size is more suitable for AC and AS. Due to the large patches in some stages of swin transformer, the patch-level features extracted by it do not work well under our method.
A.2.2 ADDITIONAL LNAMD ABLATION RESULTS
(1)
E (3)
(5)
image
(b)
ground truth
(d) r C {3.5)
(e) E (1,3,5)
Figure 9: Visualization of the combination results of different aggregation degrees T.
We further study the implications of combining different aggregation degrees r E (1,3,5), and visualize the results of these combinations in Fig. [ The defect in the screw is relatively small in this example. The small defect is smoothed and has a lower score in large aggregation degree (r E (5}). We add the anomaly score with aggregation degree (r (5)) to the anomaly score with aggregation degree (r E (3)). We can see from (d) that the combination of (r (3, 5)) results in smaller anomaly score being added to a larger anomaly score in the small anomaly region. This leads to a decrease in the anomaly score and false negative, When we add larger anomaly score with aggregation degree (r {1}), the anomaly score within the black box in (e) increases and the false- neggaiee reduces. In order to take into account the various sizes of anomalies real industrial scenarios, we (r 11,3,55 5)) maintain balance.
Table 9: Ablation study of LNAMD with different aggregation degrees r in image-AUROC and piXel-AUROC (%) on MVTec AD and VisA benchmarks.
MVTec AD
VisA
r
AC
AS
AC
AS
(1)
96.9
94.6
92.8
97.9
[1.3]
97.7
96.3
93.7
98.6
(1,3.5)
97.8
97.3
92.8
98.7
3,5,7
97.3
97.5
90.8
98.7
[1, 5, 7.9
96.4
97.8
89.5
98.6
[1,
95.7
97.9
88.4
98.5
We explore the results of using aggregation degrees in the LNAMD module, shown in Tab, With the addition of a larger aggregation degree, we observe that the classification results continued to decline, and the effect segmentation is related to the size of the anomalies in the two datasets.
15

---

Published as a conference paper at ICLR 2024
There more large anomalies in MVTèc AD, a larger aggregation degree is conducive to the segmentation of large anomalies, while most of the anomalies in VisA are small, and the use of smaller aggregation degree has greater advantages.
A.2.3 ADDITIONAL MSM RESULTS
In the MSM module, we observe that some normal patches can only find a small amount of similar patches due to the appearance variations of different normal images. To reduce these dissimilar patches in scoring, we perform Interval Average operation the minimum 30% of A In this process, small number of patches that are similar to abnormal patches are retained, shown in the red dotted box the left corner of Fig. T0, which is the same Fig. This part will make the anomaly scores of some abnormal patches smaller, which may lead to false negatives. Therefore, conduct the experiment perform the interval average operation on the minimum Y% 30% of A instead of only 30%. Results are shown in Tab. 10 as the minimum Y% are removed, the classification and segmentation results get worse. For the unlabeled test set, directly removing the minimum part of anomaly scores will increase the anomaly scores of some normal patches, which may lead to some false positive cases, and the impact caused by this part will be greater than false negative cases caused by abnormal patches on two datasets.
1e4
7-
Quantity
5-
4-
3-
2-
0
0.0
0.2
0.4
0.6
0.8
1.0
Figure 10: The histogram of anomaly score of abnormal patch token.
Table 10: Ablation study of MSM with the different percentage interval selection in iURgc-AUROC and piXel-AUROC (%) on MVTec AD and VisA.
MVTec AD
VisA
percentage interval
AC
AS
AC
AS
0% ~ 30%
97.8
97.3
92.5
98.7
2% 30%
97.7
97.2
92.5
98.7
1% 30%
97.5
97.2
92.4
98.7
6% 30%
97.3
97.3
98.7
8% 30%
97.3
97.3
92.2
98.7
10% ~ 30%
97.1
97.4
92.0
98.7
A.2.4 ADDITIONAL RSCIN RESULTS
In Tab. verify that RsCIN module can be considered general plug-and-play module in
the industrial anomaly classification task, which works well in the classification optimization among
different anomaly classification methods. We choose some representative methods for each type of anomaly classification. Decconstruction-basee approaches such as AEM (Zavrtanik et al.) 2021)
and DSR (Zavrtanik et al.) 2022). Methods based on student-teacher architecture such as STPM
Waan et al. 202 And methods for using memory bank such SPADE (Cohen & Hoshen 2020)
and PatchCore (Roth et al.) 2022). In addition, we also conduct experiments on some zero-shor and
few-shot methods, such RegAD (Huang et al. 2022 in the and 8-shot settings, APRIL-GAN CCen et al. 2023 the 0-shot and 4-shot settings.
We insert RsCIN module the end of these methods optimize the classification results. In all experiments, we maintain the same parameter settings, and uniformly use the class token extracted from ViT-L--14-336 image-level features and input them into the RsCIN module. We observe
16

---

Published as a conference paper at ICLR 2024
Table 11: Ablation study of RsCIN. We report anomaly classification (AC) performance in AUROC, Fl-max, and AP. All the methods are tested on MVTec AD except the method marked with an asterisk(*) represents testing on the VisA benchmark. All metries in %.
Methou
RSCIN
AUROC
Fl-max
AP
Method
RsCIN
AUROC
max
AP
SPADE
W/o
85.4
90.1
93.6
PatchCCore
wlo
99.0
98.4
99.7
Hoshen
2020
(Roth et
2022
98.4
DRAEM
w/o
98.0
97.0
99.0
DSR Zavrtanik al.
w/o
98.2
96.6
99.1
Zayrtanik al.
2021
97.9
97.0
99.1
2022
98.2
96.8
99.3
STPM
98.2
Rog.DD2-shoot
w/o
84.8
90.7
92.5
Wang et al.
2021
95.6
96.5
98.5
(Huang et al 2022
86.2
91.6
93.1
APRIL-GAN
w/o
86.1
90.4
93.5
RegADX4-shoot)
w/o
89.1
92.4
94.9
Chen al.
2023
86.1
90.8
93.7
W
91.0
93.5
95.8
w/o
92.8
92.8
96.3
RegAD(S-shoo)
w/o
91.2
92.9
95.7
92.1
APRII-GAN' (O-shot)
w/o
81.4
APRIL- -AN"(4-shooo (4-shot)
92.6
88.4
94.5
78.7
80.1
82.0
94.5
90.5
95.8
that our RsCIN module can improve the classification results of different AC and AS methods. RsCIN significantly improves SPADE, DRAEM, RegAD, and other methods. However, of the limitations of RsCIN is that for some one-class methods that do not contain image-level features in their backbones, another backbone needed to extract image-level features. Another limitation is that for methods with very high classification results, the improvement brought by RsCIN is limited by the representations of image-level features.
A.3 THE INFLUENCE OF ORIENTATIONS SCALES
Comparison with some zero-shot methods in image-AUROC (AC) and pixel-AUROC (AS) on the categories of MVTec AD and VisA with inconsistent orientations and scales.
Table 12:
WinCLIP
APRIL-GAN
MuSc
Category
AC
AS
AC
AS
AC
AS
screw
83.3
89.6
84.9
97.8
83.5
98.9
hazelnut
93.9
94.3
89.6
96.1
99.6
99.4
metal
97.1
61.0
68.4
65.4
96.3
86.0
capsules
85.0
61.2
88.8
macaroni2
63.7
59.3
64.6
97.8
69.9
97.2
mean
84.6
77.2
73.7
90.9
87.6
96.1
mean-ALL
91.8
85.1
86.1
87.6
97.8
97.3
We select the categories with inconsistent orientations or scales from MVTec AD and VisA datasets for the additional experimental evaluation, which is given in Tab. [2] We compare method with WinCLIP and APRIL-GAL and present the inage-AUROC/pixe-AA...ee that both our approach and WinCLIIP influenced by inconsistent orientations scales. find the reason is that both us use the fixed pre-trained vision transformer as the feature extractor, and introduced in (Radford al. 202I), during the pre-training process, the vision transformer only performs minor data augmentation on orientations and scales. Therefore, neither of us can effectively address the inconsistent orientations or scales. APRIL-GAN also achieves a decreased AC score, while it obtaining better AS score. We guess the reason is that APRIL-GAN utilizes an additional training set to optimize the AS. addition, our approach still outperforms WinCLIP and APRIL-GAN in case of inconsistent orientations and scales.
in this table. We observe
A.4 DETAILED COMPARISON OF INFERENCE TIME AND MEMORY COST
In Tab. 13 we show the inference time and GPU memory cost of our method under different settings. We also compare them with other zero/few-shot methods. WinCLIP and APRIL-GAN are in zero-shot setting, and RegAD is 4-shot setting. In order to maintain the same backbone these methods, we use both ViT-I 14-336 and VIT-B-16-plus-240. We find that the replacement
17

---

Published as a conference paper at ICLR 2024
Table 13: Comparison with some zeroffew-shot methods in per-image inference time (ms), GPU
cost (MB), image-AUROC (%) and AUPRO (%) on MVTec AD,
Method
Backhone
Training
time(ms)
GPU(MB)
AC
AS
RegAD Huang et al. 2022
ResNet-18
309.4
8920
89.1
88.0
APRIL--AAN (Chen et al.
2023
ViT-L-14-336
yes 100.2
4996
86.1
44.0
MuSo(s
ViT-L-14-336
998.8
7168
97.8
93.8
MuSc(s = 2)
ViT-L-14-3366
no
605.8
5666
97.1
93.8
MuSc(s = 3)
ViT-L-14-336
no
513.5
5026
96.7
93.7
APRIL-GAN
VTT--166-plu-220
64.7
3226
WinCLIP (Jeong al. 2023
389
91.8
64.6
MuSc(s =
ViT-B-16 I6T-B-16-plus-200
455.3
3002
95.4
91.9
MuSc(s = 2)
ViT-B- VTT-B-16-pin plus-ls 240
p-00s-2240
no
285.7
2740
95.4
91.5
MuSc(s = 3)
ViT-B- 1e-B--1--luus244
no 238.3
2684
95.2
91.4
of ViT-L-14-336 with ViT-B- WTT-B-16-phhe 16-plus-240 resulted in a significant reduction in inference time and GPU memory cost. When we use ViT-L-14-333 as the feature extractor, the large ViT divides the original image into more patches. In the LNAMD module, all patches are required perform local neighborhood aggregation, and the time increases the number of patches increases. These patches are aggregated with different aggregation degrees r (1,3,5), 5), and saved for later use. The aggregated features in the LNAMD module consume significant amount of memory. In the MSM module, mutual scoring is performed on each combination of stage and aggregation degree r, the patch of image needs to calculate the distances with all patches of all other unlabeled images. The size of unlabeled images is closely related to the inference time of the MSM module.
For method, divide the entire dataset of unlabeled images into subsets the way the ablation study, which highly effective in improving the inference time and reducing GPU memory cost. Under the setting of using VTT-L-14-336
the feature extractor, the inference time of our method is 513. 5ms when 3. which is half the time consumed when using the entire unlabeled set. Under the setting of ViT-B-16-plus-
our inference time is further reduced to 238.3m. and the GPU memory cost is only 2684M B, which is 150.7ms less than that of the WinCLIP method. When we divide the original dataset into three subsets, the segmentation performance reduces by less than 0.1%, and the classification result decreases by 1.1% with ViT-L-14-336. When we use ViT-B- ViT-B-16-plus 16-plus-224 240 as our backbone, the segmentation performance reduces by less than 0.5%, and the classification result decreases by 0.2%. As the subset size decreases, the classification and segmentation performance decreases very little, which proves that our method also achieves good results the small-scale of unlabeled images.
Table 14: Comparison with some existing zero/many-shot methods in image-AUROC; and pixel-
AUROC (%) on BTAD dataset.
Method
Setting
AC
AS
VT-ADL Mishra al. 2021
full-shot
83.7
90.0
P-SVDD & Yoon 2020)
full-shot
83.3
92.1
SPADE CChen & Hoshon
2020
full-shot
87.6
96,9
PaDIM Detard et all 2021
full-shot
93.7
97.3
PyramidFlww (Lei al. 2223
full-shot
95.8
97.7
PatchCore (Roth
4-shot
91.4
96.3
RegAD (Huang et al. 2022
4-shot
APRIL-GAT (Chen et al. 2023)
4-shot
91.3
92.4
APRIL-GAN
0-shot
72.3
89.6
MuSe
0-shot
94.8
97.3
A.5 ADDITIONAL DATASET RESULTS
To demonstrate the stability of the hyperparameters MuSc, we directly applied the hyperparam- eters on the MVTTec dataset test additional datasets. In Tab. 14 we compare our MuSe with
18

---

Published as a conference paper at ICLR 2024
Table 15:
k-shot results of MuSc in image- AUROC and pixel-AUROC (%) on MVTec AD.
Table 16: k-shot results of MuSc+ in image-
AUROC and pixel-AUROC (%) on MVTec AD.
setting
AC
AS
setting
AC
AS
O-shot
97.8
97.3
O-shot(MuSe)
97.8
97.3
I-shot
97.8
97.3
I-shot
92.7
97.2
2-shot
97.7
97.3
2-shot
93.5
97.4
4-shot
97.8
97.4
4-shot
96.5
97.8
8-shot
97.8
97.4
8-shot
97.3
98.1
16-shot
97.9
97.5
16-shot
98.0
98.3
32-shot
98.0
97.4
32-shot
98.7
98.4
full-shot 98.4
97.9
full-shot
99.5
98.5
several many-shot methods on the BTAD dataset. In zero-shot setting, MuSc achieves 22.5%
image-AUROC gain and 7.7 pixel-AUROC gain compared to APRIL-GAN. MuSc even out- performs RegAD and PatchCore in 4-shot setting and is competitive with some methods in full-shot setting. We emphasize that MuSc achieves excellent results even without adjusting the hyperparam- eters for specific dataset.
A.6 RESULTS OF TWO EXTENSION METHODS IN FEW/MANY-SHO SETTING
In the few/many-shot setting, several normal reference images Dret {4, I,..., R} are ran- domly drawn from labeled normal train set D Keeping similar structure to MuSc, we de- signed two few/many-sho extension methods. In the first extension, we use the same structure as original MuSe and input the Dref- the second extension, we design the MuSc+ with minor modifications to MuSc to make better use of labeled normal images.
A.6.1 MUSC FOR FEW-SHOT
Keep the MuSc method, we simply add these reference images Diet in MSM module. Given a test image 4, when using other images calculate the anomaly score for each aggregated patch token of 1, following Eq. [. we not only use the images in { ,D,, }. but also use these given normal reference images Dret- Then we follow Eq. X and Eq. 3 obtain few/many-shot segmentation results and
get classification results by RsCIN in Section 3.3 The results are shown in Tab. [5] We notice that some normal reference images are added resulting large amount of normal implicit information from unlabeled test images. The information is limited when a small number of normal images are added.
small improvement because we have used a
A.6.2 MUSC+
In our MuSc, some abnormal images also participate in mutual scoring, which is unavoidable in unlabeled test images. Our MuSc is designed with this case, so when some labeled normal images are added, we design different scoring mechanism to process these images. By changing the MSM module in our MuSc, we propose new scoring mechanism to obtain the anomaly score by only using the normal reference images Diet- Given a test image 4, we leverage the images in D to assign the anomaly score for each aggregated patch token of I; in layer using aggregation degree r, following Eq. 9
m. (42) min |P -
where I, indicates that image I, E (Der) is employed scoring, and hence we obtain the scoring m. ( vector A.t. [a")"((1)) (I1), a, (I2), a. m.r (IR)] for each and according D We perform minimization operation on A,. i.e.
mB. min a. (4₃)
(10)
we perform Eq. 9 and Eq. 10 on each combination layer and aggregation degree r, respectively, and compute the average of all a to achieve the patch-level anomaly a,"
19

---

Published as a conference paper at ICLR 2024
Σ
m T
(1I)
1E(1...... rE(13355
Then we follow the rest of the steps in Section 3.2 and Section 3.3 to obtain classification results
C and segmentation results Á. We show the MuSc+ results in Tab. 16 We observe that when the number of reference images is small, the classification result of MuSc+ is much lower than that of MuSe, while the segmentation result is close to that of MuSc. With the increasing number of reference images, the classification and segmentation results are also improving.
A.7 DETAILED QUANTITATIVE RESULTS
In this section, we report the detailed results of our MuSe for each category on the MVTec AD and VisA datasets, which are presented Tab. 17 and Tab. 18
Table 17: Quantitative results the MVTec AD dataset. All metrics in %.
class
AUROC-els Fi-max-cls AP-cls AUROC-segm Fl-max-segm
AP-segm
PRO-segm
bottle
99.9
99.2
100.0
79.6
83.2
cable
99.0
97.3
99.4
96.3
62.3
59.0
capsule
96.7
95.3
99.3
98.9
49.5
48.1
95.5
carpet
99.9
99.4
100.0
99.5
73.4
75.6
97.6
grid
98.7
96.5
99.5
98.4
44.6
39.4
94.7
hazelnut
98.6
99.8
73.8
leather
100.0
100.0
100,0
99.7
63.7
65.4
98.7
metal.nut
96.3
97.4
99.1
86.0
46.3
47.5
89.5
pill
96.4
96.2
99.3
97.6
66.5
68.0
98.0
screw
83.5
89.4
91.3
98.9
42.9
37.6
94,8
tile
100.0
75.3
78.9
94.9
toothbrush
100.0
100.0
100.0
99.5
70.3
67.5
95.8
transistor
99.1
94.7
98.8
92.0
59.6
59.0
78.3
wood
98.5
98.3
99.5
97.4
69.0
75.5
94.8
zipper
99.9
99.6
100.0
98.4
62.4
61.5
94.7
Mean
97.8
97.5
99.1
97.3
62.6
62.7
93.8
Table 18: Quantitative results the VisA dataset. All metrics are in %.
class
AUROC-cls F1-max-cls AP-els
AUROC-segm F1-max-segm A A--segm PRO-segm
candle
91.3
39.4
28.7
capsules
86.1
93.9
98.8
50.5
44.0
89.3
cashew
98.6
96.0
99.3
99.3
74.5
77.0
94.3
chewinggum
98.3
96.0
99.2
99.5
59.7
59.2
88.5
fryum
99.0
98.0
99.6
97.8
$7.1
49.5
94.5
macaronil
83.9
22.6
15.3
macaroni2
69.9
70.5
68.9
97.2
12.4
4.5
88.8
pebl
89,8
85.6
89,9
99.5
80.3
88.2
92,9
pcb2
93.4
88.6
94.3
97.6
34.8
22.1
86.6
pcb3
86.8
93,9
41.3
41.6
pcb4
93.8
46.1
44.6
92.5
pipe_fryum
98.4
97.0
99.2
99.4
67.4
67.2
97.5
Mean
92.8
89.5
93.5
98.8
48.8
45.1
92.7
A.8 DETAILED QUALITATIVE RESULTS
In this section, we provide further qualitative results of every category on the MVTec AD and VisA datasets. We report the segmentation results of our MuSe for varying sizes and types of anomalies
20

---

Published as a conference paper at ICLR 2024
each category of both two datasets. The VisA results are from Fig.
11 to Fig. 13 and the MVTec
AD results are from Fig. 14 to Fig. 16
Image
GT
Ours
Image
GT
Ours
Qualitative results of our anomaly segmentation results the VisA benchmark. In the first row, from left to right are the results for candle and capsules categories. the second row, from left to right are the results for chewinggum and fryum categories.
Figure 11:
21

---

Published as a conference paper at ICLR 2024
Image
GT
Ours
Image
GT
Ours
Qualitative results of anomaly segmentation results the VisA benchmark. In the first row, from left to right are the results for cashew and macaronil categories. In the second row, from left to right the results for pcbl and pcb2 categories.
Figure 12:
22

---

Published as a conference paper at ICLR 2024
Image
GT
Ours
Image
GT
Ours
Qualitative results of anomaly segmentation results the VisA benchmark. In the first row, from left to right are the results for pcb4 and macaroni2 categories. the second row, from left to right are the results for pipe fryum and pcb3 categories.
Figure 13:
23

---

Published as a conference paper at ICLR 2024
Image
GT
Ours
Image
GT
Ours
Image
GT
Ours
800
500
Qualitative results of our anomaly segmentation results on the MVTec AD benchmark. In the first row, from left to right are the results for bottle, cable, and capsule categories. In the second row, from left to right are the results for carpet, grid, and hazelnut categories.
Figure 14:
24

---

Published as a conference paper at ICLR 2024
Image
Ours
Image
Ours
Image
Ours
Qualitative results of our anomaly segmentation results the MVTec AD benchmark. In the first row, from left to right are the results for leather, metal nut, and pill categories. In the second row, from left to right the results for screw, tile, and toothbrush categories.
Figure 15:
25

---

Published as a conference paper at ICLR 2024
Image
GT
Ours
Image
GT
Ours
Image
GT
Ours
Figure 16:
Qualitative results of our anomaly segmentation results on the MVTec AD benchmark. From left to right are the results for transistor, wood, and zipper categories.
26


Reviews:
[{"review_id": "p5amrLZXIi", "reviewer": "Reviewer_NLA8", "summary": "This paper presents a MuSc model for zero-shot AC/AS. The method exploits the normal and abnormal information implicit in unlabeled test images without any training or prompts, A Mutual Scoring Mechanism (MSM) is proposed to assign abnormal scores to each other using unlabeled test images, and an optimization method based on constrained image-level neighborhoods (RsCIN) for image-level anomaly classification. The method shows better performance on MVTec AD and VisA datasets.", "questions": "See above", "limitations": "", "rating": 6, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "1. The idea that normal image patches can be found in a relatively large number of similar patches in other unlabeled images, while abnormal image patches have only a small number of similar patches, is novel. The authors were able to accomplish this task simply by using a test dataset and utilizing the test images to score each other without any training or prompting.\n\n2. The authors considered the size of the anomalies in different datasets and used patch tokens with multiple aggregation degrees to obtain high-quality anomaly scores even when using a simple distance measure.\n\n3. The authors found that the image-level features satisfy the conditions of high-dimensional manifolds, and designed RsCIN based on manifold learning to optimize the pixel-level anomaly classification results, and experimentally verified the effectiveness of the module, proving that the proposed RsCIN module can further improve the performance of the existing methods.", "weaknesses": "Even without training or prompts, the method requires long inference times and high memory costs. While the authors provide a solution to increase speed and reduce memory by dividing the test set into subsets, this also reduces performance by a small margin. Are there any other approaches that could have been considered to solve the problem?\n\nThe authors should provide more detailed theories for MSM and RsCIN.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "yinO0jnVvj", "reviewer": "Reviewer_52S5", "summary": "The paper proposes an anomaly detection and segmentation method utilizing unlabeled images from a test set. It is assumed that a set of images with some form of anomaly is available for inference and it is claimed to be zero shot. \nThe method starts with computing a representation for every image patch through aggregating (pooling) token features generated by ViT at different layers. An anomaly score for each patch is then computed by comparing the aggregated patch token with that from each image in the *test* set. Further heuristic tricks are applied to refine this score and the final pixel level anomaly score is given by the max of the refined score. \nThe method then produces an image level classification by defining a weighted graph over *test* set images where weights are defined by the class token generated by ViT. The image level classification score is the determined by by some graph operations that were not explained well. The method was tested on public datasets and compared with solid baselines.", "questions": "...", "limitations": "", "rating": 3, "confidence": 3, "soundness": 1, "presentation": 1, "contribution": 1, "strengths": "I am afraid I was not able to find a notable strength of this paper.", "weaknesses": "1. Technical soundness: The method relies heavily on availability of a set of images in order to compute the anomaly scores. I dont think this emulates the practical scenario of anomaly detection in industry. The more realistic scenario is a method is required to classify and segment the anomaly given one image. Using test set images to produce the output on the same set of images also does not conform with the scientific procedure. I dont think this is the definition of a zero shot method either. The setting sounds completely unreasonable to me.\n\n2. Contribution: The algorithm appears to be a set of heuristic tricks applied in a sequence. There is not a solid technique that is novel, elegant, theoretically justified and of broad interest.\n\n3. Clarity: None of the techniques were adequately explained as to why they are being performed and why it makes sense (intuitively or conceptually) to apply them. Just stating the process does not qualify as a good scientific exposition.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "nd4rOlaFoy", "reviewer": "Reviewer_idjR", "summary": "The paper introduces MuSc, a novel zero-shot framework for industrial anomaly classification and segmentation. It utilizes cues from unlabeled test images and combines local patch tokens with a mutual scoring mechanism. The method notably outperforms existing zero-shot approaches and rivals many few-shot and full-shot methods.", "questions": "See the weaknesses.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The MuSc framework introduces a novel approach to zero-shot anomaly classification and segmentation, particularly in the industrial domain. The method of leveraging implicit cues from unlabeled test images for anomaly detection is an innovative concept.\n\n2. The empirical results demonstrate a substantial improvement over existing zero-shot approaches and competitiveness with few-shot and full-shot methods.\n\n3. The approach holds significant potential for industrial applications, where anomaly detection is crucial but training data is often scarce or expensive to obtain.", "weaknesses": "In Table 3, the ablation study of LNAMD with different aggregation degrees \\(r\\) raises a question regarding the effectiveness of combining aggregation degrees. Specifically, it's unclear why the combination of \\({3, 5}\\) performs worse in the anomaly classification (AC) task than using \\({3}\\) alone. This observation seems to contradict the paper's claim that using all aggregated patch tokens with different degrees is beneficial for detecting anomalies of various sizes. Based on this claim, one would expect the combination of \\({3, 5}\\) to outperform either \\({3}\\) or \\({5}\\) individually. This inconsistency warrants further clarification or investigation to reconcile the results with the stated claims.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Uk9PdZmZYz", "reviewer": "Reviewer_dCfU", "summary": "This paper addresses zero-shot anomaly classification (AC) and segmentation (AS) with following contributions:\n1) Using unlabeled test images for AC/AS.\n2) A new mutual scoring mechanism for identification of abnormal patches.\n3) SOTA performance, significantly outperforming existing zero-shot methods", "questions": "1) Could your approach handle a setting when a test dataset lacks ground truth? What is performance when a test dataset lacks ground truth for optimizing the proposed heuristic procedure? \n\n2) One of the claimed contributions, the mutual scoring mechanism, appears to depend significantly on the relative positions of feature patches. This limits the application of this approach in real settings where orientations and scales are not consistent. What is performance when test images exhibit inconsistent orientations or scales?\n\n3) Could your method be extended to (or readily address) the few-shot setting?", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The claimed contributions summarized above.", "weaknesses": "1) This paper makes the assumption that access to the entire test dataset is available. This allows for a direct application of the proposed mutual scoring mechanism. However, for zero-shot settings, such an assumption is too restrictive.  Access to test data is mostly very limited in real-world scenarios, especially for zero-shot.\n\n2) The method heavily relies on many heuristic design choices. It consists of a three-step pipeline which depends on many hyperparameters for feature representation, mutual scoring estimation, and classification rescoring. The paper poorly presents sensitivity of performance to optimizing all these parameters.\n\n3) Overall technical novelty seems incremental, since the method incorporates well-established multiscale features (Sec 3.1), norm-2 distance (Sec 3.2), and clustering (Sec 3.3). \n\n4) Since the method lacks a training phase and directly generates results based on estimating the test-set statistics, inference is likely to be much slower than in previous approaches. I was not able to find a report/comparison of the inference times.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "y0kysomv1q", "reviewer": "Reviewer_H5pV", "summary": "This paper targets industrial zero-shot anomaly detection. Leveraging mutual scoring on unlabeled data, the proposed method achieves SOTA performance on several well-known industrial anomaly detection benchmarks.", "questions": "See the weakness.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "1.\tThe paper is well written, with clear architecture.\n2.\tThe motivation is insightful, with clear ablation studies to show its effectiveness. The motivation and the proposed method are well-matched.", "weaknesses": "1.\tThis paper employs a methodology that utilizes unlabeled test images to collectively measure anomaly scores, differing somewhat from the traditional zero-shot recognition setting. In the conventional approach, each image is independently evaluated, such as WinCLIP. This difference in evaluation methodologies can result in a somewhat unfair comparison.\n2.\tThe underlying concept is reminiscent of [a], which presumes homogeneous input texture and identifies image regions that disrupt this homogeneity as anomalies. In other words, if a patch significantly differs from its neighboring areas, it's deemed an anomaly.\n3.\tThe approach of jointly measuring anomaly scores across entire datasets aligns closely with [b]. While not mandatory, it would be beneficial for the authors to discuss or draw comparisons with [b].\n\n[a] Aota et al. \"Zero-shot versus Many-shot: Unsupervised Texture Anomaly Detection.\" In WACV 2023.\n\n[b] Li et al. \"Zero-Shot Batch-Level Anomaly Detection.\" arXiv, February 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)