Hierarchical Vector Quantized Transformer for Multi-class Unsupervised Anomaly Detection
Authors: Ruiying Lu
YuJie Wu
Long Tian
Dongsheng Wang
Bo Chen
Xiyang Liu
Ruimin Hu
Venue: NeurIPS 2023 poster (2023)
Keywords: Anomaly Detection
Transformer
Vector Quantization
Unsupervised Anomaly Detection

Abstract: Unsupervised image Anomaly Detection (UAD) aims to learn robust and discriminative representations of normal samples. While separate solutions per class endow expensive computation and limited generalizability, this paper focuses on building a unified framework for multiple classes. Under such a challenging setting, popular reconstruction-based networks with continuous latent representation assumption always suffer from the "identical shortcut" issue, where both normal and abnormal samples can be well recovered and difficult to distinguish. To address this pivotal issue, we propose a hierarchical vector quantized prototype-oriented Transformer under a probabilistic framework. First, instead of learning the continuous representations, we preserve the typical normal patterns as discrete iconic prototypes, and confirm the importance of Vector Quantization in preventing the model from falling into the shortcut. The vector quantized iconic prototypes are integrated into the Transformer for reconstruction, such that the abnormal data point is flipped to a normal data point. Second, we investigate an exquisite hierarchical framework to relieve the codebook collapse issue and replenish frail normal patterns.  Third, a prototype-oriented optimal transport method is proposed to better regulate the prototypes and hierarchically evaluate the abnormal score. By evaluating on MVTec-AD and VisA datasets, our model surpasses the state-of-the-art alternatives and possesses good interpretability. The code is available at https://github.com/RuiyingLu/HVQ-Trans.

Hierarchical Vector Quantized Transformer for Multi-class Unsupervised Anomaly Detection
Ruiying Lu YuJie Wu Long Tian Dongsheng Wang
Bo C Xiyang Liu², Ruimin Hu'
School of Cyber Engineering' Software Engineering Institute
National Key Laboratory of Radar Signal Processing
Xidian University tianlyinnn,taillooo. @xidian. edu..edduccc .n.
Abstract
Unsupervised image Anomaly Detection (UAD) aims learn robust and discrimi- native representations normal samples. While separate solutions per class endow expensive computation and limited generalizability. this paper focuses on build- ing a unified framework for multiple classes. Under such a challenging setting, popular reconstruction-basee networks with continuous latent representation sumption always suffer from the "identical shortcut" issue, where both normal and abnormal samples can be well recovered and difficult distinguish. To address this pivotal issue, we propose a hierarchical vector quantized prototype-oriented Transformer under probabilistic framework. First, instead of learning the con- tinuous representations, we preserve the typical normal patterns as discrete iconic prototypes, and confirm the importance of Vector Quantization in preventing the model from falling into the shortcut. The vector quantized iconic prototypes are integrated into the Transformer for reconstruction, such that the abnormal data point is flipped to normal data point. Second, we investigate an exquisite hi- erarchical framework relieve the codebook collapse issue and replenish frail normal patterns. Third, a prototype-oriented optimal transport method is proposed better regulate the prototypes and hierarchically evaluate the abnormal score. By evaluating on MVTee-AD and VisA datasets, our model surpasses the state-of- the-art alternatives and possesses good interpretability. The code is available at https: //tptuu///ithub.... com http/RRiiiiin cow/Ruiyynggu HVQ Trans
Introduction
Anomaly detection is an essential task with increasingly wide applications various areas, such as video surveillance [I], industrial inspection [2], and medical image analysis Due the scarcity of anomalous samples, the unsupervised anomaly detection [4HZ] methods gain wide attention by modeling the distribution of normal data only, and then identify the samples deviates from the normal profile as anomalies. Common approaches follow the one-for-one scheme g 9 by training separate models for different classes of objects, which is time-mimmen uncongenial to the object class with large intra-class diversity. Recently, newly emerging one-for-all scheme [10 tries use a unified model to detect anomalies from the different object classes without any fine-tuning. Modeling high-dimensional data is notoriously challenging, and the problem becomes even more difficult to capture the multi-class distribution unified model precisely.
for real application and
EEqual Contribution
* Corresponding Author
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Under the unsupervised setting, a powerful approach to modeling data distribution follows the deep autoencoding frameworks. reckoning that well-trained model with normal data will always recon- struct normal patterns regardless of the defects present in the input data. Thus, it is generally assumed that the reconstruction error will be larger for the anomalous input, making them distinguishable from the normal samples. However, this assumption may not always hold that sometimes the abnormal inputs can also be well reconstructed, which is named as "identical shortcut" issue [2II]. Intuitively, compared to working extremely hard to learn the joint distribution, returning direct copy of the input disregarding its content appears as a far easier solution. This phenomenon has been observed in existing researches [4 9]. Furthermore, under the unified case, the "identical shortcut" issue becomes even more severe as the distribution of multi-class data is more complex [10]. This motivates us to enhance the discriminability of model encountering normal and anomalous samples.
Learning representations with continuous features have been the foeus of many previous works 12- 14]. However, these methods lack reliable mechanism to encourage the model to induce large reconstruction error on the anomalies, restricting the performances by the under-designed represen- tation of the latent space. In recent researches, a branch of approaches memory-augmentee networks for mitigating the "identical shortcut" issue of AEs. Those approaches augment the deep autoencoder with memory module to record the normal patterns in the normal training data, manifesting in different forms such as the memory set in the latent space [9]. the fixed Transformer value matrix in the attention layer [10). or neighborhood-aweer aware patch-level memory bank [8). This kind of method hopes to obtain low reconstruction error for normal samples and highlight the reconstruction error if the input is not similar to normal data, that is, anomaly. The most relevant items in the memory are retrieved and weighted averaging all the related memory content are aggregated into the decoder for reconstruction. However, the discrete memory items recombined and weighted averaged, falling into an unknown continuous latent space which might be distorted. Intuitively, some anomalous regions can not be reconstructed by the discrete latent memory but could be decoded from the unknown latent space. To intrinsically mitigate the problems, we aim to learn a representative and discriminative discrete latent space for anomaly detection.
[8] 9 15 investigate the
To preserve the typical normal patterns in the dis- crete latent space, we hope to successfully model critical features that usually span many dimensions in the normal data space, opposed to focusing or spending capacity on noise and imperceptible details. Incorporating ideas from vector quantization (VQ), we model the discrete latent space as codebooks for each category, consisting of iconic prototypes learned from normal training data. During reconstruction, we replace the original encoding features with the nearest iconic prototypes, and then decoded with VQ-based transformer decoder to intensify the use of iconic prototypes. As a result, the abnormal data point is flipped to normal data point, highlighted by large reconstruction errors, as shown in Fig. [ However, the model may suffer from the codebook collapse sue [6. At some point during training, part of latent codes in the codebook may no longer work and the modeling capacity is limited by the discrete repre- sentations, resulting in collapsed reconstruction [18]. Thus, further investigate the hierarchical nature of images and propose hierarchical VQ frame- work by merging fine- mne-grained and abstract features to prevent codebook collapse, which could also reduce the decoding search time and retain high inference speeds. In addition, most abnormal scoring methods are constrained to the observation space and can be fallible to complex data distributions. Therefore, we have introduced hierarchical prototype-oriente optimal transport (OT) based opti- mization and anomaly detection scoring method to enhance the robustness and discriminability of our model for normal and anomaly samples.
hazelnut
hazellmt
reconenue
bazelnut
(ugemnal)
fanomaly
Iconic normal protatypes
tanomalyy)
(Bormal)
By replacing the continuous latent features with the normal iconic prototypes of corresponding category, the normal regions are reconstructed as normal patterns (shown in yellow boxes), while the anomalies are also reconstructed as normal (shown in red boxes).
Figure 1:
In conclusion, we carefully tailor a variational autoencoding framework for unsupervised anomaly detection, called hierarchical vector quantized Transformer (HVQ-Trans). Our work contributes in the following ways: (1) We realize the learning of discrete normal representations by extracting
2

---

Vanillr Eno Lagerd
Quantization
VQ Layeet
Des Layeed
vgTnas Layarà
Imput
0010000000 Protokypes
Ouiput
Feed Forward
Enc Layor
Layed!
TanilleTrar Ene
viJtans Des Laye2
POT
Laye??
K
VasillaTarss Ene Layeri
vQiinss Dei Layeri
Layert
Traqquet
(b) Iconic Protely pe-rietp-- Learning
Aftertion
VQ Layerti
Swiching:
Exper-i
Q
Position
Embedding
Expert2
Reconstructed
Roster
Output of the
Output of the last ViTTi Doc
Input Image
Fentures
Inpor Classiñeer
Eogarten
(a) Overall Framework
(c) Switching mechanlsm
(d) bused Transformer Decuder
Figure 2: (a) The overall framework of our HVQ-Trans. (b) Each VQ-based Layer replaces continuous features with iconic prototypes, equipped with the POT module to promote better learning and scoring, (c) The codebook and expert network switched for individual image. (d) The detailed structure of each VQ-based Transformer decoder, where the prototypes are integrated via cross-ates ausss--ttention.
prototypes and propose VQ-based transformer address the "identical shortcut" issue by inducing large feature discrepancy for anomalies. (2) We develop a hierarchical VQ-based approach with switching mechanism to overcome the "prototype collapse" problem and effectively use multi- level feature representations to maximize the nominal information available. (3) A hierarchical prototype-ories oriented -rriente learning and anomaly scoring method is developed to guide prototype learning and dexterously measure the feature level anomaly score to robustly and accurately identify anomalies. (4) Extensive experiments demonstrate our method achieves sfate-ofthhe-ee performances for anomaly detection and localization, and possesses enhanced interpretability through prototype visualization.
2
Methodology
2.1 Overview
Our proposed model is Transformer-bast reconstruction method that assumes normal and anomaly cannot be reconstructed with comparable performances. In contrast to other Transformer based autoencoding
[19] with continuous embeddings, we focus on compressing input images into discrete representations and achieve discriminative reconstruction for anomaly detection and localization. We denote the set of normal images available at training time as Xx (Vx Xx yı = 0), with Ve (0. 1} denoting if an image æ is normal (0) or anomalous (1). Accordingly, we define the test sample as Vx: E Xr y, {0, 1}. including both the normal test images and abnormal test images. The model pipeline, shown in Figure 2 can be summarized
follows: i) The input image is fed into the pre-trained EfficientNet (20) to extract visual tokens by splitting 3-D feature maps; ii) The extracted tokens are passed through the cascaded vanilla transformer encoder for non-local multi-level feature aggregation; iii) Aggregated features at certain layer are hierarchically fed into the corresponding VQ-based layer select the most relevant iconic prototypes; iv) The visual tokens are then successively fused with vector quantized embeddings via cascaded VQ-based transformer decoder for reconstruction: v) Decoded tokens passed through the switching experts to reconstruct features, which possess flexibility in high diversity multi-class image scenarios; vi) Finally, anomaly detection and localization are achieved through calibrated anomaly score map refined via prototype-oriented module by measuring the OT-based hierarchical feature discrepancy.
2.2 Improving Feature Reconstruction with Hierarchical Vector Quantization
Motivation: Normal memory augmentation was initially introduced by Gong et al. 9 and has obtained wide interests in unsupervised anomaly localization and detection. To record the "normal" appearance, image features are augmented by weighted averaging the similar patterns in the memory matrix. This augmentation is, however, rebuilding continuous latent space again which might be distorted to contain abnormal patterns. Relied on the vector quantization, discrete variational autoencoder(2 21 learns the discrete latent space, but suffers from the issue of codebook collapse.

---

Therefore, to simultaneously learn the discrete representation and avoid codebook collapse, we proposed the hierarchical VQ-based framework.
Our proposed HVQ-Tran Trans can be viewed as communication system serving as an information bottleneck to better capture the normal patterns during training, which could be further generalized to test unknown images with arbitrary anomalies. We denote the input image as a E R =x*xxx then the N visual tokens h" = fo (x) E R RNCC
Hierarchical Vector Quantized (HVQ) Transformer:
extracted by the pre-trained EfficientNet o² [20] to be fed into the Transformer encoder. As shown in the graphical model of Fig. 3 HVQ-Trans comprises a cascaded vanilla Transformer encoder (vanTrans-enc) parameterized by ø' that encodes multi-layer patch embeddings as h' = fa (1' E RN To further enlarge the normality and suppress the anomaly, we subsequently develop hierarchical VQ-based layers (VQLayer) to layer-wisely quantize the refined visual tokens to the prototypes ex in the learnable codebooks E' R
{h li=1
0 = Quantize (T') e = min || (h") - e2,12.
z' = Quantize (T' ((h-1.0)) = e k min || ,211 et 12.
l [.,... LJ,
where ('()) refers to the layer-wise embedding function, and denotes the concatenation operation. Intuitively, we hierarchi- cally replace visual tokens h with their most similar prototypes in the codebook E' quantized vector z' (note that this process can be lossy). Moreover, we find that merging fine- ame-ggrined con- crete information with abstraction-level semantics is critical for robust anomaly detection. Hence, we fuse the multi-level visual tokens with the global quantized vector I to learn hierarchical prototypes, maximizing the preserved nominal information.
[:]
8
VQ3
VQ2
VQ0
To make full use of the quantized multi-level visual tokens z derived from by Eq. [ we further developed a cascaded VQ- based Transformer decoder (VQTrans-dec). As shown in Fig. 2
Figure 3: Graphical model of our HVQ-Trans.
(d), given the quantized visual tokens z' of the 1th layer and its
corresponding output +1 from the last VQTrans-dec layer, the operation in each VQTrans-des layer can be expressed as:
q' MSA(query= = Wadtt key = Wid +1 value = W. + d'
(2)
d' MCA(query- = W,g. key Wiz,, value W + d, d' FF=')) +
where M.A(-) and MCA(.) share the same architecture as the standard multi-head self attention and multi-head cross attention in vanilla Transformer [22]. An essential aspect of our method is that, in the M-A(-) operation, the refined visual tokens from the previous layer q' crossly attend to the
prototypes of normal images z'_ Hence, the values at abnormal regions of q' will be suppressed, and the abnormal signals could be rarely transmitted to the output terminal for reconstruction. Namely, the fewer reconstructed anomalies there are, the larger the reconstruction difference will be, which in turn leads to better performance in localizing and detecting anomalies.
During training, the typical normal patterns are recorded in the discrete variables, i.e., iconic proto- types. When encountering the anomalous during testing, the abnormal patterns will also be quantized as the normal prototypes, leading to larger feature migration and information loss, highlighted by higher reconstruction error. It is worth noting that while information loss triggered by VQ is exist for normal images, is significantly more pronounced for anomaly images. This discrepancy in information loss serves key factor in effective anomaly detection. By investigating this difference. we can enhance the accuracy of our model in distinguishing abnormal regions.
Switching Mechanism: We adopt the switching mechanism to make the proposed model more suitable for multi-class anomaly detection. On the one hand, we develop the switching codebooks by assembling the independent iconic prototypes for each class. On the other hand, we develop the switching experty for flexible reconstruction of multi-classes, inspired by the Mixture of Experts (MoE) models 23 24) and its sparsely-activater version (25). Here, we choose to reconstruct at the

---

feature level rather than the pixel level, due to the invariance to subtle noise, rotation, and translation the pixel level.
Specifically, the switching mechanism contains a multi-category classifier, M codebooks, and M reconstruction experts. The multi-category classifier takes the image feature as input and outputs the classification probability over the M category. In order to fit the data diversity property in the one-for-all setting, we switch the specific codebook (including group of prototypes) from M codebooks according to the classification probability. Furthermore, we switch the individual reconstruction network (dubbed as expert) for feature reconstruction. The visual tokens from the last VQTrans-dec layer are depicted d' TRN which is expected to reconstruct the input patch features h. E NRN as:
exp()(()), Σ Σド exp((((z)))
h V vm(d"), m argmax py(0)
where (2(-) is a classifier for producing logits, which are then normalized via a Softmax function over the total M experts. (-) is the probability of selecting the codebook and reconstruction expert, as shown in Fig. 2 (c). The codebook and expert V n. with the highest probability are employed for reconstruction. The switching mechanism under the one-for-all setting will classify each input image into a single category and choose the corresponding codebook and expert for reconstruction. For the normal images, it is highly likely to be classified into the correct category and thus switch the proper reconstruction expert and codebook. For the abnormal images, there remains big uncertainty that which reconstruction expert and codebook will be switched, because the anomalies are unseen during training. Thus, the reconstruction uncertainty of the abnormal image is increased. Noting that the difference between normal and abnormal is the key factor deciding the anomaly detection performance. To this end, the uncertainty during anomalous sample switching could facilitate multi-class anomaly detection.
2.3 Prototype-oriented Learning and Scoring for Anomaly Detection
During training with normal data, the HVQ-Trans enhances the point-wise correlations of the selected iconic prototypes from codebooks and the continuous visual features of normal images. However, it may despise the global relations between the above two sets. Motivated by previous efforts on OT theory and applications 29) we propose a hierarchical prototype-orienter optimal transport (POT) for anomaly detection, which is a transport solver defined within the scope of the basic distance between two unknown sampling sets to improve the tightness between the codebooks and the normal features. Meanwhile, at the testing stage, it is worth noting that the anomaly score adopted in the existing methods [10 17 mainly concern the most significant difference of the score map between the input features and the reconstructed ones, as measured by the Euclidean distance. However, the importance of hierarchical differences at multiple feature levels is neglected. To this end, we also proposed a hierarchical POT-based anomaly scoring method to reinforce the identification of the score map and further boost the anomaly detection performance.
Motivation:
Learning Iconic Prototypes with POT: Each POT module, included within each VQLayer as shown in Fig. 2 (b). is responsible for enhancing consistency between codebooks and normal image features per layer. This enables the prototypes in codebooks to be more representative of normal patterns and less so of anomalous patterns. At the 1-th layer, the codebook of each category contains a group of prototypes e' = [e| .... ....
exxl E R C We omit the index / in the following for simplicity without causing ambiguity. To assemble the normal patterns conveyed by images, we represent N patches per image as an empirical distribution PA, = >m'1 Ó where E R NxC; is the features sampled from the latent variables of the IVQ-Trran. The prototypes serve represent normal patterns across different classes. As a result, when attempting to identify suitable prototypes to reconstruct a specific normal image, each prototype is given equal importance. Thus, the distribution over normal prototypes can also be expressed as an empirical distribution P.- Σ5= 7 Se,. In this way, the transport matrix M' N from IA to P. can be estimated by M min ΣN f Mi,Ci, where the K
M
transport matrix M should satisfy T[[ I [+)) (M MI J = [N)) [ and [ are two uniform distributed prior defined in Pr,, and P... respectively. The cost matrix C E N is defined as Ct, V(h: e2 In order to learn the prototypes of normal codebook certain layer,
5

---

we define the average POT loss inspired by Sinkhorn algorithm 30 as:
K
LPOT min E
E F (w)
ΣΣ =1
M MI
Ci.
EE
M Min ==
Calibrating Anomaly Score with POT: The anomaly score computed via Transformer-base based methods always suffers from the sub-optimal distance measurement, which is usually calculated as the point-wise L2 norm of the reconstruction differences as Borg = Ilf org f freell. 12. In this paper, on the hand, alleviate the mismatch by restricting the distance of prototypes and visual features during training. On the other hand, we further calibrate the anomaly score with multi-level POT at test time. Accordingly, in proposed method, note that the anomaly degree could also be reflected by the dissimilarity between visual features and normal iconic prototypes in the codebooks. As for the 1-th layer, the dissimilarity is evaluated by 8POT M'C. The transport matrix M" acts as a probability to re-weight the cost with different prototypes C, which measures the importance of different distances between image features and the normal prototypes. Therefore, calibrate the multi-level anomaly score as Scab = Sorg x >t
sPoT for better anomaly detection.
2.4 Overall Optimization
we approximate the gradient by copying gradients from the refined visual tokens z' to the visual tokens h for
Noting that there is no real gradient defined for eq following 17 21 31 32
I = O ..... L. Thus, our proposed HVQ-Tram -Tans incorporates five terms into its objective, specified as:
LavQ-Trane = 1 h 2
[(sg(th) - e'1 + s'In'' sg(e'))l + a'LLoor]
Σ (5(2)) log Ps,
(5) where the sg(+) refers to the stop-gradient operation and Pa the category label. , and a' are peprpaarametes. The first term refers to the reconstruction loss. The second one in the scope of summation along L layers is the hierarchical prototypical loss, pushing the selected prototype e closer to the visual token h'. The third term denotes the hicrarchical commitment loss, optimizing the encoder by encouraging the output of the encoder h' to stay close to the chosen prototype and prevent it from fluctuating too frequently from
prototype another. The fourth term is the POT loss defined in Eq. 4 The last term is the cross entropy loss for training the classifier to adaptively switch the proper reconstruction expert and codebook. Following [21], use the exponential moving average updates for codebooks. More details can be found in Appendix.
3 Connection with previous works
In unsupervised anomaly detection, only normal samples are available the training stage. Unsuper- vised anomaly detection methods can be roughly categorized into density- based and reconstruction- based methods. Density- based methods estimate the distribution of normal data points to identify anomalous data points 4 Our HVQ-Trans method adheres to the probabilistic variation frame- work but refrains from assuming a specific distribution of normal data. Therefore, the image prior learned dynamically rather than relying based methods assume that the model trained on normal data only can well reconstruct normal regions, but fail in anomalous regions 35) Typical approaches include Auto-Encoder (AE) (36-38), Variational Auto-Encoder (VAE) 33 [40] and Generative Adversarial Net (GAN) 441433) However, most of these methods do not incorporate a reliable mechanism for encouraging the model to induce large reconstruction error on the anomalous region.
static distribution. On the other hand, reconstruction-
Adopting a memory matrix for unsupervised anomaly detection has proven to be an effective solution. The idea was first proposed MemAE 9 by injecting an extra memory matrix to assemble normal patterns during training. Based on this paradigm, the memory-based models have attracted attentions recent years [5]
These methods always record the normal patterns into the memory, then recombine and re-weight the relevant patterns for reconstruction. However, if anomalous representations can be recovered through the re-weighting of normal patterns, the discriminating process may collapse. In contrast, our method compresses images into a discrete latent space, inspired by the vector quantization technology [2] referring to ideas from lossy compression to relieve the model from modeling negligible information. Additionally, our hierarchical framework allows us to increase the size of the codebooks without incurring the codebook collapse problem, achieving meticulous anomaly detection at multiple levels with prototype-orien oriented scoring method.

---

Table 1: Anomaly detection/localization results with AUROC metric on MVTec-AD. All methods are evaluated under the one-for-all settings. The learned model is applied to detect anomalies for all categories without fine-tuning. The best results are bold with black.
Category
De
PEVDn
PaD 16
50
SimpleNer [1]
ParchCore
RDMADDS22
(TRAD[S]
UniAD (1)
Durs
Bottl
64.01 67./6679
85.5/86.7
97.0
96.1
08.7/9188
07.5
07.5/91.2
96.7 96.7/07.7
109 /95.
99.7
0.00
98.3
0.04
Cable
60.17 78.3
64.4/622
70.9
78.2792.3
$7.8
95.
85.0/55.1
07.8/97.1
05.2/97.3
0.29/
98.1
0.04
Cagsule
57.6 85.5
61.3/83.1
734/966
68.3
65.3/5055
8997
96.8
19.0
05.5/96.5
02.0/97.2
/6.5998.5
98.8
Ohga
62.7/766
60.9/96.
63.9.97.4
88.0/848
85.5/96.3
64.9/64.
99.1
98.7/96.9
72.8/62.2
09.2/00
85.1
90.4/94.1
96.3
092/9945
0.07
Pill
80.3
89.4/96.5
6987/87.7
79.7/60.7
/9.2
52.6/96.5
89.7/95.7
03.7/95.0
Sweew
H0.11
74.3
56.9/94.1
75.6/921
92.0/95.5
34.2/983
97.3/99.4
75.1/95.2
K7.5/98.3
98.9 2
Tonbbrush
57.87
86.9
98.0
95.3/95.6
75.3/88.9
99.6/99..7
94.7/97.5
100/98.5
99.4/99.0
89.7/97.5
94.2/%
93.6
98.6 0.04
Transitcc
734/71.7
76.5
99.9
Zipgor
84.3
79.7
07.1
/08.1
0,
63.3/ 78.6
93.8
69.5
98.0/58..
95.9/92.4
97.0/98.
57.1 99.8
90.3/94.4
98.7 0.11
Teture
Grid
69.2/68.5
66.0/ 70.8
73.9
99.3/988
46.7
91.4/ 99.4
99.7 999.
93.9/955
98.2/ 29.3
97.0 0.00
Leather
97.2/95.4
6088/95.
99.9
93.6
96.7
/9.3
92.91 96.9
100 199.2
100
99.8/98.4
190/
0,00 0000/9... 0.002
Tile
93.7/82.7
93.3 68.3/92.1
39.5 6 85.3
99.8
95.77
96.0 /90.3
07. 97.5/95.6
/6.8/91.2
99.3 99.3/91.8
0.52 2
Wood
72.1
98.4/9
93.8/
99.2/960
99.7
0.40/
Mean
745/8118
768//856
84.2 89.5
$1.9 84,9
35.1
/5.1/889
96.4/95.7
93.4/ 96.0
92.6 95.6
96.3 /968
9 0.11 0.05
Normal
Anomaly Recon- Sing Recon Hier
GT
Prod Sing Pred-Hier
Nermal
Anomaly Recon-Sing Recon-tilur
GT
Pred-Sing
Pred-ilice
hazeloat tramsitoor
Figure Qualitative results for anomaly localization on MVTec-AD. 'Recon/Pred-Sing and 'Recon/Pred-Hier are reconstructionssscoree maps with single/hierarchical VQLayer, respectively.
Experiment
4.1 Datasets and Metrics
MVTee-AD 2 a wildly-used industrial anomaly detection dataset with 15 classes, it covers more than 5k high-resolution images including objects and textures. For each class, the training samples are normal while the test samples can be either normal or anomalous. For each anomalous sample, the ground-truths of image label and segmentation are available for evaluation. In this paper, we investigate the unified case following [T], where only one model used to handle all categories.
VisA 45 is recently published large dataset, which consists of 9,621 normal and 1,200 anomalous high-resolution images. The dataset includes images with complex structures, objects placed in sporadic locations, and various types of objects. Anomalies encompass scratches, dents, color spots, cracks, and structural defects. All images are spatially resized to 224 x 224 to facilitate training.
CIFAR-10 [45] is a classical image classification dataset of 10 categories. We adopt the challenging many--vrsiue versus-many setting in [10], where 5 classes are viewed normal while the rest 5 classes are viewed as anomalies that remain unseen during training.
Evaluation metries: We report the Area Under the Receiver Operator Curve (AUROC) on image- level anomaly detection and pixel-wise anomaly localization following the previous works 2 10
4.2 Anomaly detection and localization performance on MVTec-AD
Implementation details: The input image size of MVTec-AD is 224 224 3. after being fed into the pre-trained EfficientNet (20), the feature maps become 14 x 14 x 272, namely, the patch size 16. Then we reduce the channel dimension of each patch into 256, followed by feeding them into 4-layer vanTTrns--ecc followed by the corresponding and 4-layer VQTrans-dec. We use Adam W (53] with weight decay 0,0001 for optimization. Our model is trained for 1000 epochs on 2 GPUs (NVIDIA GeForee RTX 3080 10GB) with batch size 16. The learning rate is initialized as 1 x 10-4 4 and dropped by 0. after 800 epochs. Our model is trained from scratch besides the pre-trained EfficientNet. For more details, please refer to the Appendix.
As shown in Table I the proposed HVQ- Trans generally outperforms all the competitive baselines. Specifically, our model surpasses PatchCore and UniAD by 1.6% and 1.5% on average. The former is a SOTA method under the one- for-one setting, the latter is SOTA method under the one-for-all setting. Especially, in the one-for-all case, our model far exceeds UniAD by 8.5% and 8. 8.1% 1% on Capsule and Screw, respectively. We attribute this to that our model is more robust for identical shortcuts thanks to our well-designed
Quantitative results of anomaly detection on MVTec-AD:
7

---

Table 2: Anomaly detection/location results (image AUROC, pixel AUROC) VisA. Our model is applied all categories without specific parameter-tunin tunnngg on each category.
Category
[0
[NLD
OmniAL 54
UniAD
Ours
PCBI
84.9
/9.9
/976
/5.4/99.3
96.7 99.4
PCB2
81.7/94.1
79.1
81.0/939
93.6 97.8
93.4 98.0
Complex structure
PCB3
87.7/94.1
90.1 / 198.5
/8.1
88.6 98.3
92.0 98.3
PCB4
87.1/72.3
96.2 00.9/97
95.3/97.1
99.4 / 97,9
99.5 / 97.7
Macaroni
68.6 89.8
99.5 /93.3
92.6/98.6
92.2 / 99.3
93.1 99.4
Macaroni 2
60.3/831 83,2
71.3 / 12.1
75.2/97.9
85.9 98.0
86.2 98.5
Multiple instances
Capsules
89.6 96.6
91.4 / 99.6
72.0 98.3 99.6/99.4
77.1 99.0
Candles
70.2/ 82.6
85.4 19455
86.8/95.8
96.8 / 99.2
96.8 99.22
Cashew
68.5
82.5 / 94.1
88.6/95.0
92.4 / 98,7
94.9 / 99.2
Chewing gum
9 98.9 90.0/92.7
/6.4/99.0
99.4 99.2
99.4 98.8
Single instance
Fryum
86.2/ 83,2
91.9 90.00
89.8 97.7 94.6/92.1
90.4 97.7
Pipe fryum
87.1/72.3
87.5 /9..5
86.1 /98.2
97.4 99.2
98.5 99.4
Mean
80.5
877.1 / 95.2
87.8 96.6
91.9 98.6
933.2 98.7
architecture and algorithm. Notably, the performance of various categories are significantly different (Toothbrush), which might due to the data distribution of each category is different, corresponding to different requests for representation ability. The switching mechanism learns individual codebooks and experts for each class, decreasing the distortion between multiple classes. To sum up. our model is proven to be effective and efficient for one-for-all anomaly detection applications.
Quantitative results of anomaly localization MVTec-AD: Anomaly localization aims to detect the anomalous regions given anomalous samples. The localization results under the one-for-all setting are shown in Table As we can see, our model outperforms all the competitive baselines on average. Specifically, as strong SOTA baseline, UniAD is also left behind by our model by 0.5% on both settings averagely speaking. We attribute this that the hierarchical VQ also plays an important role in precise localization besides the learnable query embeddings verified by UniAD since our HVQ-Trans only employs the traditional query embeddings. Moreover, localization requires more precise position information compared with its detection counterpart, a proper measurement alignment to enhance the real anomalous regions may be important, which is exactly what our POT is good at. Another interesting finding is reported in Appendix that although there exists information loss in reconstructing normal images, such information loss even more significant in reconstructing anomalous images. Hence, the actual anomaly localization performance is improved.
Qualitative results of anomaly localization MVTec-AD: As shown in Fig. 4 our method can successfully recover the anomalous regions with their corresponding normal patterns for both object anomalies (Left) and texture damages (Right). It can be seen that the model with hierarchical VQ layers could better generate normal patterns at abnormal regions, resulting in more accurate anomaly localization. More qualitative results are given Appendix.
4.3 Anomaly detection on VisA
Quantitative results on ViSA: Compared to MVTecAD, VisA poses greater difficulty due to its more complex structures and scenes with multiple misaligned instances. Table Sublegun demonstrates the superior performance of our model in comparison to the other three oeconstruction-basee methods under the unified setting. Our proposed model surpasses the best of the comparison methods, i.e., UniAD, by 1.3% inage-AUROC, leading to significantly superior performances than the modest recent unified model OmniAL (44 (5.4% and 2.1% on detection and localization).
Qualitative results on VisA: Figure 5 illustrates the impressive performance of our reconstruction and localization in various categories. Even in multi-instance scenes, Our model effectively restores the anomaly region to its normal state. The reconstructed images exhibit high level of fidelity, closely matching the appearance of normal regions and meeting expectations in recovering anomalies. More qualitative results are given in Appendix.
8

---

Vermal
Anmaay
Becw CalAD
Bexe-Ours
Profdhsib
Pref-turs
Nennal
Aauraly Rece-frdAn Bacoo-raan
GF
Prod-traLD Pred-Ours
Figure 5: Qualitative results for anomaly localization VisA.
Table 3: Anomaly detection results with AUROC metric on CIFAR-10 under the one-for-all setting. Normal indices indicate the names of normal classes. The best results are bold with black.
Normal Indices
US
FCDD 56
FCDD+0E
PANDA
MKD 49
UniAD 10
Ours
(01234)
51.3
55.0
71.8
66.6
64.2
84.4
82.6 0.02
(56789)
51.3
50.3
73.7
73.2
69.3
80.9
84.3 0.03
(02468)
63.9
59.2
85.3
77.1
76.4
93.0
92.4 2 0.10
(13579)
56.8
58.5
85.0
72.9
78.7
90.6
91.9 + 0.06
Mean
55.9
55.8
78.9
72.4
72.1
87.2
87.8 0.04
Table 4: Ablation studies with AUROC metric on MVTec-AD, w/o VQ means without VQ.
with VQ
Switching
w/o VQ
POT
Detection
Localization
Single
Hierarchical
Codebook-Switching Switching
Expert-Switeen
70.5
81.4
96.2
96.8
96.4
96.8
97.1
96,9
97.2
97.0
97.4
97.2
V
97.6
97.2
V
98.0
97.3
4.4 Anomaly detection on CIFAR-10
Implementation details: In order to implement many-versus-mee anomaly detection, we select normal classes while the rest classes are viewed anomalies. As shown in Table 3 (01234) means normal samples include images from classes 2, 3, and 4, while images from 5, 6, 7, and 9 are anomalies. For statistical robustness, we repeat the splitting and obtain four combinations.
Quantitative results on CIFAR-10: shown in Table the performance of our model surpasses all the other competitors under maany-versss-aate setting with each dataset splitting. Besides, CIFAR-10 dataset itself more complex than MVTec-AD because of its poor shooting conditions. Hence, the one-for-all setting on CIFAR-10 10 imposes stricter requirements for the model to exactly distinguish normal patterns from anomalous interference. Therefore, the substantial improvement further verifies the superiority of our method.
4.5 Ablation studies
To verify the effectiveness of the proposed modules, including single VQLayer. hierarchical VQLayers, switching mechanism for codebooks and reconstruction experts, and POT scoring, we implement extensive ablation studies on MVTec-AD. As shown in Table we have the following observations: (i) The performance of the model without VQ drops by nearly 26% (96.4 to 70.5), which demonstrates that VQ plays the key role in anomaly detection and the vanilla Transformer powerful well reconstruct both the normal and anomaly. Our VQ module acts as the information bottleneck where only the normal information is allowed to pass through, thus degrading the reconstruction of anomalous; (ii) The hierarchical structure also presents performance gain since it provides local access to multi-level codebooks, thus reducing the search complexity per layer and releasing the codebook collapse issue; (iii) The switching brings slight improvements on MVTec-AD, while it achieves significant gains up to 3.2% on CIFAR-10, as shown in the Appendix. attribute this to the different degrees of difficulty posed by the two dataset distributions. Our switching mechanism plays a more critical role for the complex datasets, i.e., CIFAR-10 in this case; (iv) The POT module effective in detecting and localizing anomalies due cascade measurement alignment property.
Component study:

---

Table 5: Different hierarchical structures.
Table 6: Different prototype numbers K.
Structure
Detection
Localization
K
Detection
Localization
R +
94.9
96.5
1024
97.1
97.2
97.2
97.0
512
98.0
97.3
1
00
98.0
97.3
256
97.0
97.1
Different hierarchies: We demonstrate experiments to investigate the impact of various hierarchies, as shown in Table [ where the different information is encoded in different layers. While the multi-level features /t' are concatenated with the global prototypes A, the joint performance over anomaly detection and localization increases. One possible explanation is that the global prototypes at much higher abstraction levels may result efficient latent representation for anomaly detection.
Robustness to prototype number: K and show the AUC values in Table Given different numbers of prototypes, our HVQ-Tramm T-Trans can still surpass most of the competitors in Table proving the robustness of our method.
We conduct the experiments by using different prototype numbers
Visualizing how the prototypes works: In order to investigate what exactly the prototypes have learned, we further train a mapping function from the reconstructed feature to the observation space and present the visualization results in Fig- 6 We compulsively instructed the model to reconstruct the patches within the red box using prototypes from the codebook of different, irrelevant category. As demonstrated in the figure, the reconstructed region highly correlated the prototypes, e.g. the centering region in the 'Cable' image is reconstructed as 'Grid', The observations confirm that our iconic prototypes accurately represent typical normal patterns of each category, and only reconstruct the corresponding normal appearances as intended.
Cable Prototypes
Bottle Prototypes
Cable
Reconstruction
Bottle
Reconsrruction
Grid Prototypes
Tile Protolypes
(a) Image of Cable
(b) Image of Bottle
Figure 6: The top row illustrates the images can be well reconstructed with switched prototypes. The bottom row displays when the patches in the red box are forced to be reconstructed with irrelevant prototypes, the reconstructed region accordance with given prototypes.
Conclusion
We introduce unified model, HVQ-Trans for multi-class Unsupervised Anomaly Detection under the one-for-all setting. The latent space is modeled as hierarchical discrete prototypes learned from normal training data. We vector quantize visual features reconstruct normal patterns and employ switching mechanism for codebook selection and exquisite reconstruction. Our hierarchical designation incorporates multi-level normative information and encourages the model to reconstruct anomalous images normal. Furthermore, we propose the hicrarchical prototype-oriente optimal transport module to regulate the prototypes and calibrate the anomaly score. Under the one-for-all setting, our model significantly surpasses competitors on MVTec-AD and VisA datasets, and provides visualization and interpretability for both anomaly localization and detection.
this work, the category labels are assumed to be available during the training stage. How to incorporate the model with clustering methods rather than category labels should be further studied. In practice our model can assemble the normal iconic prototypes which may facilitate domain adaption for real scenes, and be potentially applied time series, text, and video data. However, anomaly detection for video surveillance or social multimedia may raise privacy concerns.
Discussion:
Acknowledgements
This work was supported in part by the National Natural Science Foundation of China (NSFC) under Grant U21B2006: part by the NSFC under Grant 82172860; in part by NSFC under Grant 6220010437; in part by the Shaanxi Youth Innovation Team Project; in part by the Fundamental Research Funds for the Central Universities QTZX23037 and QTZX22166 160;; in part by the Fundation of Acrospace under Grant SAST2021012; in part by the 111 Project under Grant B B18039.
10

---

References
[1] Bharathkumar Ramachandra, Michael J Jones, and Ranga Raju Vatsavai. A survey of single- scene video anomaly detection. IEEE transactions on pattern analysis and machine intelligence, 44(5):2293-22 2020.
[2] Paul Bergmann, Michael Fauser, David Sattlegger, and Carsten Steger. Mvtec ad-a a comprehen- sive real-world dataset for unsupervised anomaly detection. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 9592-960M, 2019.
[3] Tharindu Fernando, Harshala Gammulle, Simon Denman, Sridha Sridharan, and Clinton Fookes. Deep learning for medical anomaly detection-a survey. ACM Computing Surveys (CSUR), 54 (7):1-37, 2021.
[4] Bo Zong, Qi Song, Martin Renqiang Min, Wei Cheng, Cristian Lumezanu, Daeki Cho, and Haifeng Chen. Deep autoencoding gaussian mixture model for unsupervised anomaly detection. In International conference on learning representations, 2018.
[5] Thomas Schlegl, Philipp Seeböck, Sebastian M Waldstein, Ursula Schmidt- Erfurth, and Georg Langs. Unsupervised anomaly detection with generative adversarial networks to guide marker discovery. In Information Processing in Medical Imaging: 25th International Conference, IPMI 2017, Boone, NC, USA, June 25-30, 2017, Proceedings, pages 146-157. Springer, 2017.
[6] Thomas Schlegl, Philipp Seeböck, Sebastian M Waldstein, Georg Langs, and Ursula Schmidt- Erfurth. f-anogan: Fast unsupervised anomaly detection with generative adversarial networks. Medical image analysis, 54:30-44, 2019.
[7] Denis Gudovskiy, Shun Ishizaka, and Kazuki Kozuka. Cflow-ad: Real-time unsupervised anomaly detection with localization via conditional normalizing flows. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 98-107, 2022.
[8] Karsten Roth, Latha Pemula, Joaquin Zepeda, Bernhard Schölkopf, Thomas Brox, and Peter Gehler. Towards total recall in industrial anomaly detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14318-14328, 2022.
[9] Dong Gong, Lingqiao Liu, Vuong Le, Budhaditya Saha, Moussa Reda Mansour, Svetha Venkatesh, and Anton van den Hengel. Memorizing normality to detect anomaly: Memory- augmented deep autoencoder for unsupervised anomaly detection. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1705-1714, 2019.
[10] Zhiyuan You, Lei Cui, Yujun Shen, Kai Yang, Xin Lu, Yu Zheng, and Xinyi Le. A unified model for multi-class anomaly detection. Proceedings of the Thirty-Sixth Annual Conference on Neural Information Processing Systems (NeurIPS 2022), 2022.
[11] Zhiyuan You, Kai Yang, Wenhan Luo, Lei Cui, Yu Zheng, and Xinyi Le. Adtr: Anomaly detection transformer with feature reconstruction. pages 298-310, 2023.
[12] Guansong Pang. Chunhua Shen, Longbing Cao, and Anton Van Den Hengel. Deep learning for anomaly detection: A review. ACM computing surveys (CSUR), 54(2):1-38, 2021.
[13] Lukas Ruff, Jacob R Kauffmann, Robert A Vandermeulen, Grégoire Montavon, Wojciech Samek, Marius Kloft, Thomas G Dietterich, and Klaus-Robert Müller. A unifying review of deep and shallow anomaly detection. Proceedings of the IEEE, 109(5):756-755, 2021.
[14] Wenchao Chen, Long Tian, Bo Chen, Liang Dai, Zhibin Duan, and Mingyuan Zhou. Deep variational graph convolutional recurrent network for multivariate time series anomaly detection. In International Conference on Machine Learning, pages 3621-3633 PMLR, 2022.
[15] Tiange Xiang, Yongyi Lu, Alan L Yuille, Chaoyi Zhang, Weidong Cai, and Zongwei Zhou. In-painting radiography images for unsupervised anomaly detection. arXiv preprint 1rXiv:2111.1344 2021.
[16] Sander Dieleman, Aaron van den Oord, and Karen Simonyan. The challenge of realistic music generation: modelling raw audio at scale. Advances Neural Information Processing Systems, 31, 2018.

---

[17] Mohammad Adiban, Marco Siniscalchi, Kalin Stefanov, and Giampiero Salvi. Hierarchical residual learning based vector quantized variational autoencorder for image reconstruction and generation. In 33rd British Machine Vision Conference, 2022.
[18] Adrian łańcucki, Jan Chorowski, Guillaume Sanchez, Ricard Marxer, Nanxin Chen, Hans JGA Dolfing, Sameer Khurana, Tanel Alumãe, and Antoine Laurent. Robust training of vector quantized bottleneck models. In 2020 International Joint Conference on Neural Networks (IJCNN), pages 1-7. IEEE, 2020.
[19] Diederik P Kingma and Max Welling, Auto-encoding variational bayes. 13 14, 2013.
arXiv preprint
[20] Mingxing Tan and Quoc Le. Efficientnetv2: Smaller models and faster training. In International conference on machine learning, pages 10096-10106. PMLR, 2021.
[21] Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning. Advances in neural information processing systems, 30, 2017.
[22] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, tukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in nearal information processing systems, 30, 2017.
[23] Saeed Masoudnia and Reza Ebrahimpour. Mixture of experts: a literature survey. The Artificial Intelligence Review, 42(2):275, 2014.
[24] Carlos Riquelme, Joan Puigcerver, Basil Mustafa, Maxim Neumann, Rodolphe Jenatton, André Susano Pinto, Daniel Keysers, and Neil Houlsby. Scaling vision with sparse mixture of experts. Advances in Neural Information Processing Systems, 34:8583-8599 2021.
[25] Ruiying Lu, Bo Chen, Jianqiao Sun, Wenchao Chen, Penghui Wang, Yuanwei Chen, Hongwei Liu, and Pramod K Varshney. Heterogeneity-awee recurrent neural network for hyperspectral and multispectral image fusion. IEEE Journal of Selected Topics in Signal Processing, 16(4): 649-665, 2022.
[26] Korawat Tanwisuth, Xinjie Fan, Huangjie Zheng, Shujian Zhang, Hao Zhang, Bo Chen, and Mingyuan Zhou. A prototype-oriented framework for unsupervised domain adaptation. Ad- vances in Neural Information Processing Systems, 34:17194-17208 2021.
[27] Wang, Guo, Zhao, H Zheng, Tanwisuth, Chen, and M Zhou. Representing mixtures of word embeddings with mixtures of topic embeddings. In International Conference on Learning Representations, 2022.
(28) Dandan Guo, Zhuo Li, He Zhao, Mingyuan Zhou, Hongyuan Zha, al. Learning to re-weight examples with optimal transport for imbalanced classification. Advances in Neural Information Processing Systems, 35:25517-25538 2022.
[29] Dandan Guo, Long Tian, He Zhao, Mingyuan Zhou, and Hongyuan Zha. Adaptive distribution calibration for few-shot learning with hierarchical optimal transport. Advances in Neural Information Processing Systems, 35:6996-7010, 2022.
[30] Marco Cuturi. Sinkhorn distances: Lightspeed computation of optimal transport. Advances in neural information processing systems, 26, 2013.
[31] Yoshua Bengio, Nicholas Léonard, and Aaron Courville. Estimating or propagating gradients through stochastic neurons for conditional computation. arXiv preprint arXiv: 1...083333,, 2013.
[32] Ali Razavi, Aaron van den Oord, and Oriol Vinyals. Generating diverse high-fidelity images with vq-vae-2, 2019.
[33] Liyang Chen, Zhiyuan You, Nian Zhang, Juntong Xi, and Xinyi Le. Utrad: Anomaly detection and localization with u-transformer. Neural Networks, 147:53-62, 2022.
12

---

[34] Wenqian Liu, Runze Li, Meng Zheng, Srikrishna Karanam, Ziyan Wu, Bir Bhanu, Richard J Radke, and Octavia Camps. Towards visually explaining variational autoencoders. In Pro- ceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 1642-8651, 2020.
[35] Hyunjong Park. Jongyoun Noh, and Bumsub Ham. Learning memory- normality for anomaly detection. In Proceedings of the IEEE/CVE conference on computer vision and pattern recognition, pages 14372-14388 2020.
[36] Yiru Zhao, Bing Deng, Chen Shen, Yao Liu, Hongtao Lu, and ShannShnng.an Hua. Spatio-temporal autoencoder for video anomaly detection. In Proceedings of the 25th ACM international conference on Multimedia, pages 1943-1941, 2017.
[37] Paul Bergmann, Sindy Löwe, Michael Fauser, David Sattlegger, and Carsten Steger. Improving unsupervised defect segmentation by applying struetural similarity to autoencoders. In Inter- national Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applicaation(ViSiGRRAAl 2019.
[38] Anne-Sophie Collin and Christophe De Vleeschouwer. Improved anomaly detection by training an autoencoder with skip connections on images corrupted with stain-shaped noise. In 2020 25th International Conference on Pattern Recognition (ICPR), pages 7915-7922. IEEE, 2021.
[39] Jinwon An and Sungzoon Cho. Variational autoencoder based anomaly detection using recon- struction probability. Special lecture on IE, 2(1):1-18, 2015.
[40] Haowen Xu, Wenxiao Chen, Nengwen Zhao, Zeyan Li, Jiahao Bu, Zhihan Li, Ying Liu, Youjian Zhao, Dan Pei, Yang Feng, et al. Unsupervised anomaly detection via variational auto-encoder for seasonal kpis in web applications. In Proceedings of the 2018 world wide web conference, pages 187-196, 2018.
[41] Samet Akcay, Amir anomaly detection via adversarial training. Computer Vision-ACCV 2018: 14th Asian Conference on Computer Vision, Perth, Australia, December 2-6, 2018, Revised Selected Pupers, Part III 14, pages 622-637. Springer, 2019.
and Toby P Breckon. Ganomaly: Semi-supervised
[42] Pramuditha Perera, Ramesh Nallapati, and Bing Xiang. Ocgan: One-class novelty detection using gans with constrained latent representations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2898-2906, 2019.
[43] Mohammad Sabokrou, Mohammad Khalooei, Mahmood Fathy, and Ehsan Adeli. Adversarially learned one-class classifier for novelty detection. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3379-3388, 2018
[44] Hui Lv. Chen Chen, Zhen Cui, Chunyan Xu, Yong Li, and Jian Yang. Learning normal dynamics in videos with meta prototype network. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14425-15434 2021.
[45] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
[46] Paul Bergmann, Michael Fauser, David Sattlegger, and Carsten Steger. Uninformed students: Student-teacher anomaly detection with discriminative latent embeddings. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 4183-4192, 2020.
[47] Jihun Yi and Sungroh Yoon. Patch svdd: Patch-level svdd for anomaly detection and segmenta- tion. In Proceedings of the Asian Conference on Computer Vision, 2020.
[48] Thomas Defard, Aleksandr Setkov, Angelique Loesch, and Romaric Audigier. Padim: a patch distribution modeling framework for anomaly detection and localization. In Puttern Recognition. ICPR International Workshops and Challenges: Virtual Event, January 10-15, 2021, Proceedings, Part IV, pages 475-489. Springer, 2021.
13

---

[49] Mohammadreza Salehi, Niousha Sadjadi, Soroosh Baselizadeh, Mohammad H Rohban, and Hamid R Rabice. Multiresolution knowledge distillation for anomaly detection. Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 14902-14912, 2021.
[50] Vitjan Zavrtanik, Matej Kristan, and Danijel Skočaj. Draem-a discriminatively trained re- construction embedding for surface anomaly detection. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 8330-8339, 2021.
[51] Zhikang Liu, Yiming Zhou, Yuansheng Xu, and Zilei Wang. Simplenet: A simple network for image anomaly detection and localization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattenn Recognition, pages 20402-20411, 2023.
[52] Hanqiu Deng and Xingyu Li. Anomaly detection via reverse distillation from one-class embed- ding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Patter Recognition, pages 9737-9746, 2022.
[53] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In Int. Conf. Learn Represent, 2019.
[54] Ying Zhao. Omnial: A unified enn framework for unsupervised anomaly localization. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 3924-3933, 2023.
[55] Ying Zhao. Just noticeable learning for unsupervised anomaly localization and detection. In 2022 IEEE International Conference on Multimedia and Expo (ICME), pages 01-06. IEEE, 2022.
(56) Philipp Liznerski, Lukas Ruff, Robert A Vandermeulen, Billy Joe Franks, Marius Kloft, and Klaus-Robert Müller, Explainable deep one-class classification. Int. Conf. Learn. Represent., 2021.
[57] Tal Reiss, Niv Cohen, Liron Bergman, and Yedid Hoshen. Panda: Adapting pretrained features for anomaly detection and segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattrrn Recognition, pages 2806-28144 2021.
14


Reviews:
[{"review_id": "GtqZ9IoGJC", "reviewer": "Reviewer_JdWV", "summary": "The paper proposes using hierarchical vector-quantized transformer-based autoencoders for image anomaly detection. The key contribution is the use of prototypes learned using an optimal transport algorithm.  ", "questions": "Please see weaknesses.", "limitations": "All the limitations of the proposed method have not been discussed and addressed. The paper does not appear to have any potential negative social impact.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The paper addresses a critical problem in reconstruction-based anomaly detection (good reconstruction of both normal samples and anomalies) and proposes a reasonably solid solution based on discrete latent prototypes.", "weaknesses": "1) The first and foremost problem with the proposed approach is the claim that it is \"unsupervised\". As the paper admits in the final discussion paragraph on page 9, the current work assumes that the category labels are available during the training stage. In that case, how can it be claimed that the proposed method is unsupervised? Also, this may make comparison with unsupervised approaches such as UniAD unfair.\n\n2) The second critical issue is the novelty of the proposed solution because it combines some well-known ideas such as vector-quantized VAE [21] and hierarchical VQ-VAE [17] and some recent ideas such as learning latent prototypes for autoencoders using optimal transport [A, B].\n\n[A] Bie et al., \"Learning Discrete Representation with Optimal Transport Quantized Autoencoders\", 2023\n[B] Oliveira et al., \"Improving Variational Autoencoders Reconstruction Using Prototypes\", 2023.\n\nHowever, in my opinion, the paper smartly combines these known ideas and makes some marginal improvements to implement a transformer-based VAE model.\n\n3) The switching mechanism and mixture-of-experts concepts on page 4 have not been clearly explained. In particular, it is not clear what the so-called \"classifier for producing logits\" and \"expert\" mean and how these components tie in with the POT module described subsequently. Furthermore, the ablation study talks about \"codebook switching\" and \"expert switching\", but these are never described in the paper.\n\n4) Though the paper claims that the proposed method achieves SOTA results, it is not obvious if this is true. Except for UniAD (published in NeurIPS 2002), all the other baseline methods selected for benchmarking are from 2020 or 2021. A cursory glance at more recent works such as [C, D] indicates that the reported performance results fall short of SOTA results.\n\n[C] Liu et al., \"SimpleNet: A Simple Network for Image Anomaly Detection and Localization\" CVPR 2023\n[D] Tien et al., \"Revisiting Reverse Distillation for Anomaly Detection\", CVPR 2023\n\n5) The calibration of anomaly score based on POT has been emphasized many times, but there appears to be no experiment to demonstrates the importance of this idea.\n\n6) It is not clear how to determine the number of prototypes required? Will this depend on the number of classes (or number of anomaly types)? Why does the performance start decreasing when the number of prototypes start increasing beyond 512?\n\n7) There is no mention about the computational complexity of the proposed approach.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "A1: Thanks!\n1) The unsupervised AD methods only utilize the normal images during training stage, while both the abnormal and normal images are utilized at the testing stage. Thus, the 'unsupervised' in AD commonly refers to inaccessible to any abnormal data as supervision at the training stage. \n2) The category labels are accessible in advance and UniAD also claims that ``The category labels may help the model better fit multi-class data. How to incorporate the unified model with category labels should be further studied''. Thus, in this work, we make full use of category information to better serving our model. Furthermore, we replenish an experiment without category information (no switching mechanism), resulting in 97.4 and 97.2 for  anomaly detection and location, which surpass UniAD. \n3) Under the one-for-one case, the category information is accessible to all the comparison models, including UniAD. \nUnder this case, our model surpass UniAD (the SOTA method for unified case) by 0.3\\% and 0.5\\% for anomaly detection and localization.\n\nA2: Our model is problem-oriented and elaborately-designed to alleviate the pivotal `identical shortcut' issue in anomaly detection. \n1) In contrast to most previous methods that model the continuous latent space and suffer from good generalizability to anomalies, we aim to model discrete space to intrinsically prevent anomaly information leakage into reconstruction. However, the existing memory-based methods recombine and aggregate the discrete memory items, falling into an unknown continuous latent space which might be distorted. In contrast, we force the anomaly features to be replaced by a single discrete prototype. Therefore, VQ is a proper pathway to optimise prototypes for crimping information bottleneck. Although VQ-VAE is a well-known technique, it rightly fits our motivation. As far as we know, this is the first try to impose strict restrictions on discrete latent space, and it is the first time to verify the validity of VQ for anomaly detection. \n2) Moreover, the optimal transport learning is not only developed to facilitate prototype learning as the previous works, but also dexterously measure the feature level anomaly score to robustly and accurately identify anomalies. \n3) The hierarchical structure is originally designed by merging fine-grained concrete information with abstraction-level semantics, which could avoid the issue of codebook collapse, as shown in Table 5. \n4) In addition, our model targets for the challenging 'one-for-all' case, which suffers from the 'identical shortcut' issue more severely, as the model generalizability increases due to the complex data distribution of multiple classes. To tightly fit the unified case, we propose the switching mechanism to choose corresponding codebook and expert for different data distribution, thus alleviating the `identical shortcut' issue.\n\nA3: \n1) The switching mechanism contains a multi-category classifier, $M$ codebooks, and $M$ reconstruction experts. The multi-category classifier takes the image feature as input and output the classification probability (logit) over $M$ category. In order to fit the data diversity property in one-for-all setting, we switch individual codebook (including a group of prototypes) from $M$ codebooks, according to the classification probability. Furthermore, we switch individual reconstruction network (expert) for decoding features from the last decoder layer to re-build the input features from the pre-trained EfficientNet. \n2) As for POT, it is used to regularize the relationships between features and vector-quantized prototypes, making them better matched compared with Euclidean or Cosine distance used in original VQ-VAE. Therefore, POT only takes effects with features at hierarchical layers.\n3) 'codebook switching' and 'expert switching' all refer to the action that classifier choose corresponding codebook or reconstruction network.\n\nA4: We replenish the experiments compared to the mentioned methods in Table 1 of the uploaded PDF. The results show that we achieve superior performances. \n\nA5: As we reported it in Table 4 of the ablation study, we can see the POT module leads to 0.4\\% and 0.1\\% performance gain on anomaly detection and localization, respectively, on MVTec-AD dataset. Furthermore, as shown in Table 1 in appendix, the POT module results in 2.6\\% improvements on CIFAR-10 dataset. This demonstrate the POT module is effective to detect anomalies due to its well-established measurement alignment, especially showing the stability on the complex scenarios of CIFAR-10.\n \nA6: The prototypes are grouped into $M$ categories. We only decide the number of prototypes $K$ per group, which is related to the complexity of the data distribution. For the category of texture or simple object, such as bottle and grid, the iconic normal patterns are limited. The corresponding number of prototypes can be set smaller than those complex objects.\n\nWithout exhausted parameter tuning for individual category, we set the same number of prototypes for each category. We found that 512 is an empirically proper setting, which performs averagely decent on all the 15 categories. This might because: i) The smaller number of prototypes can't cover all the iconic normal patterns for the complex objects, leads to poor reconstruction ability; ii) Overmuch prototypes will cause the abundance and repetition of iconic normal patterns, which is harmful to model training and disturbs the precise vector quantization. Specifically, redundant prototypes may never be used and optimized during training. At inference, those non-optimized prototypes may be closer with anomaly patterns, leading to leakage of abnormal information, and finally resulting in poor anomaly detection. Moreover, it's worth noting that different numbers of prototypes can consistently surpass all the competitors in Table 1 and Table 2. \n\nA7: We replenish the computational complexity in Table  2 of the uploaded PDF.\n"}, {"review_id": "lHeFoIeQbZ", "reviewer": "Reviewer_Ah19", "summary": "This paper introduces a novel approach to multi-class anomaly detection (AD) by integrating hierarchical embedding vector quantization. To tackle the problem of identical shortcuts in the reconstruction-based AD paradigm, the authors suggest to enlarge abnormality's reconstruction residue by introducing discrete prototypes in the model. Additionally, the model incorporates a switching mechanism to further improve the feature reconstruction process. The proposed model is evaluated on MVTec and CIFAR-10 datasets and compared against prior arts.", "questions": "Please refer to the weakness section. Thanks.", "limitations": "No discussion on limitations and potential negative social impact.", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper is well-written and easy to follow.\n\n2. The hierarchical vector quantized transformer is well presented. Appendix about the loss back-propagation explains parameter updates regardless quantization operation.\n\n3. Extensive experiments are conducted. Ablation studies on model components suggest effectiveness of the proposed method.", "weaknesses": "1. The vector quantization mechanism described in this paper appears to be a specific instance of the memory mechanism, where a continuous-valued feature is substituted with a numerical prototype. If this is indeed the case, such a memory mechanism has been widely employed in previous studies on AD.\n\n2. What does 'C' represent in Figure 2 and Figure 3? Is it indicating feature concatenation? The paper briefly mentions feature aggregation but does not provide a specific definition or explanation of the feature aggregation operation.\n\n3. Within the switching mechanism, a multi-category classifier, N codebooks, and reconstruction experts are necessary. In this scenario, can we consider the model as a combination of a single encoder and N decoders that are specific to each class?\n\n4. The experimental evaluation compares the proposed method with US, PSVDD, PaDim, CuPaste, MKD, and DREAM. However, there exist several recently proposed AD models, such as Patchcore [1], RD4AD [2], CS-Flow [3], UTRAD [4], which outperform the aforementioned methods. It is highly recommended to include these methods in the experimental analysis as well.\n\n[1] Karsten Roth et al. “Towards total recall in industrial anomaly detection”. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022, pp. 14318– 14328.\n[2] Hanqiu Deng and Xingyu Li. “Anomaly Detection via Reverse Distillation from One-Class Embedding”. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022, pp. 9737–9746.\n[3] Marco Rudolph et al. “Fully convolutional cross-scale-flows for image-based defect detection”. Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision. 2022, pp. 1088–1097.\n[4] Liyang Chen et al. “UTRAD: Anomaly detection and localization with U-Transformer”. Neural Networks 147 (2022), pp. 53–62.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks a lot for your positive comments! We have tried our best to address the mentioned concerns/problems. Feel free to let us know if there is anything unclear or so. We are happy to clarify them.\n\nA1: To some extent, our vector-quantized prototype can be regarded as a special kind of memory item.\nAs the reviewer mentioned, a branch of approaches~\\cite{roth2022towards,gong2019memorizing,xiang2021painting} have investigated the memory-augmented networks recently, which augment the deep autoencoder with a memory module to record the normal information of training data. These methods hope to obtain low reconstruction error for normal samples and highlight the reconstruction error if the input is not similar to normal data, that is, an anomaly. The relevant memory items are retrieved and weighted averaging all the related memory content are aggregated into the decoder for reconstruction. \n\nThus, we claim three differences between our proposed method with the previous methods:\n1) The discrete memory items are recombined and weighted averaged in previous works, falling into an unknown continuous latent space that might be distorted. Intuitively, some anomalous regions can not be reconstructed by the discrete latent memory but could be decoded from the unknown latent space. In contrast, we set a strict information bottleneck to enforce the abnormal data point flipping to a normal data point, constraining the leakage of abnormal information.\n2) The prototypes are individually learned for each category, and adaptively chosen by a switching mechanism. This mechanism tightly fits the unified case, which suffers from the `identical shortcut' issue more severely, as the model generalizability increases due to the complex data distribution of multiple classes. We choose specific codebooks for different data distributions to alleviate the 'identical shortcut' issue.\n3) A hierarchical VQ-based approach is developed to better overcome the \"codebook collapse\" problem and effectively merge fine-grained concrete information with abstract semantics to maximize the nominal information available.\n4) In addition, we introduce the POT module to guide the learning process of prototypes with the help of OT theory, to facilitate prototype learning and dexterously measure the feature level anomaly score. On the contrary, the existing memory-based methods always use cosine similarity or Euclidean distance to learn memory vectors. \n\nA2: Yes, the `C' represents the feature concatenation. \n\nThe feature aggregation includes two steps: Firstly, the visual tokens are concatenated with the global quantized vector $\\theta$; Secondly, a layer-wise embedding function $\\Upsilon^l(\\cdot)$ is utilized for feature fusion. Then the Quantization is performed on the aggregated features.\nSpecifically, we fuse the multi-level visual tokens $h^{l-1}$ with the global quantized vector $\\theta$ to learn hierarchical prototypes, maximizing the preserved nominal information, stated as $\\texttt{Quantize}(\\Upsilon^l\\left(\\left[ h^{l-1}, \\theta \\right]\\right)) = e_k^l$. Here, $\\left[ \\cdot \\right]$ denotes the concatenation operation, and $\\Upsilon^l(\\cdot)$ refers to the embedding function. We replace the fused feature with its most similar prototype $e_k^l$ in the codebook as the corresponding quantized vector.\n\nA3: The switching mechanism did contain a multi-category classifier, N codebooks, and N reconstruction experts. However, the reconstruction experts are only the final part of our HVQ-Transformer decoder.\n\nAs shown in Figure 2 in the main paper, the HVQ-Transformer has four graded decode layers. In each decoder layer, the refined queries $q^l$ from the previous layer are crossly connected to the quantized normal prototypes $z^l$, through the multi-head cross-attention layer. Hence, the values at abnormal regions of $q^l$ will be suppressed, and the abnormal signals could be rarely transmitted for reconstruction. The switching reconstruction experts are placed after the four decoder layers, for flexible feature reconstruction. Specifically, the visual tokens from the last \\textit{VQTrans-dec} layer are depicted as $d^0 \\in \\mathbb{R}^{N \\times C}$, which is expected to reconstruct the patch features as $\\Psi_m(d^0)$,\nwhere $m^{th}$ expert network $\\Psi_m$ is selected for reconstruction.\n\nA4: We replenish the experiments compared to all the mentioned methods, as shown in Table 1 of the uploaded PDF. The results show that we achieve superior performances. \n\nLimitations: The limitation and potential negative social impact are presented in the 'discussion' part of the conclusion section.\n "}, {"review_id": "jCHXQ0bLPI", "reviewer": "Reviewer_XjaL", "summary": "This paper proposes a feature reconstruction based framework for multi-class anomaly detection, called hierarchical vector quantized Transformer (HVQ-Trans). To address the \"identical shortcut\" problem occurring in the reconstruction-based framework, the proposed method replaces the original encoding features with the nearest iconic prototypes learned from normal training data, and then decoded with a VQ-based transformer decoder to reconstruct the anomaly regions into normal regions. Besides, a hierarchical prototype-oriented learning and anomaly scoring method is developed to guide prototype learning and accurately identify anomalies.\n\n\n", "questions": "see the weaknesses", "limitations": "see the weaknesses", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The motivation is clear and reasonable\n2. The writing quality and paper structure are good.\n3. The idea of using vector quantization for feature reconstruction in anomaly detection is interesting. The proposed method is technically sound.\n4. The proposed method obtains decent performance improvement than the current methods.", "weaknesses": "1. Using prototypes may lose high-frequency information leading to imprecise feature reconstruction.\n\n2. Is the optimal transport necessary? How about only simply using similarity scores as the weights?\n\n3. The proposed HVQ-Trans has large model parameters. Please compare the proposed method with UniAD with the inference speed and model parameters.\n\n4. Please add more visualization comparisons with related methods, such as UniAD.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks a lot for your positive comments for this submission! We have tried our best to address the mentioned concerns in the rebuttal. Feel free to let us know if there is anything unclear or so. We are happy to clarify them.\n\nA1: Yes. We want to emphasize that the different degrees of imprecise reconstruction between normal and anomaly are the key factor in anomaly detection. In other words, we pursue the difference in reconstruction ability between normal and anomaly, rather than imprecisely reconstruct them.\n\nIn specific, during training, the typical normal patterns are recorded in the discrete variables, i.e., iconic prototypes. When encountering the anomalous during the testing stage, the abnormal patterns will also be quantized as the normal prototypes, leading to larger feature migration and information loss, highlighted by higher reconstruction error. It is worth noting that while information loss triggered by VQ is exist for normal images, it is further significantly pronounced for anomaly images. Thus, this discrepancy in information loss serves as a key factor in effective anomaly detection. By investigating this difference, we can enhance the accuracy of our model in distinguishing abnormal regions.\n\nA2: We employ optimal transport (OT) for better exploring relationships between learnable prototypes and visual features of input images. Specifically, traditionally used similarity scores such as Euclidean or Cosine only concern the point-wise relationship between two sets and pose equally important prior to any such kind of relationship. In practice, however, it is usually sub-optimal to introduce such non-informative prior and leads to poor performance. In our model, optimal transport learning is not only developed to facilitate prototype learning with the regularization on the feature distribution level, but also dexterously measure the feature level anomaly score to robustly and accurately identify anomalies.\n\nMoreover, as we reported in Table 4 of the main paper, we can see the POT module leads to 0.4\\% and 0.1\\% performance gain on anomaly detection and localization, respectively, on the MVTec-AD dataset. \nFurthermore, as shown in Table 1 in the appendix, the POT module results in 2.6\\% improvements on the CIFAR-10 dataset. This demonstrates the POT module is effective to detect anomalies due to its well-established measurement alignment, especially showing the stability for the complex scenarios of CIFAR-10. \n\nA3: The comparison results are listed in Table 2 of the uploaded PDF. With the image size fixed as $224 \\times 224$, we compare our model with all competitors regarding the inference FLOPs and learnable parameters. We can tell that the advantage of our approach does not come from a larger model capacity. This table will be included in the camera ready.\n\nA4: The visualization comparisons with UniAD are shown in the unloaded PDF. More visualization comparisons will be added to the appendix of our revised paper."}, {"review_id": "PtYHDar2HE", "reviewer": "Reviewer_ksQL", "summary": "This paper proposes a variational autoencoding framework for unsupervised anomaly detection. This work addresses the identical shortcut issue by preserving the typical normal patterns as discrete iconic prototypes and also overcomes the problems of prototype collapse problem.", "questions": "Why the proposed methods under one-for-one setting has lower performance than the one-for-all setting, which differs from those in most other methods? What is the reason for the significant performance differences of the proposed method on different categories, such as poor performance on “Toothbrush”, and is this explainable? ", "limitations": "The authors have clarified the limitation of this method that the category labels are assumed to be available during the training stage， and identify this issue as future study by incorporating the model with clustering methods.", "rating": 3, "confidence": 4, "soundness": 1, "presentation": 2, "contribution": 1, "strengths": "It designs a network from the perspective of solving an issue identical shortcut. The method achieves good experimental results on classical datasets.", "weaknesses": "1. The descriptions of the figures need to be clearer. For example, in Fig.1, it would be helpful to label each image and coordinate system to indicate what they represent. Additionally, labeling the subfigures with (a) and (b) would make the explanation clearer, such as in Fig.5.\n2. The performance on chosen datasets are close to saturation, especially in some categories where the results achieve 100, this is unreasonable. In this field, more challenging datasets should be tested to further push the boundaries of research.\n3. The performance of proposed method under one-for-one setting still has a gap compared to the SOTA method. \n4. The innovation is limited, where most of the ideas have already be proposed. \n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "A1: Thanks for your suggestions. We have revised the descriptions and figures to be clearer. Due to the page limitation, we show revised Fig. 1 and Fig. 5 in the uploaded PDF. We promise to carefully polish all the descriptions and figures in our revised version.\n\nA2: Thanks for your suggestion.\n\n1) As the reviewer mentioned, the performance on the wildly-utilized industrial anomaly detection dataset MVTec-AD tends to saturation. This might be because the images in the dataset have simple backgrounds and high resolution, thus the anomalies are relatively easy to be detected. In contrast, our model targets for the challenging ‘one-for-all’ case, which\nsuffers from the ‘identical shortcut’ issue more severely, as the model generalizability increases due to the complex data distribution of multiple classes. The performance under the ‘one-for-all’ setting on MVTec-AD still needs to be improved.\n\n2) It's worth noting that we also demonstrate experiments on a challenging dataset, \\ie, CIFAR-10, which contains complex background and various anomalies. \nOur model achieves 86.1, while the comparison models result in \\{55.9, 55.8, 78.9, 72.4, 72.1, 82.1\\} for anomaly detection.\n\n3) According to the reviewer's suggestion, we have further implemented experiments on another industrial anomaly detection dataset, the VisA dataset. The performance is shown in Table 4 of the unloaded PDF. As we can see, our model surpasses the previous SOTA method, UniAD, by 1.3\\% for anomaly detection on the new dataset. We will add the results on the VisA dataset to the revised manuscript.\n\nA3: We hope to address this concern from the following aspects:\n\n1. Focusing on the one-for-all model handling the one-for-one task, our model surpasses UniAD (the SOTA method for unified case) from 96.6 to 96.9 for anomaly detection, and improves the anomaly localization performance from 96.6 to 97.1. Thus, under this fair comparison setting, our model could better generalize to the one-for-one setting.\n\n2. Without exhausted individual parameter tuning for each category, we set the same hyperparameters for each category, which might lead to a gap. For example, the category of texture or simple objects, such as bottle and grid, possess limited iconic normal patterns. The corresponding number of prototypes might be smaller than those complex objects. Rather than specific and exhausted model tuning for each category, our network architecture is simply implemented under the one-for-one setting. Accordingly, we particularly tune the hyperparameters on the 'Screw' category in the MVTec-AD dataset, and result in 94.5\\% performance, surpassing all the comparison methods. \nBesides, although DRAEM (the SOTA one-for-one method) achieves good performances under the separate case, it drops severely (9.9\\% and 10.1\\%) changing to the unified case. Thus, on the whole, our model achieves decent performance in both settings.\n\n3. Last but not least, we want to reclaim that our purpose is to alleviate the pivotal ‘identical shortcut’ issue in anomaly detection under the challenging one-for-all setting. When changing the one-for-one setting to the one-for-all setting, most of the existing alternatives fail to handle the challenging cases. Apart from the UniAD getting a slight drop (0.1\\%) for detection, the other methods drop severally (12.4\\% on average) when changing to the one-for-all setting.\nHowever, our model gets a 1.1\\% performance gain for anomaly detection. This indicates the framework of UniAD and our model could handle the challenging setting. Thus, instead of saying there is a gap compared to the SOTA method on the separate case, what we want to express is that there is no performance drop (even a slight improvement) from the separate case to the unified case.\n\nA4: We think there are two main reasons causing the better performance under the one-for-all setting: \n\n1) The amount of training data is bigger and the training data diversity is increased. Thus, the model representation ability can be improved. Furthermore, diverse training data force the model to learn robust and discriminative representations and separate from each category, leading to tighter boundaries modeled by prototypes of each category.\n\n2) The switching mechanism under the one-for-all setting will classify each input image into a single category and choose the corresponding codebook and expert for reconstruction. For the normal images, it is highly likely to be classified into the correct category and thus switch the proper reconstruction expert and codebook. For the abnormal images, there remains big uncertainty that which reconstruction expert and codebook will be switched, because the anomalies are unseen during training. Thus, the reconstruction uncertainty of the abnormal image is increased. \nNoting that the difference between normal and abnormal is the key factor deciding the anomaly detection performance. Thus, this uncertainty of anomalous could improve the performance under the one-for-all setting, which also proves that the switching mechanism of our proposed method is more suitable for multi-class anomaly detection.\n\nThe reason for the significant performance differences:\n\nThe performance differences in different categories are due to: \ni) the data distribution of each category is different, corresponding to different requests for model representation ability. \nii) our network architecture is simply implemented without many tricks, such as hyperparameter tuning for each category (\\eg. prototype numbers, latent dimension) or adaptive architecture for individual categories. On the whole, it gets efficient performances under both the one-for-all and one-for-one settings. \nHowever, this insightful observation refers to a common phenomenon for all those one-for-all methods, please refer to UniAD. Even for the one-for-one method, such as the well-known CutPaste, its performance on cable is only 81.2 due to the complex and noisy data distribution.\n\n"}, {"review_id": "6Bx0pJyxtE", "reviewer": "Reviewer_uU2U", "summary": "This work builds a unified framework for multi-class anomaly detection (AD) by using normal images only. It identifies the identical shortcut issue of the reconstruction-based AD methods and tries to alleviate it by augmenting the memory with hierarchical discrete iconic prototypes. A switching mechanism is employed to deal with multi-class scenario. Furthermore, the hierarchical prototype-oriented optimal transport module is used to calibrate the anomaly scores. Experiments on MvTec-AD and CIFAR-10 demonstrate the superiority of the method.", "questions": "1. In Table 1, why the results under one-for-all setting are better than that under one-for-one setting? For example, the results on Capsule is far lower under one-for-one setting. Though the authors claim that the increased data diversity is beneficial, it requires more experiments to support the argument, e.g., try taking images from different N (N >= 2) classes for training.\n2. The ablation study is incomplete. Missing the results when only Hierarchical VQ is enabled.", "limitations": "Limitations are discussed in the Discussion section and potential negative societal impacts are not mentioned.", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The motivation of using discrete iconic prototypes makes sense to alleviate the “shortcut” of reconstruction on abnormal images.\n2. It achieves state-of-the-art performance on the two popular benchmarks.\n3. The paper is easy-to-follow and well-written with clear motivation for each module.", "weaknesses": "1. Missing necessary comparison on additional datasets, e.g., the VisA dataset [1], a multi-class industrial anomaly detection dataset. \n2. As a core contribution, the details of vector quantization (VQ) (e.g., the quantization method) are not thoroughly presented and some related questions are not discussed, e.g., how different quantization precisions/methods affect the performance?\n\n[1] Zou, Yang, Jongheon Jeong, Latha Pemula, Dongqing Zhang, and Onkar Dabeer. \"Spot-the-difference self-supervised pre-training for anomaly detection and segmentation.\" In European Conference on Computer Vision, 2022.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks a lot for your effort in reviewing! \n\nA1: According to the reviewer's suggestion, we have implemented experiments on the VisA dataset. The performance is shown in Table 4 of the unloaded PDF. As we can see, our model surpasses the previous one-for-all SOTA method, UniAD, by 1.3\\% and 0.1\\% for anomaly detection and localization, respectively. \nMoreover, our model surpasses the previous one-for-one SOTA method, DRAEM, by 12.7\\% and 11.7\\% for anomaly detection and localization. This demonstrates the effectiveness and robustness on VisA dataset, which is more challenging than the MVTec-AD dataset.\n\nWe will add the results on the VisA dataset to the revised manuscript.\n\nA2: The hierarchical VQ-based layers layer-wisely quantize the visual tokens $h^l$ to the prototypes $e_k^l$ in the learnable codebooks $E^l \\in R^{K \\times C}$.\nFor the output $h^L$ final layer of encoder, we replace the visual tokens $h^L$ with its most similar prototypes $e_i^L$ in the codebook $E^L$ as: $ \\theta = Quantize(h^L)=e_i^L,  i= {min}_{j} \\| h^L - e^L_j\\|_2^2,$ where $\\theta$ represents the global quantized vector. The visual token is quantized based on its distance to the prototype vectors in the codebook $E^L$, such that each visual token is replaced by the nearest prototype vector in the codebook, and is transmitted to the decoder.\n\nMoreover, we find that merging fine-grained concrete information with abstraction-level semantics is critical for robust anomaly detection. Hence, we fuse the multi-level visual tokens with the global quantized vector $\\theta$ to learn hierarchical prototypes, maximizing the preserved nominal information, stated as equation 1 in the main paper. Intuitively, we hierarchically replace visual tokens $h^l$ with their most similar prototypes in the codebook $E^l$ as quantized vector $z^l$. \n\nNote that there is no real gradient defined for quantization, however we approximate the gradient similar to the straight-through estimator and just copy gradients from decoder input $z^l$ to embedding before quantization. One could also use the subgradient through the quantisation operation, but this simple estimator worked well for the initial experiments in this paper. \n\nWe promise to add detailed and clear descriptions of vector quantization in the revised paper.\n\n\\textbf{Affection of VQ:} As shown in Table 4, the vanilla Transformer obtain 70.5 and 81.4 on AD and AL, while increasing to 96.4 and 96.8 by adding the VQ-layers.\nThe performance of the model increases by nearly 26\\%, which demonstrates that VQ plays the key role in anomaly detection. Our VQ module acts as the information bottleneck where only the normal information is allowed to pass through, leading to larger feature migration and information loss for anomalies.\nThis discrepancy of information loss for normal and anomaly serves as a key factor in effective anomaly detection.\n\nA1-to-Q1: We think there are two main reasons causing the better performance under the one-for-all setting: \n\n1) The amount of training data is bigger and the training data diversity is increased. Thus, the model representation ability can be improved. Furthermore, diverse training data force the model to learn robust and discriminative representations and separate from each category, leading to tighter boundaries modeled by prototypes of each category.\nWe have replenished experiment on smaller $N$ in Table 5 of the uploaded file. Specifically, we randomly choose 5 categories from all the 15 categories of MVTec-AD for 3 times without overlap, namely $N=5$ in this case. We find that the mean AUCs of detection and localization at this moment are 97.6\\% and 97.3\\%, respectively. Compared with our model with $N=15$, the detection performance drops 0.4\\%, thus it could verify the benefits of training data diversity to some extent.\n\n2) The switching mechanism under the one-for-all setting will classify each input image into a single category and choose the corresponding codebook and expert for reconstruction. For the normal images, it is highly likely to be classified into the correct category and thus switch the proper reconstruction expert and codebook. For the abnormal images, there remains big uncertainty that which reconstruction expert and codebook will be switched, because the anomalies are unseen during training. Thus, the reconstruction uncertainty of the abnormal image is increased. Noting that the difference between normal and abnormal is the key factor deciding the anomaly detection performance. Thus, this uncertainty of anomalous could improve the performance under the one-for-all setting, which also proves that the switching mechanism of our proposed method is more suitable for multi-class anomaly detection.\n\n3) The data distribution of each category is different, corresponding to different requests for model representation ability. It's worth noting that our network architecture is simply implemented without many tricks, such as hyperparameter tuning for each category (\\eg. prototype numbers, latent dimension) or adaptive architecture for individual categories. After tuning parameters on the Capsule category, we improve the performance from 88.3 to 94.7 on this category. However, the original intention\nof the one-for-all case is to save computational resources and achieve efficient modeling for all the category simultaneously.\nOn the whole, our model can efficiently get decent performances under both the one-for-all and one-for-one settings. \n\nA2-to-Q2: We have replenished extra ablation studies in Fig. 5 of the uploaded file. As we can see that only employing Hierarchical VQ achieves 97.1 \\% and 96.9 \\% in detection and location, achieving 26.6\\% and 15.5\\% compared with the vanilla Transformer. The results can verify that the main performance gain come from our proposed hierarchical VQ-based framework.\n\nPotential negative societal impacts: Anomaly detection for video surveillance or social multimedia may raise privacy concerns.\n"}]

Author rebuttal:
Dear reviewers and AC

Thanks a lot for your effort in reviewing this submission! We have tried our best to address the mentioned concerns/problems in the rebuttal. Feel free to let us know if there is anything unclear or so. We are happy to clarify them.

Best, Authors

Meta-review: The paper introduces an approach based on hierarchical vector-quantized transformer autoencoders for image anomaly detection. The manuscript received ratings of: borderline accept, borderline accept, reject, borderline accept and borderline accept. While the reviewers appreciated the motivation, idea and paper writing, they also raised some concerns such as, lack of clarity regarding the switching mechanism, comparison with more recent AD models, parameter comparison and performance under one-for-one setting and one-for-all setting. Authors responded to reviewer's queries by submitting a rebuttal. The rebuttal largely addressed the initial concerns raised by the reviewers. While four reviewers were on the positive side, one reviewer remained on the negative side. Given that four reviewers are generally positive about the paper as well as author's rebuttal (specifically justifying the performance comparison under one-for-all setting), the recommendation is accept. Authors are strongly encouraged to take into account the suggestions of the reviewers as well as changes acknowledged in the rebuttal when preparing the final manuscript.
Decision: Accept (poster)