A Unified Model for Multi-class Anomaly Detection
Authors: Zhiyuan You
Lei Cui
Yujun Shen
Kai Yang
Xin Lu
Yu Zheng
Xinyi Le
Venue: NeurIPS 2022 Accept (2022)
Keywords: multi-class anomaly detection

Abstract: Despite the rapid advance of unsupervised anomaly detection, existing methods require to train separate models for different objects. In this work, we present UniAD that accomplishes anomaly detection for multiple classes with a unified framework. Under such a challenging setting, popular reconstruction networks may fall into an "identical shortcut", where both normal and anomalous samples can be well recovered, and hence fail to spot outliers. To tackle this obstacle, we make three improvements. First, we revisit the formulations of fully-connected layer, convolutional layer, as well as attention layer, and confirm the important role of query embedding (i.e., within attention layer) in preventing the network from learning the shortcut. We therefore come up with a layer-wise query decoder to help model the multi-class distribution. Second, we employ a neighbor masked attention module to further avoid the information leak from the input feature to the reconstructed output feature. Third, we propose a feature jittering strategy that urges the model to recover the correct message even with noisy inputs. We evaluate our algorithm on MVTec-AD and CIFAR-10 datasets, where we surpass the state-of-the-art alternatives by a sufficiently large margin. For example, when learning a unified model for 15 categories in MVTec-AD, we surpass the second competitor on the tasks of both anomaly detection (from 88.1% to 96.5%) and anomaly localization (from 89.5% to 96.8%). Code is available at https://github.com/zhiyuanyou/UniAD.

A Unified Model for Multi-class Anomaly Detection
Zhiyuan You'- Lei Cui
Yujun Shen?
Kai Yang" Xin Lu
Yu Zheng' Xinyi Le
Shanghai Jiao Tong University
TTiinghua University SCUHK *SenseTime
shiyuanyouffoxmail com, com, cuil190mails tsinghua edu. en, shenyujun03028gmail com. (yangkai,
com, yuzhengosjtu edu. en, lexinyidsjtu, edu.
Abstract
Despite the rapid advance of unsupervised anomaly detection, existing methods require to train separate models for different objects. this work, we present UniAD that accomplishes anomaly detection for multiple classes with a unified framework. Under such challenging setting, popular reconstruction networks may fall into an "identical shortcut", where both normal and anomalous samples can be well recovered, and hence fail to spot outliers. To tackle this obstacle, we make three improvements. First, we revisit the formulations of fully-com fully-comeected layer, convolutional layer, as well as attention layer, and confirm the important role of query embedding (i.e., within attention layer) in preventing the network from learning the shortcut. We therefore come up with a layer-wise query decoder to employ neighbor masked
help model the multi-class distribution. Second, atlention module to further avoid the information leak from the input feature to the reconstructed output feature. Third, we propose a feature jittering strategy that urges the model to recover the correct message even with noisy inputs. We evaluate our algorithm MVTec-AD and CIFAR- 10 datasets, where we surpass the fate--of-the-art alternatives by a sufficiently large margin. For example, when learning unified model for 15 categories MVTec-AD, we surpass the second competitor on the tasks of both anomaly detection (from 88. 18.1% to 96.5%) and anomaly localization (from 89.5% 96.8%). Code is available at https: :// github. com cww/zhiyuanyou w2hiyuayoo/UnnAD
1 Introduction
Anomaly detection has found an increasingly wide utilization in manufacturing defect detection [4], medical image analysis [17], and video surveillance [46]. Considering the highly diverse anomaly types, common solution is to model the distribution of normal samples and then identify anomalous ones via finding outliers. It is therefore crucial learn a compact boundary for normal data, as shown in Fig. la. For this purpose, existing methods [6. 11, 25, 27, 48, 49, 52] propose to train separate models for different classes objects, like in Fig. Ic. However, such one-class-one-mene scheme could be memory-ccnsumee also uncongenial the scenarios where the normal samples manifest themselves in a large intra-class diversity (i.e., one object consists of various types).
especially along with the number of classes increasing, and
In this work, target more practical task, which is detect anomalies from different object classes with a unified framework. The task setting is illustrated in Fig. 1d, where the training data covers normal samples from range of categories, and the learned model is asked accomplish anomaly detection for all these categories without any fine-tuning It is noteworthy that the categorical information (i.e., class label) inaccessible both the training and the inference stages, considerably
Contribute Equally.
Corresponding Author.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Normal
Anomaly
Boundary
Train on Normal Data
Infer to Detect Anomalies
Class
Class 2
Net 1
Net 2
Net
Net
(a)
e (b)
e
(d)
Figure 1: Task setting of unified anomaly detection. (a) Existing methods learn separate decision boundaries for different object classes, while (b) our approach models the multi-class data distribution such that one boundary is enough to spot outliers regarding all categories. As result, we escape from the conventional one-class-om sas--one-ooeed paradigm in (c), and manage to accomplish anomaly detection for various classes with a unified framework in (d).
easing the difficulty of data preparation. Nonetheless, solving such a task is fairly challenging. Recall that the rationale behind unsupervised anomaly detection model the distribution of normal data and find a compact decision boundary as in Fig. la. When it comes to the multi-class case, we expect the model capture the distribution of all classes simultaneously such that they can share the same boundary as in Fig, lb. But if we focus on particular category, say the green one in Fig, 1b, all the samples from other categories should be considered as anomalies no matter whether they are normal (i.e., blue circles) or anomalous (i.e., blue triangles) themselves. From this perspective, how to accurately model the multi-class distribution becomes vital.
widely used approach to learning the normal data distribution draws support from image (or feature) reconstruction [2, 5, 26, 39, 51], which assumes that well-trained model always produces normal samples regardless of the defects within the inputs. In this way, there will be large reconstruction errors for anomalous samples, making them distinguishable from the normal ones. However, find that popular reconstruction networks suggest unsatisfying performance on the challenging task studied in this work. They typically fall into an "identity shortcut", which appears returning direct copy of the input disregarding its content." As a result, even anomalous samples can be well recovered with the learned model and hence become hard to detect. Moreover, under the unified case, where the distribution of normal data is more complex, the "identical shortcut" problem is magnified. Intuitively, to learn a unified model that can reconstruct all kinds of objects, it requires the model to work extremely hard to learn the joint distribution. From this perspective, learning an "identical shortcut" appears as a far easier solution.
To address this issue, we carefully tailor a feature reconstruction framework that prevents the model from learning the shorteut. First, revisit the formulations of fully-connected layer, convolutional layer, as well as attention layer used in neural networks, and observe that both fully-connected layer and convolutional layer face the risk of learning a trivial solution. This drawback further amplified under the multi-class setting in that the normal data distribution becomes far more complex. Instead, the attention layer is sheltered from such a risk, benefiting from learnable query embedding (see Sec. 3.1). Accordingly, we propose a layer-wise query decoder to intensify the use of query embedding. Second, we argue that the full attention (i.e., every feature point relates to each other) also contributes to the shortcut issue, because it offers the chance of directly copying the input to the output. To avoid the information leak, we employ neighbor masked attention module, where feature point relates to neither itself nor its neighbors. Third, inspired by Bengio et al. [3], we propose a feature jittering strategy, which requires the model to recover the source message even with noisy inputs. All these designs help the model escape from the "identity shortcut", as shown in Fig. 2b. Extensive experiments on MVTec-AD [4] and CIFAR-IO [23] demonstrate the sufficient superiority of our approach, which we call UniAD, over existing alternatives under the unified task setting. For instance, when learning a single model for 15 categories in MVTec-AD, we achieve state-of-the-ae performance on the tasks of both anomaly detection and anomaly localization, boosting the AUROC from 88.1% 1% to 96.5% and from 89.5% to 96.8%, respectively.
AA detailed analysis can be found in See. 3.1 and Fig. 2.

---

2
Related work
Anomaly detection. 1) Classical approaches extend classical machine learning methods for one-class classification, such as one-class support vector machine (OC-SVM) [38] and support vector data description (SVDD) [35, 41]. Patch-level embedding [48], geometric transformation [18], and elastic weight consolidation [33] are incorporated for improvement. 2) Pseudo-anomaly converts anomaly detection to supervised learning, including classification [25, 32, 45], image denoising [52], and hyper- sphere segmentation [27]. However, these methods partly rely on how well proxy anomalies match real anomalies that are not known [13]. 3) Modeling then comparison assumes that the pre-trained network is capable of extracting discriminative features for anomaly detection [11, 34]. PaDiM [11] and MDND [34] extract pre-trained features to model normal distribution, then utilize a distance metric to measure the anomalies. Nevertheless, these methods need to memorize and model all normal features, thus are computationally expensive. 4) Knowledge distillation proposes that the student distilled by a teacher normal samples could only extract normal features [6, 13, 37, 44, 45]. Recent works mainly focus on model ensemble [6], feature pyramid [37, 44), and reverse distillation [13].
Reconstruction- anomaly detection. These methods rely on the hypothesis that reconstruction models trained on normal samples only succeed in normal regions, but fail in anomalous regions [5, 8, 26, 36, 49]. Early attempts include Auto-Encoder (AE) [5, 9]. Variational Auto-Encoder (VAE) [22, 26], and Generative Adversarial Net (GAN) [2. 30, 36, 51]. However, these methods face the problem that the model could learn tricks that the anomalies are also restored well. Accordingly, researchers adopt different strategies to tackle this issue, such as adding instructional information (i.e., structural [53] or semantic [39, 46)), memory mechanism [19, 20, 29], iteration mechanism [12), image masking strategy [47], and pseudo-anomaly [9, 32]. Recently, DRAEN [52] first recovers the pseudo-anomahy disturbed normal images for representation, then utilizes discriminative net to distinguish the anomalies, achieving excellent performance. However, DRAEM [52] ceases to be effective under the unified case. Moreover, there is still an important aspect that has not been well studied, i.e., what architecture is the best reconstruction model? In this paper, we first compare and analyze three popular architectures including MLP, CNN, and transformer. Then, accordingly, we base on the transformer and further design three improvements, which compose our UniAD.
Transformer in anomaly detection. Transformer [42] with attention mechanism, first proposed in natural language processing, has been successfully used in computer vision [7, 16]. Some attempts try to utilize transformer for anomaly detection. InTra [31] adopts transformer recover the image by recovering all masked patches one by one. VT-ADL [28] and Ano [50] both apply transformer encoder to reconstruct images. However, these methods directly utilize vanilla transformer, and do not figure out why transformer brings improvement. In contrast, we confirm the efficacy of the query embedding prevent the shortcut, and accordingly design a layer-wise query decoder. Also, avoid the information leak of the full attention, we employ a neighbor masked attention module.
3 Method
3.1 Revisiting feature reconstruction for anomaly detection
In Fig. 2, following the feature reconstruction paradigm [39, 49], we build an MLP, a CNN, and transformer (with query embedding) to reconstruct the features extracted by pre-trained backbone. The reconstruction errors represent the anomaly possibility. The architectures of the three networks are given in Appendix. The metric is evaluated every 10 epochs. Note that the periodic evaluation is impractical since anomalies are not available during training. As shown in Fig. 2a, after a period of training, the performances of the three networks decrease severely with the losses going extremely small. We attribute this to the problem of "identical shortcut", where both normal and anomalous regions can be well recovered, thus failing to spot anomalies. This speculation is verified by the visualization results in Fig. 2b (more results in Appendix). However, compared with MLP and CNN, the transformer suffers from much smaller performance drop, indicating a slighter shorteut problem. This encourages us to analyze as follows.
We denote the features in a normal image as + E KRK where K is the feature number, C' is the channel dimension. The batch dimension is omitted for simplicity. Similarly, the features in an anomalous image are denoted as x e kgxx xC, The reconstruction loss is chosen the MSE loss.
3

---

070
0.0
039
340
[2]
CNN
Transformer (with query embedding)
[b)
Anomaly
MLP
CNN
Transformer
Ours
Anomaly
MLP
CNN
Transformer
Ours
Comparison among MLP, CNN, transformer, and our UniAD on MVTec-AD [4]. (a) Training loss (blue) as well the testing AUROC on anomaly detection (green) and localization (red). During the training of MLP, CNN, and transformer, the reconstruction error keeps going smaller on normal samples, but the performance on anomalies suffers from severe drop after reaching the peak. This is caused by the model learning an "identical shortcut", which tends to directly copy the input as the output regardless of whether normal or anomalous. (b) Visual explanation of the shortcut issue, where the anomalous samples can be well recovered and hence become hard to detect from normal ones. In contrast, UniAD overcomes such a problem and manages to reconstruct anomalies as normal samples. It is noteworthy that all models are learned for feature reconstruction and separate decoder employed to render images from features. This decoder is only used for visualization.
Figure 2:
We provide rough analysis using a simple I-layer network the reconstruction net, which is trained with at and tested to detect anomalous regions in x.
Fully-connected layer in MLP. Denote the weights and bias in this layer as E RCC b RCC respectively, this layer can be represented as,
y = w W KxC.
With the MSE loss pushing y x= the model may take shorteut regress - (identity matrix), + 0. Ultimately, this model could also reconstruct æ well, failing in anomaly detection.
Convolutional layer in CNN. A convolutional layer with 1x1 kernel is equivalent to fully- connected layer. Besides, An n X n (n > 1) kernel has more parameters and larger capacity, and can complete whatever 1xl kernel can. Thus, this layer also has the chance to learn shortcut.
Transformer with query embedding. In such model, there an attention layer with a learnable query embedding, E KC When using this layer as the reconstruction model, it is denoted as,
softmax (1 aaaaa++y //tlg///*////////////////// /))+ E ERXx
To push y to at, the attention map, softmax(g(2)) ()) /CC//// should approximate I (identity matrix), so q must be highly related to x* Considering that in the trained model is relevant to normal samples, the model could not reconstruct x= well. The ablation study in Sec. 4.6 shows that without the query embedding, the performance of transformer drops dramatically by 18.1 1% and 13.4% in anomaly detection and localization, respectively. Thus the query embedding is of vital significance to model the normal distribution.
However, transformer still suffers from the shorteut problem, which inspires our three improvements. 1) According to that the query embedding can prevent reconstructing anomalies, we design a Layer- wise Query Decoder (LQD) by adding the query embedding in each decoder layer rather than only the first layer in vanilla transformer. 2) We suspect that the full attention increases the possibility of the shorteut. Since one token could see itself and its neighbor regions, it is easy reconstruct by simply copying. Thus we mask the neighbor tokens when calculating the attention map, called Neighbor Masked Attention (NMA). 3) We employ Feature Jittering (FJ) strategy to disturb the input features, leading the model to learn normal distribution from denoising. Benefiting from these designs, our UniAD achieves satisfying performance, illustrated in Fig. 2.
Relation between the 'identical shortcut" problem and the unified case. Fig. 2a, we aim to visualize the "identical shortcut" problem, where the loss becomes smaller yet the performance drops.

---

NME
LQD
Reconstructed Tokens
Train on Normal Samples
Add Followed by Layer Norm
N
Reconstructed Tokens
N X
Food Forward Net
MSE Loss
Feature Tokens
Food Forward Net
Test for Anomaly Localization Detection
Neighbor Masked
Attention
Reconstructed Tokens
Noighbor Masked
Difference
Attention
Neighbor Masked
Feature Tokens
Atlention
Localization
م Perturbation
= Query Embedding
Feature Tokens
Avg Pool Max
Feature Jittering
Outputs of Last Layer
Detection Results
Figure Framework UniAD, consisting of Neighbor Masked Encoder (NME) and a Layer-wise Query Decoder (LQD). Each layer in LQD employs learnable query embedding to help model the complex training data distribution. The full attention in transformer is replaced by neighbor masked attention to avoid the information leak from the input to the output. The feature jittering strategy encourages the model recover the correct message with noisy inputs. All the three improvements assist the model against learning the "identical shortcut" (see Sec. 3.1 and Fig. 2 for details).
Available Value
Key
Masked Neighbor
Query
Unified Case
Separate Case
Flatten
Comparison between the unified case and the separate case on the training curves of MLP. In the separate case, the curves obtained by averaging all categories. Compared with the separate case, the unified case has smaller reconstruction error but much worse performance, indicating a severer "identical shortcut" problem.
Figure 4:
Query
Key
Attention Map
Figure 5: attention, where a feature point relates to neither itself nor its neighbors.
Illustration of neighbor masked
We conduct the same experiment under the separate case on MLP. As shown in Fig. 4, the accuracy (green for detection and red for localization) keeps growing up along with the loss (blue) getting smaller. This helps reveal the relation between the "identical shortcut" problem and the unified case, which is that the unified case more challenging and hence magnifies the "identical shortcut" problem. Therefore, since our approach is specially designed to solve the "identical shortcut" problem, our method can be effective in the unified case.
3.2 Improving feature reconstruction for unified anomaly detection
Overview. As shown in Fig. 3, our UniAD is composed of a Neighbor Masked Encoder (NME) and Layer-wise Query Decoder (LQD). Firstly, the feature tokens extracted by fixed pre-trained backbone are further integrated by NME to derive the encoder embeddings. Then, in each layer of LQD, a learnable query embedding successively fused with the encoder embeddings and the outputs of the previous layer (self-fusion for the first layer). The feature fusion is completed by the Neighbor Masked Attention (NMA). The final outputs of LQD are viewed as the reconstructed features. Also, we propose a Feature Jittering (FJ) strategy to add perturbations to the input features, leading the model to learn normal distribution from the denoising task. Finally, the results of anomaly localization and detection obtained through the reconstruction differences.
5

---

Neighbor masked attention. We suspect that the full attention in vanilla transformer [42] contributes to the "identical shortcut". In full attention, one token is permitted see itself, it will be easy to reconstruct by simply copying. Moreover, considering that the feature tokens are extracted by CNN backbone, the neighbor tokens must share lots of similarities. Therefore, we propose to mask the neighbor tokens when calculating the attention map, called Neighbor Masked Attention (NMA). Note that the neighbor region is defined the 2D space, as shown in Fig. 5.
Neighbor masked encoder. The encoder follows the standard architecture in vanilla transformer. Each layer consists of an attention module and a Feed-Forwaid Network (FFN). However, the full attention is replaced by our proposed NMA to prevent the information leak.
Layer-wise query decoder. It is analyzed in Sec. 3.1 that the query embedding could help prevent reconstructing anomalies well. However, there is only one query embedding in the vanilla transformer. Therefore, we design Layer-wise Query Decoder (LQQD) to intensify the use of query embedding, as shown in Fig. 3. Specifically, in each layer of LQQ, a learnable query embedding is first fused with the encoder embeddings, then integrated with the outputs of the previous layer (self-integration for the first layer). The feature fusion is implemented by NMA. Following the vanilla transformer, a 2-layer FFN applied handle these fused tokens, and the residual connection is utilized facilitate the training. The final outputs of LQD serve as the reconstructed features.
Feature jittering. Inspired by Denoising Auto Encoder (DAE) [3, 43], we add perturbations to feature tokens, guiding the model to learn knowledge of normal samples by the denoising task. Specifically, for feature token, ftok CRC, sample the disturbance 1 from Gaussian distribution,
D N~ = 0, o² = (a (lokl
C
where a is the jittering scale to control the noisy degree. Also, the sampled disturbance is added to Stok with a fixed jittering probability,
3.3 Implementation details
Feature extraction. We adopt fixed EfficientNet-b4 [40] pre-trained ImageNet [14] as the feature extractor. The features from stage-1 I to stage-4 are selected. Here the stage means the combination of blocks that have the same size of feature maps. Then these features resized the same size, and concatenated along channel dimension to form a feature map. forg E Rora x H W
Feature reconstruction. The feature map, forg, is first tokenized to Hl X W feature tokens, followed by a linear projection to reduce Corg to a smaller channel, C. Then these tokens are processed by NME and LQD. The learnable position embeddings [15, 16] are added in attention modules to inform the spatial information. Afterward, another linear projection is used to recover the channel from C to Corg. After reshape, the reconstructed feature map, free E Cory ff W is finally obtained.
Objective function. Our model trained with the MSE loss as.
1
L
H x W |l - freelle.
(4)
Inference for anomaly localization. The result of anomaly localization is an anomaly score map. which assigns an anomaly score for each pixel. Specifically, the anomaly score map, is calculated as the L2 norm of the reconstruction differences as,
s = frg freell2 #
Then 8 is up-sampled to the image size with linear interpolation to obtain the localization results.
Inference for anomaly detection. Anomaly detection aims to detect whether an image contains anomalous regions. We transform the anomaly score map, s, the anomaly score of the image by taking the maximum value of the averagely pooled s.
Experiment
4.1 Datasets and metrics
MVTec-AD [4] is a comprehensive, multi-ohject, multi-defect industrial anomaly detection dataset with 15 classes. For each anomalous sample in the test set, the ground-truth includes both image

---

Table 1: Anomaly detection results with AUROC metric on MVTec-AD [4]. All methods are evaluated under the unified / separate case. In the unified case, the learned model is applied to detect anomalies for all categories without fine-tuning.
Category
US [6]
PSVDD [48] PaDiM [11] CutPaste [25]
MKD [37] DRAEM [52]
Ours
Bottle
84.0 / 99.0
85.5 / 98.6
97.9 / 99.9
67.9/ / 98.2
98.7 / 99.4
97.5 99.2
99.7 = 0.04 / 100
Cable
60,0 / 86.2
64.4/ 90.3
70,9 / 92.7
69.2 81.2
78.2 89.2
57.8 /9.8
95.2 = 0.84/ 97.6
Capsule
57.6 / 86.1
61.3 / 76.7
73.4 /9.3
63.0 /
68.3
65.3/
86.9 = 0.73 / 85.3
Hazelnut
95.8 / 93.1
83.9 92.0
85.5 92.0
80.9 / 98.3
97.1 / 98.4
93.7 / 100
99.8 = 0.10 99.9
Obleerr
Metal Nut
62.7 / 82.0
80.9 94.0
88.0 / 98.7
60.0 / 99.9
64.9 04.9/70 73.6
72.8 / 98.7
99.2 = 0.09 / 99.0
Pill
56.1 / 87,9
89.4 86.1
68.8 / 93.3
7 94.9
79.7 82.7
82.2 / 98.9
93.7 - 0.65 / /883
Screw
66,9 / $4,9
80.9 / 81.3
56.9 85.8
85.2
75.6 /
92.0 93.9
87.5 - 0.57 / 91.9
Toothbrush
57.8 95.3
99.4 100
95.3 96.1
63.9 99.4
75.3 92.2
90.6 100
94.2 - 0.20 / 95.0
Transistor
61.0 / 81.8
77.5 91.5
86.6 97.4
57.9 96.1
73.4/ 85.6
74.8 / 93.1
99.8 = 0.09 100
Zipper
78.6 / 91.9
77.8 / 97.9
79.7 / 90.3
93.5 / 99.9
87.4 93.2
98.8 / 100
95.8 = 0.51 / 96.7
Carpet
86.6 / 91.6
63.3 / 92.9
93.8 / 99.8
93.6 / 93.9
69.8 / 79.3
98.0 97.0
99.8 + 0.02 / 99.9
Grid
69.2 / 81.0
66.0 / 94.6
73.9 99.7
93.2 / 100
83.8 78.0
99.3 99.9
98.2 = 0.26 / 98.5
Texture
Leather
97.2 / 88.2
60.8 90.9
99.9 / 100
93.4/ /
93.6 95.1
98.7 / 100
100 - 0.00 / 100
Tile
93.7 / 99.1
88.3 97.8
93.3 98.1
88.6 / 94.6
89.5 91.6
99.8 99.6
99.3 = 0.14 99.0
Wood
90.6 / 97.7
72.1 / 96.5
98.4 99.2
80.4/ 99.1
93.4/ 94.3
99.8 99.1
98.6 = 0.08 / 97.9
Mean
74.5 / 87.7
76.8 92.1
84.2 95.5
77.5 96.1
81.9 87.8
88.1 99.0
96.5 + 0.08 96.6
label and anomaly segmentation. In the existing literature, only the separate case is researched. In this paper, introduce the unified case, where only one model is used to handle all categories.
CIFAR-10 [23] is a classical image classification dataset with 10 categories. Existing methods [6, 24, 37] evaluate CIFAR-10 10 mainly in the one-versus-mary setting, where one class is viewed as normal samples, and others serve as anomalies. Semantic AD [1, 10] proposes a many-versus-one setting, treating one class anomalous and the remaining classes as normal. Different from both, we propose a unified case tmany-versus-mee manssssssmaayaeeee setting), which is detailed in Sec. 4.4.
Metrics. Following prior works [4, 6, 52], the Area Under the Receiver Operating Curve (AUROC) used as the evaluation metric for anomaly detection.
4.2 Anomaly detection on MVTec-AD
Setup. Anomaly detection aims to detect whether an image contains anomalous regions. The anomaly detection performance is evaluated on MVTec-AD [4]. The image size is selected as 224 224, and the size for resizing feature maps is set as 14 x 14. The feature maps from stage-1 1 to stage-4 of EfficientNet-b4 [40] are resized and concatenated together to form a 272-channe feature map. The reduced channel dimension is set as 256. AdamW optimizer [21] with weight decay 1 x 10 is used. Our model is trained for 1000 epochs on 8 GPUs (NVIDIA Tesla V100 16GB) with batch size 64. The learning rate is 1 x 10-4 initially, and dropped by 0.1 after 800 epochs. The layer numbers of the encoder and decoder are both 4. The neighbor size, jittering scale, and jittering probability are set as 7x7, 20, and 1, respectively. The evaluation is run with 5 random seeds. In both the separate case and the unified case, the reconstruction models are trained from the scratch.
Baselines. Our approach is compared with baselines including: US [6]. PSVDD [48], PaDiM [11], CutPaste [25], MKD [37], and DRAEM [52]. Under the separate case, the basclines' metric is reported in their papers except the metric of US borrowed from [52]. Under the unified case, US PSVDD, PaDiM, CutPaste, MKD, and DRAEM are run with the publicly available implementations.
Quantitative results of anomaly detection on MVTec-AD [4] are shown in Tab. 1. Though all baselines achieve excellent performances under the separate case, their performances drop dramatically under the unified case. The previous SOTA, DRAEM, reconstruction based method trained by pseudo-anomaly, suffers from a drop of near 10% For another strong baseline, CutPaste, a pseudo-anomaly approach, the drop is as large as 18.6% However, our UniAD has almost no performance drop from the separate case (96.6%) to the unified case (9.55)). Moreover, we beat the best competitor, DRAEM. by a dramatically large margin (8.4%), demonstrating our superiority.
7

---

Table Anomaly localization results with AUROC metric on MVTec-AD [4]. All methods are evaluated under the unified / separate case. In the unified case, the learned model is applied to detect anomalies for all categories without fine-tuning.
Category
US [6]
PSVDD [48] PaDiM [11] FCDD [27]
MKD [37] DRAEM [52]
Ours
Bottle
67.9 / 97.8
86.7 / 98.1
96.1 / 98.2
56,0 / 97
91.8 / 96.3
87.6/ 99.1
98.1 + 0.04 / 98.1
Cable
78.3 91.9
62.2 / 96.8
81.0/ 96.7
64.1 / 90
89.3 / 82.4
71.3
97.3 + 0.10 96.8
Capsule
85.5
83.1 / 95.8
96.9
67.6/ 93
88.3 95.9
50.5 94.3
98.5 + 0.01 97.9
Hazelnut
93.7 98.2
97.4 97.5
96.3 / 98.1
79.3 / 95
91.2 94.6
96.9 99.7
98.1 + 0.10 98.8
Obleet
Mctal Nut
76.6 97.2
96.0 / 98.0
84.8 97.3
57.5 / 94
64.2 / 86.4
62.2/ 99.5
94.8 + 0.09 95.7
Pill
80.3 96.5
96.5 / 95.1
87.7 95.7
/5.9/8t 81
69.7 / 89.6
94.4/
95.0 1 0.16 95.1
Screw
90.8 97.4
74.3 / 95.7
94.1
67.2 / 86
92.1 / 96.0
95.5 97.6
98.3 1 0.08 97.4
Toothbrush
86.9 97.9
98.0 98.1
95.6 98.8
60.8 / 94
88.9 99.1
97.7 98.1
98.4 1 0.03 97.8
Transistor
68.3 73.7
78.5 / 97.0
92.3 97.6
54.2 / 88
71.7 76.5
64.5 /0.9
97.9 = 0.19 98.7
Zipper
84.2 95.6
95.1 / 95.1
94.8 98.4
63.0 92
86.1 93.9
98.3 / 98.8
96.8 + 0.24 96.0
Carpet
88.7 / 93.5
78.6 / 92.6
97.6 77.6/%9 99.0
68.6 / 96
95.5 / 95.6
98.6 / 95.5
98.5 + 0.01 / 98.0
Grid
64.5 89.9
70.8 / 96.2
71.0 97.1
65.8 91
82.3 / 99.8
98.7
99.7
96.5 + 0.04 / 94.6
Texture
Leather
95.4 97.8
93.5 / 97.4
84.8 99.0
66.3 / 98
96.7 / 98.1
97.3
98.6
98.8 + 0.03 / 98.3
Tile
82.7 92.5
92.1 / 99.4
80.5 94.1
59.3 / 91
85.3 82.8
98.0
99.2
91.8 + 0.10 / 91.8
Wood
83.3 92.1
80.7 / 90.8
89.1 / 94.1
53.3 / 88
80.5 / 84.8
96.0 / 96.4
93.2 + 0.08 / 93.4
Mean
81.8 93.9
85.6 / 95.7
89.5 97.4
63.3 92
84.9 90.77
87.2 97.3
96.8 + 0.02 96.6
Normal
Anomaly
Recon
GT
Pred
Normal
Anomaly
Recon
GT
Pred
(a)
(b)
(c)
(d)
Figure 6: Qualitative results for anomaly localization on MVTec-AD [4]. From left to right: normal sample as the reference, anomaly, our reconstruction, ground-truth, and our predicted anomaly map. The approach to visualizing reconstruction is the same the one used in Fig. 2.
4.3 Anomaly localization on MVTec-AD
Setup and baselines. Anomaly localization aims to localize anomalous regions in an anomalous image. MVTec-AD [4] is chosen the benchmark dataset. The setup is the same as that in Sec. 4.2. Besides the competitors in Sec. 4.2, FCDD [27] is included, whose metric under the separate case is reported in its paper. Under the unified case, run FCDD with the implementation: FCDD.
Quantitative results of anomaly localization on MVTec-AD [4] are reported in Tab, 2. Similar to Sec. 4.2, switching from the separate case to the unified case, the performance of all competitors baseline, drops significantly. For example, the performance of US, an important distillation-baseee decreases by 12. %. FCDD, pseudo-anomale approach, suffers from a dramatic drop of 28.7%, reflecting the pseudo-anomaly is not suitable for the unified case. However, our UniAD even gains a slight improvement from the separate case (96.6%) to the unified case (96.8%), proving the suitability of our UniAD for the unified case. Moreover, we significantly surpass the strongest baseline, PaDiM, by 7.3%. This significant improvement reflects the effectiveness of our model.
Qualitative results for anomaly localization on MVTec-AD [4] are illustrated in Fig. 6. For both global (Fig. 6а) and local (Fig. 6b) structural anomalies, both scattered texture perturbations (Fig. 6c) and multiple texture scratches (Fig. 6d), our method could successfully reconstruct anomalies to their corresponding normal samples, then accurately localize anomalous regions through reconstruction differences. More qualitative results are given Appendix.
4.4 Anomaly detection on CIFAR-10
Setup. To further verify the effectiveness of our UniAD, we extend CIFAR-10 [23] to the unified case, which consists of four combinations. For each combination, five categories together serve as normal samples, while other categories are viewed as anomalies. The class indices of the four
8

---

Table Anomaly detection results with AUROC metric on CIFAR-10 [23] under the unified case. Here, {01234) means samples from class 0, 0, 4 are borrowed as the normal ones.
Normal Indices
US [6]
FCDD [27]
FCDD+OE [27]
PANDA [33]
MKD [37]
Ours
{01234)
51.3
55.0
71.8
66.6
64.2
84,4 + 0.02
(56789)
51.3
50.3
73.7
73.2
69.3
80.9 0.02
(02468)
63.9
59.2
85.3
77.1
76.4
93.0 0.03
(13579)
56.8
58.5
85.0
72.9
78.7
90.6 0.09
Mcan
55.9
55.8
78,9
72.4
72.1
87.2 + 0.03
Table 4:
Performance comparison and architecture comparison between UniAD and transformer- based competitors on MVTec-AD [4]. All methods are evaluated under the unified / separate case.
Method
Det.
Loc.
query
Layer-wise query
InTra [31]
65.3 95.0
70.6 / 96.6
VT-ADL [28]
55.4 78.7
64.4 / 82.0
X
AnoVit [50]
69.6 / 78
68.4 /
X
X
Ours (baseline)
87.6 94.7
92.8 / 95.8
V
Ours
96.5 / 96.6
96.8 / 96.6
x
combinations are {01234), (56789). {02468), {13579}. Here, {01234) means the normal samples include images from class 0, 1,2, 3, 4, and similar for others. Note that the class index is obtained by sorting the class names of 10 classes. The setup of the model is detailed in Appendix.
Baselines. US [6]. FCDD [27], FCDD+OE [27]. PANDA [33], and MKD [37] serve as competitors. US, FCDD, FF PANDA, and MKD) are run with the publicly available implementations.
Quantitative results of anomaly detection CIFAR-10 [23] are shown in Tab. 3. When five classes together serve as normal samples, two recent baselines, US and FCDD, almost lose their ability detect anomalies. When utilizing 10000 images sampled from CIFAR-I00 [23] as auxiliary Outlier Exposure (OE), FCDD+OE improves the performance by a large margin. We still stably outperform FCDD+OE by 8.3% without the help of OE, indicating the efficacy of our UniAD,
4.5 Comparison with transformer-based competitors
As described Sec. 2, some attempts [31, 1,,,20 28, 50] also try to utilize transformer for anomaly detection. Here compare UniAD with existing transformer-based competitors MVTec-AD [4]. Recall that, we choose transformer as the reconstruction model considering its great potential in preventing the model from learning the "identical shortcut" (refer to Sec. 3.1). Concretely, learnable query embedding is essential for avoiding such shortcut but is seldom explored in existing transformer- baased approaches. As shown in Tab. 4, after introducing even only one query embedding, our baseline already outperforms existing alternatives by sufficiently large margin in the unified setting. Our proposed three components further improve our strong baseline. Recall that all three components are proposed to avoid the model from directly outputting the inputs.
find that the
4.6 Ablation studies
To verify the effectiveness of the proposed modules and the selection of hyperparameters, implement extensive ablation studies on MVTec-AD [4] under the unified case.
Layer-wise query. Tab. 5a verifies assertion that the query embedding is of vital significance. 1) Without query embedding, meaning the encoder embeddings are directly input to the decoder, the performance is the worst. 2) Adding only one query embedding the first decoder layer (i.e., vanilla transformer [42]) promotes the performance dramatically by 18.1% and 13.4% in anomaly detection and localization, respectively. 3) With layer- wise query embedding in each decoder layer, lmaee-level and pixel-level AUROC is further improved by 7.4% and 3.7%, respectively.
Layer number. We conduct experiments to investigate the influence of layer number, shown in Tab. 5b. 1) No matter with which combination, our model outperforms vanilla transformer by large margin, reflecting the effectiveness of our design. 2) The best performance is achieved with

---

Table Ablation studies with AUROC metric on MVTec-AD [4]. Default settings are in blue.
(a) Layer-wise query, NMA, & FJ
(b) Layer Number of Encoder & Decoder
w/o q. 1..
Layer-wise 4.
NMA
Det.
Loc.
Vanilla [42]
Ours
#Enc, #Dec
Det.
Loc.
Det.
Loc.
69.5
79.4
87.6
92.8
4, 0
69.8
79.2
94.9
96.0
95.0
96.5
0,4
80.5
88.3
96.1
96.3
96.1
96.3
2,2
84.7
90.6
95.1
96.0
95.0
95.8
4,4 4
87.6
92.8
96.5
96.8
96.5
96.8
6, 6
86.1
91.9
96.5
96.7
(c) Neighbor Size in NMA
(d) Where to Add NMA
(e)
Jitter Scale a FJ
(f) Jitter Proh. p in FJ
Size
Det.
Loc.
Place
Det.
Loc.
a
Det.
Loc.
Det.
Loc.
1x1
94.6
96.3
Enc
95.8
96.3
5
96.1
96.7
0,25
95.6
96.5
5x5
96.4
96.8
Enc+Decl
96.4
96.8
10
96.4
96.7
0.50
95.8
96.7
7x7
96.5
96.8
Enc+Dec2
96.5
96.7
20
96.5
96.8
0.75
96.3
96.7
9x9
96.3
96.7
All
96.5
96.8
30
95.7
96.6
96.5
96.8
moderate layer number: 4Enc+4Dec. A larger layer number like 6Enc+6Dec does not bring further promotion, which may be because more layers are harder to train.
Neighbor masked attention. 1) The effectiveness of NMA is proven in Tab. 5a. Under the case of one query embedding, adding NMA brings promotion by 8.5% for detection and 3.5% for localization. 2) The neighbor size of NMA is selected in Tab. 5c. 1x 1 neighbor size is the worst, because X is too small to prevent the information leak, thus the recovery could be completed by copying neighbor regions. A larger neighbor size (2 5 5) is obviously much better, and the best one is selected as 7. 3) We also study the place to add NMA in Tab. 5d. Only adding NMA in the encoder (Ene) is not enough. The performance could be stably improved when further adding NMA in the first second attention in the decoder (Enc+Decl Enc+Dec2) or both (All). This reflects that the full attention of the decoder also contributes to the information leak.
Feature jittering. 1) Tab. 5a confirms the efficacy of FJ. With one query embedding as the baseline, introducing FJ could bring an increase of 7.4% for detection and 3.0% for localization, respectively. 2) According to Tab. Se, the jittering scale, a, chosen as 20. A larger a (i.e., 30) disturbs the feature too much, degrading the results. 3) In Tab. 5f, the jittering probability, p. is studied. In essence, the task would be denoising task with feature jittering, and be a reconstruction task without feature jittering. The results show that the full denoising task (i.e., p 1) is the best.
5 Conclusion
In this work, we propose UniAD that unifies anomaly detection regarding multiple classes. For such a challenging task, we assist the model against learning "identical shortcut" with three improvements. First, we confirm the effectiveness of the learnable query embedding and carefully tailor a layer-wise query decoder to help model the complex distribution of multi-class data. Second, come up with neighbor masked attention module to avoid the information leak from the input to the output. Third, we propose feature jittering that helps the model less sensitive to the input perturbations. Under the unified task setting, our method achieves state-of offthhe-rrt art performance on MVTec-AD and CIFAR-10 datasets, significantly outperforming existing alternatives.
Discussion. In this work, different kinds of objects are handled without being distinguished. We have not used the category labels that may help the model better fit multi-class data. How to incorporate the unified model with category labels should be further studied. In practical uses, normal samples are not as consistent as those in MVTec-AD, often manifest themselves in some diversity. Our UniAD could handle all 15 categories in MVTec-AD: hence would be more suitable for real scenes. However, anomaly detection may be used for video surveillance, which may infringe personal privacy.
Acknowledgments and Disclosure of Funding
Acknowledgement. This work is sponsored by the National Key Research and Development Program of China 2021YFB171666 and National Natural Science Foundation of China (62176152).
10

---

References
[1] F. Ahmed and A. Courville. Detecting semantic anomalies. In Assoc. Adv. Artif. Intell., 2020.
[2] S. Akeay, A. Slapour-Abaghhoo and T. P. Breckon. GANomaly: Semi-superviser anomaly detection via adversarial training. In Asian Conf. Comput. Vis., 2018.
[3] Y. Bengio, L. Yao, G. Alain, and P. Vincent. Generalized denoising auto-encoders as generative models. In Adv. Neural Inform. Process. Syst. 2013.
[4] P. Bergmann, M. Fauser. D. Sattlegger. and C. Steger. MVTec AD-A A comprehensive real-world dataset for unsupervised anomaly detection. In IEEE Conf. Comput. Vis. Pattern Recog. 2019.
[5] P. Bergmann, S. Löwe, M. Fauser, D. Sattlegger, and C. Steger. Improving unsupervised defect segmentation by applying structural similarity to autoencoders. In International Joint Conference Computer Vision, Imaging and Computer Graphics Theory and Applications ( (VISIGRAPP), 2019.
[6] P. Bergmann, M. Fauser, D. Sattlegger, and C. Steger. Uninformed students: Student-teacher anomaly detection with discriminative latent embeddings. IEEE Conf. Comput. Vis. Pottern Recog., 2020.
[7] N. Carion, F. Massa, G. Synnaeve, N. Usunier, A. Kirillow, and S. Zagoruyko. End-to-end object detection with transformers. In Fur. Conf. Comput. Vis., 2020.
[8] L. Chen, Z. You, N. Zhang, J. Xi, and X. Le. UTRAD: Anomaly detection and localization with U- transformer. Neural Networks, 2022.
[9] A.-S. Collin and C. De Vleeschouwer. Improved anomaly detection by training autoencoder with skip connections on images corrupted with stain-shaped noise. In Int. Conf. Pattenn Recog., 2021.
[10] L. Deecke, L. Ruff, R. A. Vandermeulen, and H. Bilen. Transfer-based semantic anomaly detection. In Int. Conf. Mach. Learn., 2021.
[11] T. Defard, A. Setkov, A. Loesch, and R. Audigier. PaDim: A patch distribution modeling framework for anomaly detection and localization. In Int. Conf. Pattern Recog., 2021.
[12] D. Dehaene, O. Frigo, S. Combrexelle, and P. Fline. Iterative energy- -aased projection a normal data manifold for anomaly localization. In Int. Conf. Learn. Reprexent. 2019.
[13] H. Deng and X. Li. Anomaly detection via reverse distillation from one-class embedding. In IEEE Conf. Comput. Vis, Pattern Recog., 2022.
[14] J. Deng, W. Dong, R. Socher. L.-J. Li, K. Li, and L. Fei-Fei. ImageNet: A large-scale hierarchical image database. In IELE Conf. Comput. Vis. Pattern Recog., 2009.
[15] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding, arXiv preprint arXiv:1810.04805, 2018.
[16] A. Dosovitskiy. L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, al. An image is worth 16x16 words: Transformers for image recognition at scale. Int. Conf. Learn. Represent. 2021.
[17] T. Fernando. H. Gammulle, S. Denman, S. Sridharan, and C. Fookes. Deep learning for medical anomaly detection-a survey. ACM Computing Surveys (CSUR), 2021.
[18] Golan and R. El- Yaniv, Deep anomaly detection using geometric transformations. In Adv. Neural Inform. Process. Syst., 2018
[19] D. Gong, L. Liu, V. Le, B. Saha, M. R. Mansour, S. Venkatesh, and A. d. Hengel. Memorizing normality to detect anomaly: Conf. Comput. Vis., 2019.
deep autoencoder for unsupervised anomaly detection. In Int.
[20] J. Hou, Y. Zhang, Q Zhong, D. Xic, S. Pu, and H. Zhou. Divide-and-asemble: Learning block-wise memory for unsupervised anomaly detection. In Int. Conf. Comput. Vis., 2021.
[21] L. Ilya and Frank. Decoupled weight decay regularization.
Conf. Learn. Represent. 2019.
[22] D. P. Kingma and M. Welling. Auto-cncoding variational bayes. arXiv preprint arXiv:1312.6114 2013.
[23] A. Krizhevsky, Learning multiple layers of features from tiny images. Master's thesis, University of Trons,
2009.
11

---

[24] G. Kwon, M. Prabhushankar, D. Temel. and G. AlRegib, Backpropagated gradient representations for anomaly detection. In Eur. Conf. Comput. Vis., 2020.
[25] C.-L. Li, K. Sohn, J. Yoon, and T. Pfister. CutPaste: supe-vipe learning for anomaly detection and localization. In IEEE Conf. Comput. Vis. Pattern Recog., 2021.
[26] W. Liu, R. Li, M. Zheng, S. Karanam, Z. Wu, B. Bhanu, R. J. Radke, and O, Camps. Towards visually explaining variational autoencoders. In IEEE Conf. Comput. Vis. Pattern Recog. 2020.
[27] P. Liznerski, L. Ruff, R. A. Vandermeulen, B. Franks, M. Kloft, and K.-R. Müller. Explainable deep one-class classification. In Int. Conf. Learn. Represent, 2021.
[28] P. Mishra, R. Verk, D. Fornasier, C. Piciarelli, and G. I.. Foresti. VT-ADL: A vision transformer network for image anomaly detection and localization. In International Symposium on Industrial Electronics, 2021.
[29] H. Park, J. Noh, and B. Ham. Learning guindr normality for anomaly detection. In IEEE Conf. Comput. Vis. Pattern Recog,, 2020.
[30] P. Perera, R. Nallapati, and B. Xiang, OCGAN: One-class novelty detection using GANs with constrained latent representations. In IEEE Conf. Compul. Vis. Puttern Recog., 2019.
[31] J. Pimay and K. Chai. Inpainting transformer for anomaly detection. arXiv preprint arXiv:2104.1398 2021.
[32] M. Pourreza, B. Mohammadi, M. Khaki, S. Bouindour, H. Snoussi, and M. Sabokrou. G2D: generate to detect anomaly. In /EEE Winter Conf. Appl. Comput. Vis., 2021.
[33] T. Reiss, N. Cohen, L. Bergman, and Y. Hoshen. Panda: Adapting pretrained features for anomaly detection and segmentation. In IEEE Conf. Comput. Vis. Pottern Recog. 2021.
[34] O. Rippel, P. Mertens, and D. Merhof. Modeling the distribution of normal data in pretrained deep features for anomaly detection. In Int. Conf. Pattern Recog., 2021.
[35] L. Ruff, R. Vandermeulen, N. Goernitz, L. Deecke, A. Siddiqui, A. Binder, E. Müller, and M. Kloft. Deep one-class classification. In Int. Conf. Mach. Learn., 2018.
[36] M. Sabokrou, M. Khalonei, M. Fathy, and Adeli. Adversarially learned one-class classifier for novelty detection. In JEEE Conf. Comput. Vis. Pattern Recog., 2018.
[37] M. Salehi, N. Sadjadi, S. Baselizadeh, H. Rohban, and H. R. Rabiee. Multiresolution knowledge distillation for anomaly detection. In IEEE Conf. Comput. Vis. Pattern Recog., 2021.
[38] B. Schölkopf, C. Platt, J. Shawe-Taylor, A. J. Smola, and R. C. Williamson. Estimating the support of a high-dimensional distribution. Neural Computation, 2001.
[39] Y. Shi, Yang, and Z. Qi. Unsupervised anomaly segmentation via deep feature reconstruction. Newrocomputing, 2021.
[40] M. Tan and Q Le. EfficientNet: Rethinking model scaling for convolutional neural networks. In Int. Conf. Mach. Learn., 2019.
[41] D. M. Tax and R. P. Duin. Support vector data description. Machine Learning, 2004.
[42] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin. Attention is all you need. In Adv. Neural Inform. Process. Syst., 2017.
[43] P. Vincent, H. Larochelle, Y. Bengio, and P.-A. Manzagol. Extracting and composing robust features with denoising autoencoders. In Int. Conf. Mach. Learn., 2008
[44] G. Wang, S. Han, F. Ding, and D. Huang. Student-teacher feature pyramid matching for anomaly detection. Brit. Mach. Vis. Conf., 2021.
[45] S. Wang, L. Wu, L. Cui, and Y. Shen. Glancing at the patch: feature comparison. In IEEE Conf. Comput. Vis. Pattern Recog. 2021.
Anomaly localization with global and local
[46] Y. Xia, Y. Zhang. F. Liu, W. Shen, and A. L. Yuille. Synthesize then compare: Detecting failures and anomalies for semantic segmentation. In Eur. Conf. Compur. Vis., 2020.
[47] X. Yan, H. Zhang, X. Xu, X. Hu, and P.-A Heng. Learning semantic context from normal samples for unsupervised anomaly detection. In Assoc. Adv. Artif. Intell., 2021.
12

---

[48] J. Yi and S. Yoon. Patch SVDD: Patch-level SVDD for anomaly detection and segmentation. In Asian Conf. Comput. Vis., 2020.
[49] Z. You, K. Yang. W. Luo, Cui, Y. Zheng, and X. Le. ADTR: Anomaly detection transformer with feature reconstruction. arXiv preprint arXiv:2209.01816 2022.
[50] L. Yunseung and K. Pilsung. Vno Unsupervised anomaly detection and localization with vision transformer-bas encoder-decoder arXiv preprint arXiv:2203. J0808, 2022.
[51] Z. Zaheer, J.-h. Lee, M. Astrid, and S.-I. Lee. Old is gold: Redefining the adversarially learned one-class classifier training paradigm. In JEEE Conf. Comput. Vis. Puttern Recog., 2020.
[52] V. Zavrtanik, M. Kristan, and D. Skočaj. DRAEM-A discriminatively trained reconstruction embedding for surface anomaly detection. In Int. Conf. Comput. Vis., 2021.
[53] K. Zhou, Y. Xiao, J. Yang. J. Cheng, W. Liu, W. Luo, Z. Gu, J. Liu, and S. Gao. Encoding cenncure-texturr relation with P-Net for anomaly detection in retinal images. In Eur. Conf. Comput. Vis., 2020.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] See Discussion in Sec. 5.
(c) Did you discuss any potential negative societal impacts of your work? [Yes] See Discussion in Sec. 5.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as URL)? [Yes] See Sec. and Appendix. Also, we have released the code.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See Sec. 4.1 & Sec. 4.4 for data splits, and Sec. 4.6 for the choice of hyperparameters.
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)? [Yes] See Tab. Tab. 2, and Tab. 3.
(d) Did you include the total amount compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] See Setup in Sec. 4.2.
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes] See Sec. 4.1 and Baselines in Sec. 4.
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any new assets either in the supplemental material or as a URL? [N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
13

---

(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A]
14


Reviews:
[{"review_id": "UiTaKu_cyes", "reviewer": "Reviewer_SFc6", "summary": "The authors propose the learning of multi-class decision boundaries for the task of anomaly detection (AD) over multiple object classes. For this, they employ reconstruction-based scores obtained from a transformer network, modified with a couple of simple tricks, such as masking neighboring points in the attention map, and increasing the capacity of the decoder. Results on MVTec show this is a promising direction for AD over multiple object classes.", "questions": "This paper presents strong experimental results, however appears to target a somewhat loose definition of multi-class AD. I would suggest the authors work to improve the clarity of their manuscript.\n\nIn particular:\n- clarify how the proposed multi-object AD task of \"unified AD\" fits in or differs from existing works that assume multiple objects in the normal distribution (e.g. \"semantic AD\").\n- clarify the novelty of the proposed transformer components, which appear highly incremental. Improvements could consist in more direct comparisons (and ablations) against transformer-based competitor models.", "limitations": "Limitations are discussed in Section 5, potential societal impacts (say, anomaly detection for tasks such as video surveillance) have not been discussed.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "# Strengths\n\nTo enable their transformer-based model to work with the task of adopting a complex normal distributions, the authors come up with some modifications, in particular neighbor-masked attention which they insert directly into the architecture to replace the default attention layer. While the authors employ other approaches such as \"feature jittering\", these correspond to a simple addition of Gaussian noise during the input stage, and given its simplicity I would suggest authors can remove this from abstract etc., as it doesn't present any significant novelty.\n\nThe experimental results are the strong point of this work, with outstanding performance on MVTec-AD and CIFAR10 (which is a much less relevant benchmark though, in particular as more challenging ones have recently been used, e.g. CIFAR100/STL10, c.f. works listed below).\n\nMoreover, the paper is well-written and easy to follow.\n\n# Weaknesses\n\nThe proposed modifications are relatively straightforward, in particular the ablations in Table 4 indicate that a vanilla transformer would end up outperforming the existing state of the art on MVTec-AD. Given there are recent works on using transformers for AD, e.g. AnoVIT (as pointed out by the authors on page 3 lines 99-101), this somewhat limits the novelty of the proposed method.\n\nSurprisingly, from the ablation in Table 4 it appears as if feature jittering (FJ) boosts performance nearly as much as neighborhood-masked attention (NMA). I would suggest included pairwise coupling (e.g. NMA+FJ) to shed light on which ones of these are required in unison, or whether they obtain similar outcomes.\n\nMoreover, there have been various works in the recent past that attempt to perform anomaly detection over classes that contain more than a single object class (with no labels assumed present to distinguish the objects). A discussion of these recent works, and how they relate/differ from the way in which multiple classes are treated here is missing, e.g.:\n- \"Deep Semi-Supervised Anomaly Detection\", Ruff et al., ICLR 2020\n- \"Detection Semantic Anomalies\", Ahmed & Courville, AAAI 2020\n- \"Transfer-Based Semantic Anomaly Detection\", Deecke et al., ICML 2021\nThese manuscripts investigate the presence of multiple classes in the normal distribution. While the focus is different (latent classes), it should be compared against in this work.\n\n# Update\n\nThe authors have addressed points raised in my original review by adding ablations and contrasting their proposed setting to existing ones. Some concerns remain around the novelty of the proposed approach.\n\nThe score has been increased to reflect the newly incorporated changes.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "KhhcvZxEFE", "reviewer": "Reviewer_JpFX", "summary": "This paper aims to learn a unified framework for detecting multi-class anomalies. Anomaly detection is only trained on normal data, so the so-called “identical shortcut” phenomenon may occur. To solve this problem, this paper proposes the following strategies: 1) layer-wise query decoder, 2) neighbor masked attention, 3) feature jittering. The experimental results on the MVTec-AD and CIFAR-10 datasets show that the proposed method can alleviate the “identical shortcut” phenomenon.\n", "questions": "1. On lines 125-126, the author mentions that the model may learn a trivial solution, causing anomaly detection to fail. But in MLP, this should not be the case due to the presence of nonlinear activations. Can the authors explain this further through experiments or visualizations?\n2. In Table 2, for the categories Capsule and Transistor, the results of the method DRAEM (50.5 and 64.5) are significantly smaller than those of the method in this paper (98.5 and 97.9), but for the category Zipper, the results of the method DRAEM (98.3) are higher than those of the method in this paper (96.8). What are the reasons for the above phenomenon? Please analyze this.\n3. Please analyze the complexity of the method in this paper compared with the one-class-one model method.\n", "limitations": "The authors have adequately addressed the limitations and potential negative societal impact of their work. ", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n1. This paper proposes a novel neighbor masked attention.\n2. This paper is well organized, easy to understand, and clearly written.\n3. Good results are achieved on the MVTec-AD and CIFAR-10 datasets.\n\nWeaknesses\n1. The introduction to evaluation metric (AUROC) is missing.\n2. The ablation experiments in Table 4 are incomplete. For example, under 1 q., the results in the case of only NMA and FJ are missing; under Layer-wise q., the results in the case of only NMA are missing; under Layer-wise q., the results in the case of only FJ are missing.\n3. Feature jittering is a common practice and lacks novelty.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}, {"review_id": "Cp5IcunPYL2", "reviewer": "Reviewer_npjB", "summary": "The paper tackles anomaly detection of multiple classes without class labels. In other words, the proposed method learns the normality of multiple classes at once without the need for class label information. The paper analyzes that the reconstruction-based anomaly detectors learn ‘identity shortcut’ and introduces techniques how to prevent this phenomenon. To this end, the paper proposes three techniques: the use of query embedding in multiple layers of a transformer, neighbor masked attention and feature jittering. The experiments are conducted on MV-Tech and CIFAR10 datasets.", "questions": "How did the proposed method and competing methods are trained for the unified and separate case? In the separate case, are they trained on the whole dataset and finetuned for each class-wise dataset? It is unclear how the models are trained in each scenario.", "limitations": "Limitations are addressed in Section5.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strength\n\n### Problem setup\nAnomaly detection on a multi-mode dataset (multi-class dataset without class information) is a relatively underexplored area. Most of the out-of-distribution detection papers assume class information given. In addition, this paper targets anomaly localization tasks.\n\n### Analysis and extensive ablation study supports the idea\n- Section 3.1 shows the performance deprecation over the training epochs showing that reconstruction-based models’ performance is unstable during training (the phenomenon of identical shortcuts).\n- Section4.5 includes extensive ablation studies supporting the design of the method and sensitivity in each component and hyperparameters. Most hyperparameters are insensitive to performance showing less than ~1% gap.\n\n### Strong performance\nThe proposed method shows a notable performance gap over competing methods in a unified scenario on MV-tech and CIFAR 10.\n\n\n## Weakness\n\n### Lack of analysis\n\n- The design of the method is not targeted for a unified case but is effective. Why?\nThe idea of the proposed method uses general ML techniques not tailored for unified (multi-class data without label information) cases, yet effective. What would be the main reason for this? How is the problem of learning identical-shortcut relevant to the performance of unified anomaly detection scenarios?\n\n- Why does the proposed method not gain performance when label information is added? (Separate case) In Table1 and Table2, the proposed method does not improve much by adding label information. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept