MMRR: Unsupervised Anomaly Detection through Multi-Level Masking and Restoration with Refinement
Authors: Jaesung Ahn
Janghyeon Lee
Hanbyel Cho
Yooshin Cho
HyeongGwon Hong
Junmo Kim
Venue: NeurIPS 2022 Submitted (2022)
Keywords: Deep Learning
Computer Vision
Anomaly Detection

Abstract: Recent state-of-the-art anomaly detection algorithms mainly adopt generative models or approaches based on deep one-class classification. These approaches have hyperparameters to balance the adversarial framework of the generative adversarial network and to determine the decision boundary of the classifier. Both methods show good performance, but their performance suffers from hyperparameter sensitivity. A new category of anomaly detection methods has been proposed that utilizes prior knowledge about abnormal data or pretrained features, but it is more generic not to use such side information. In this study, we propose "Multi-Level Masking and Restoration with Refinement (MMRR)", an unsupervised-learning-based anomaly detection method based on a generative model that overcomes hyperparameter sensitivity and the need for side information. MMRR learns the salient features of normal data distributions through  restoration from restricted information via masking, resulting in a better restoration of in-distribution data than out-of-distribution data. To overcome hyperparameter sensitivity, we ensemble restoration results from information restricted to predefined multiple levels instead of finding a single optimal restriction level, and propose a novel mask generation and refinement method to achieve hyperparameter robustness. Extensive experimental evaluation on common benchmarks (i.e. MNIST, FMNIST, CIFAR10, MVTecAD) demonstrates the efficacy of the MMRR.

MMRR: Unsupervised Anomaly Detection through Multi-Level Masking and Restoration with Refinement
Anonymous Author(s)
Affiliation
Address
email
Abstract
Recent state-of-the-art anomaly detection algorithms mainly adopt generative mod- els or approaches based on deep one-class classification. These approaches have hyperparameters to balance the adversarial framework of the generative adversarial network and to determine the decision boundary of the classifier. Both methods show good performance, but their performance suffers from hyperparameter sen- sitivity. A new category of anomaly detection methods has been proposed that utilizes prior knowledge about abnormal data or pretrained features, but it is more generic not to use such side information. In this study, we propose "Multi-Level Masking and Restoration with Refinement (MMRR)", an seappervisdd-caaninng based anomaly detection method based on a generative model that overcomes hyperparameter sensitivity and the need for side information. MMRR learns the salient features of normal data distributions through restoration from restricted information via masking, resulting in a better restoration of in-distribution data than out-of of-distribution data. To overcome hyperparameter sensitivity, we ensemble restoration results from information restricted to predefined multiple levels instead of finding single optimal restriction level, and propose a novel mask generation and refinement method to achieve hyperparameter robustness. Extensive exper- imental evaluation on common benchmarks (i.e., MNIST, FMNIST, CIFAR10) 10, MVTecAD) demonstrates the efficacy of the MMRR.
2
4
6
8
10
11
12
13
14
15
17
18
19
Introduction
20
21
Anomaly detection tackles the problem of detecting abnormal data with a distribution that is signifi- cantly different from normal data. It is an important task that enables machine learning algorithms to cope with unexpected distribution Anomaly detection problems are formulated assuming the unavailability of abnormal data during the training process; therefore, anomaly detection models cannot be trained for the original purpose of anomaly detection. With same context, it is impossible to validate in advance whether proposed model performs anomaly detection well during the training process. This means that even the anomaly detection ability of the model is significantly affected by the hyperparameter values, it is impossible to find the optimal hyperparameter value through validation. Therefore, method with a robust anomaly detection performance is necessary that does not include hyperparameters that have significant influence on anomaly detection performance.
23
real-word tasks such as self-driving or medical imaging.
24
25
27
29
31
Three deep-leaming-basee leading strategies have been proposed to solve anomaly detection. The 33 first using methods based on generative model which perform anomaly detection based on efficiency of the proposed generative models in restoring data. Early generative-model -basdd methods 35 failed in the anomaly detection task owing to the good generalization capability of the autoencoder inspired by generative
37 22 Furthermore, to solve this problem, many studies 39 36
30
Submitted to 36th Conference on Neural Information Processing Systems (NeurIPS 2022). Do not distribute.

---

37
adversarial networks (GANs) [15] have attempted to create autoencoders that can only restore normal 38 data by limiting the generalization capability using an adversarial concept. The second leading
39 strategy is using deep one class classification methods [20 35 16 21 which try to find the smallest
40 hypersphere surrounding only normal data unsupervised manner. However, generative model-based methods that try to restore only normal data well and deep one class classification methods that
41
42 try to find hypersphere surrounding only normal data have hyperparameters that have a significant
43 impact anomaly detection performance. We define hyperparameter sensitivity problem as having
44 hy perpaarraeeerss that significantly affect performance even in the nature of the anomaly detection field where abnormal data is not available.
46
The third leading strategy is using cele--nfoommaion-bbee about the difference between normal data and abnormal data [T7 12 18 3 45
methods, which utilize prior knowledge
47
25 47 or
48
utilize features 4 28 38
5 3
obtained from pretrained networks. Side-information based methods
49
have shown good performance on many benchmark datasets, but it is not common to know side informations that can help distinguish normal data from abnormal data, In addition, these methods suffer from massive performace degradation in setting where used side information is not applied well.
50
51
52
58
In this paper, we propose a novel method, Multi-Level Masking and Restoration with Refinement (MMRR) that does not use side information, is based on generative model, and avoids the hyperpa- rameter sensitivity problem. The motivation behind our proposed method is that network trained to restore normal data from limited information about normal data will learn the salient features of normal data. So that restoration from limited information succeeds for normal data and fails for abnormal data, which makes possible to perform anomaly detection in terms of restoration. To this
54
55
56
57
58
59 end, our method consists of the following two key components. First, masking, which is a process that uses a mask to obtain restricted information by restricting the remaining information except for
60
61 the parts essential for restoration. Second, restoration, which is the process of restoring original data
62 by using only the restricted information obtained through masking.
63
For MMRR to perform anomaly detection, it is necessary to find the optimal masking level that causes normal data be restored successfully and restoration of abnormal data to fail: masking level is the degree to which the mask limits information. However, to avoid the hyperparameter sensitivity problem caused by the absence of abnormal data during training, we detected anomalies through ensembles at multiple masking levels rather than finding a single optimal masking level. Our novel mask generation method made possible to ensemble at multiple masking levels by enabling the manual control of the masking level of the mask, which eliminated the need for adversarial loss. In addition, our mask generation method made the mask learnable such that the mask most helpful for restoration at the corresponding masking level was generated, which led to better anomaly detection performance.
64
85
66
67
68
70
71
72
73 However, our masking method compares the degree of restoration at the same masking level without considering the complexity of each data. Therefore, masking and restoration alone often restores simple abnormal data better compared with complex normal data, in which case anomaly detection fails. To solve this problem, we propose an additional refinement process that eliminates the difference in restoration caused by the difference in data complexity. Our contributions are as follows:
74
75
76
77
Hyperparameter robustness and Prior knowledge-free. We resolve the hyperparameter sensi- tivity problem that previous studies had overlooked with the proposed Multi-Level Masking and Restoration. Also, we have empirically shown through experiments that Multi-Level Masking is robust to hyperparameters. Furthermore, our method doesn't 't need any prior knowledge.
79
80
81
Experiments on benchmark datasets. Unlike existing studies, MMRR does not strive to obtain optimal anomaly detection by solving the hyperparameter sensitivity problem. Nevertheless, we introduced Refinement Network considering the intrinsic complexity of data, and obtained comparable performance to SOTA approaches.
83
84
85
2
Related Works
Classical methods proposed to solve anomaly detection include PCA 19 OC-SVM 40 SVDD
42). iForest (66.. and KDE 77. Most of them perform anomaly detection using hand-crafted simple functions. However, advancements in deep learning have made it casier to obtain richer and more
88

---

90
complex features of data, and thus many deep-learning sraniing-baeedddd anomaly detection studies have been conducted. The following three strategies are widely used deep-learning-bese based anomaly detection tequniques.
91
93
Generative-model-basse methods. Methods based on generative model begin with the assumption that the generative model trained only with normal data will fail to restore abnormal data. However, Sakurada and Yairi [37] and An and Cho [2] have demonstrated that a, simple autoencoder and variational autoencoder sufficiently restore abnormal data, thereby leading to the failure of anomaly detection. Therefore, various autoencoders for anomaly detection have been proposed that perform certain tasks, such as denoising (36) and inpainting [48). Also, there are studies that [8] 23 used backpropagation to measure the distance from the manifold of the data. Many generative-model-aas
95
96
97
98
99
100
methods are inspired by GAN and adversarial training. Some previous studies assumed networks that learned normal distribution through adversarial training would not be able to restore abnormal data or classify them as fake data 39 I Some studies have highlighted that autoencoders have good generalization capabilities and tried to design autoencoders that have limited resotraion capability by limiting the latent space through adversarial loss (1) or by prototyping the latent space [14 20]. However, most senerative-model-boe
101
102
103
104
105
methods suffer from the hyperparameter sensitivity problem because they have to find the optimal point that balances adversarial losses and other losses to obtain the best anomaly detection performance, which is impossible because of the absence of abnormal data.
106
107
108
109
Deep one-class classification methods. Since anomaly detection cannot use abnormal data for training. it is difficult to design a classifier that distinguishes between normal data and abnormal data. Ruff et al. (35] proposed a deep learning solution called SVDD [42 that seeks to find the smallest hypersphere surrounding normal data. They used various constraints to prevent representation
110
111
112
113
collapse due to the absence of abnormal data during the training process. Hu et al. (21) proposed a constraint called holistic regularization to prevent representation collapse. Some studies have
114
115
artificially generated abnormal data for training one-class classifiers. Goyal et al. [16] obtained,
116
artificial abnormal data through adversarial search, and Pourreza et al. [32) utilized data generated
117
from immature generator as abnormal data. Methods based on deep one class classification suffer from the hyperparameter sensitivity problem as there are variables that significantly influence the
118
119
performance of anomaly detection, such as the radius variable in Goyal et al. [66]
120
Side-information-base based methods. Self-supervised methods utilize prior knowledge of the dif- ferences between normal and abnormal data. For example, some studies [13 [8] 3 41] focused on differences in terms of geometry. Golan and El- 113 assumed that a network can learn the geometric features of normal data through a learning process that predicts the geometric transsforma-
121
122
123
124
tions applied to normal data. They expected the that trained transform classifier will fail predict abnormal data with different geometric characteristics compared with normal data. Based on this study, method restore transformed data [1] and methods that combined geometric concept with Other self-supervised methods augment normal data to create synthetic abnormal data and use them to train networks that can detect locally defect areas 25, 47). However, as mentioned in Goyal et al. [16), these methods rely heavily on prior knowledge. Some studies have attempted to perform anomaly detection using features obtained from
125
126
127
constructive learning 6 were proposed [
128
129
130
pre-trained networks using external data. 4 28
3
Multi-Level Masking and Restoration with Refinement
132
The overall framework of the proposed method is shown in Fig. In this section, we provide detailed description of our method called Multi-Level Masking and Restoration with Refinement (MMRR). We describe the Multi-Level Masking and Restoration procedures that restrict the informa- tion in a given input, and finally the Refinement that further improves the restored image.
134
135
136
137
3.1
Multi-Level Masking
Masking is a process that restricts embedding e E Rd, which is generated through embedding cetwor([fE : IRd + (d)) tanh(ff(x)), by using mask m. The masking process is
138
ē= ② + ① (1 m),
(I)
3

---

Multi-Level Masking
Restoration
Refinement
(a.
Refinement Network
Network
:
D
Embedcling
D
Restoration
D
D
Network
Network
+
e
D
Figure 1: Overall framework of MMRR. Given data H, the embedding network fE generates embedding e. The embedding thus generated is limited through a mask m with a masking level Mmm generated through the masking network fM, and using only such restricted embedding, the restoration network fies performs the restoration of the original data z. Finally, the refinmmen network fiet complements the part not restored where restoration has inevitably failed due to the intrinsic complexity, which allows MMRR to perform anomaly detection only with the intended difference caused by masking and restoration.
140
where @ is the Hadamard product, and is noise sampled from uniform random noise ~ U(-1,1)) The output of the masking process, called restricted embedding.
141
142
We masked e instead of directly masking x because the training process using only normal data will make fE generate e, which helps in restoration. Thus, using e will enable our proposed masking and restoration method to have a better discrimination ability. Noise is used because, without irrespective how small is, trivial solution that can easily restore data generated because is learnable. For the same reason, tanh was used to create to prevent trivial solution that makes restoration easier by making the value of significantly different from the noise value.
143
144
145
146
147
148
We can easily infer from Eq. | that if the value of m become smaller, the portion of embedding e in decreases and becomes noisy, and restoration becomes harder. For example, if all elements of m are O, will resemble uniform noise ((-1, 1)d, and restoration will be impossible. Therefore, we consider that the average value of
149
150
represent the difficulty of restoration from and define it as a masking level pm = o Ei mi, where m, refers to i-th element of m and jm E [0, 1].
152
The restricted embedding ē Fd should meet two conditions for masking and restoration detect anomalies: normal data should be successfullt restore and abnormal data should not be restored. To accomplish the goal, we need to find m with a /'m that can best differentiate normal and abnormal data in terms of restoration. However, we cannot find an optimal masking level /m that best distinguishes abnormal data from normal data. This is because abnormal data cannot be used in the training process owingto the nature of the field of anomaly detection.
153
154
155
156
157
159
Therefore, we decided ensemble the ability distinguish multiple masking levels Pm which uniformly distributed between and 1. For example, if we use L levels of masking for the ensemble,
180
181 Um
{0. - - 1} will be used.
Our novel mask generation method made it possible to manually adjust the pm of for multilevel ensemble. Furthermore, the novel mask generation made m learnable such that is generated in a direction that is most useful for restoration from the corresponding Mm. which improves the ability to distinguish between normal data and abnormal data.
163
165
Mask generation method. We propose novel mask generation method that can generate mask m with masking level um by m = offM(() )) + b), where fM : RR - Rd is the masking network. The goal of our mask generation method is to find the appropriate bias b IR that makes the average
187
value of the mask to a predefined pm as follows: 1 E (fM + b) Pms where pm is on
169
interval (0. 1] because m E [0, 1]d. As sigmoid a is monotonically increasing function, we can use the root-finding method (in case, the bisection method) to find bias b that satisfies the condition,
170

---

172 which allows us to successfully generate mask m with masking level pmn While the root finding
173 method is con-differential, the gradient to the output of far was obtained under the assumption that the bias satisfying the condition was well found, which is as follows:
174
Oc
Σ Vj
Oc 0mj my (1-))))
(0(3.5)
Ex m1(1 m1 - m 8(i.j) (0,
m4 m,)
[1,
if
(f)
otherwise
(2)
3.2
Restoration
178
Restoration refers to the process in which restricted embedding è is restored to original data æ via the
restoration network(Sss: IRd -> IRd), where the restoration output is & tanni (?)). Owing to
177
178
the mask generation method, not only fres but also fE and fM can be trained with only the simple restoration loss, which is formulated as:
179
Les 7 Σ
(z; x2)²
180
Sres. which trained using a training dataset consisting of only normal data, learns how to restore normal data from ē. In such training process, fres will learn salient features for normal distribution
182
pt. The features for normal distribution p² obtained in this way will allow fies to restore normal
data efficiently even when the masking level jmm is small.
184
While fres has been able successfully restore normal data as mentioned above, this way of restoration will fail for abnormal data. The reason is, fres has no choice but to generate output that resembles normal data because fres will also apply learned features for p* even when restoration is performed from é of abnormal data. This failure to restore abnormal data will allow the masking and restoration method to detect anomalies through the restoration loss.
185
186
187
188
189
3.3
Refinement
Our masking and restoration method resolves the hyperparameter sensitivity problem by ensembling the anomaly detection performance at multiple masking levels Pm E {0, -...... 1].
191
However, comparing the degree of restoration the same pmm without considering the characteristics of the data causes another problem. This is because the degree of restoration is intrinsically different even if is restored from the same fmm because different x have different complexities.
192
194
195
Let us assume that the restoration loss obtained from masking and restoration is composed of two losses. The first loss is caused by the inevitable restoration failure due the intrinsic complexity of I, which is denoted as intrinsic loss. The second loss occurs when abnormal data are restored like normal data owing masking and restoration, which is denoted abnormality loss. We originally intended to perform anomaly detection based on this abnormality loss.
196
197
198
199
This problem occurs when the abnormal sample is relatively simple compared with the normal sample. In this case, the sum of the intrinsic loss and the abnormality loss of the abnormal sample can be smaller than the intrinsic loss of the normal sample, which leads to the anomaly detection failure of the masking and restoration method.
201
202
203
To address this problem, the refinement method aims to eliminate the intrinsic loss that inevitably occurs due to intrinsic complexity difference so that anomaly detection can be performed only with the abnormality loss caused by masking and restoration process. For this, the refinement network fia : Rd x Rdd x Rdd x IRd + Rd predicts x - that have not yet been restored at a particular pm as follows: fee(I e, m, m,c). fiet is trained with refinement loss formulated as:
205
207
d (#r + r:))
Leel Σ
(x;
209 3.4 Training and Evaluation
210 Training. Our method consists of two-step training process. The first phase is training process for masking and restoration. During this phase, the masking level pm is uniformly sampled, where
5

---

212 Mm ~ 2((0, 1). The embedding network fg, masking network fMr restoration network fres are trained only with restoration loss. Furthermore, we selected the model with the smallest restoration loss for the validation data. The second phase is a training process for refinement. To this end, networks that
213
214
215
have been trained in the first phase are used with fixed weights. P'm is sampled from (1, 1) as in first phase. The refinement network fret is trained with only the refinement loss. Furthermore, we selected the model with the smallest refinement loss for the validation data.
216
217
Evaluation.
For evaluation, we must first determine the number of pm required. If we decide to use L masking levels, we must use {0, ... 1} masking levels that are distributed evenly at 1/(L-1 - intervals for evaluation. Finally, we perform anomaly detection by summing the refinement loss all masking levels for ensemble.
219
220
4
Experiment
222
4.1
Experimental Settings
224
To validate the proposed anomaly detection method, MMRR using multi-class datasets (MNIST 24 FMNIST 44), CIFARIO (22),, which is not designated for anomaly detection, we used the one-vs-all strategy. The one-vs-all strategy selects one normal class 1 S c S C among C' different classes. For training, we only used the training set belonging to class c. For testing, the normality score was calculated for all the data in the test set, the extent to which normal data and abnormal data are distinguished in terms of the normality score was measured using the area under receiver operating curve (AUROC). This process was repeated for all classes C' to evaluate the anomaly detection model. On the other hand, in the case of the MVTecAD dataset, for each class the train dataset consisting of only normal data and the test dataset mixed with abnormal data are already prepared. Therefore,
226
228
230
232
233
we trained using only the train data as given material, and used the test dataset in the test process.
234
Implementation details. All proposed networks were implemented using the U-Net 34 based on the wide residual (6) blocks proposed for wide residual networks. We used group normalization for 236 all blocks. For 32x32 datasets, we used four feature map resolutions(32x32 to 4x4). For 256x256 datasets, we used five feature map (e561 16x16). We used two wide residual blocks
237
238
that consisted of convolutions with 128 output channels for each feature map resolution. RAdam (2 was used as the optimizer with learning rate of 0.0001. Batch size was set as 64 and 4 for the 32x32 and 256x256 datasets, respectively. The learning was decayed by a factor 0.5 if the validation loss did not decrease for 500 epochs. We split the normal training set into training and validation sets using a 95:5 ratio, and used the validation set to select the model with smallest validation loss.
239
240
241
242
243
4.2
Datasets and Results
244
Baseline Methods. For anomaly detection in multi-class datasets, we compared MMRR with classical approaches such as: OC-SVM 44), and KDE [29]: generative-model--aee approaches such
245
246
as: AnoGAN (39], OCGAN 330 7 AA,, TT] and CAVGA,, 43]: deep one-class classification
247
approaches such as: DSVDD (35 and DROCC [16]. For anomaly detection on MVTecAD dataset,
248
we compared our MMRR with vanilla autoencoder AE, AE with skip connectins AE+skip, variational
249
autoencoder VAE, Ganomaly M. MemAE [4) CAVGA,, and DAAD (20].
250
MNIST includes a training set of 60,000 examples, and a test set of 10,000 examples. The data are 28x28 handwritten digits(0-9). For simplicity they were resized to 32x32. It was used for training without any augmentations except resizing. Our MMRR model achieved averaged AUROC of 0.967, which is slightly lower compared to SOTA methods. The reason our model has slightly poor performance on the MNIST dataset is that the data have a very easy distribution, so reconstruction occurs well enough even at a very low masking level pmm For example, in Figure
252
254
S we can see that the digit 0 is restored well enough even if fmm is 0.01. As such, if there is already a sample that can be restored well in the masking and restoration stage of very low Mmn it can be seen that refinement has a limit in solving this problem.
256
258
259
FMNIST consists of a training set of 60,000 examples, and test set of 10,000 examples, full of 10 different types of fashion items. For simplicity they were resized to 32x32. It was used for training without any augmentations except resizing. MMRR greatly beats the existing SOTA performance of 0.885 AUROC of CAVGA.
261
262

---

263
CIFARIO consists of 60000 32x32 color images in 10 classes, with 6000 images per class. There are 50000 training and 10000 test images. The dataset was used for training without any augmentations.
264
265
As shown in Table 5 our method achieved an average AUROC performance of 0.737 on the CIFARTO dataset, which comparable to that of other SOTA methods: 0.742 for DROCC and
266
267
0.737 for CAVGA. Moreover, the performance obtained by our method is meaningful because it is
268
obtained without experiencing hyperparameter sensitivity problem unlike other SOTA methods.
MVTecAD is a dataset for benchmarking anomaly detection methods with a focus on industrial 289
270
inspection. It contains over 5000 high-resolution images divided into 15 different object and texture categories. Each category comprises a set of defect-free training images and a test set of images with a variety of defects as well images without defects. We resized all the data to 256x256. We performed two tasks on the MVTecAD dataset, image-level anomaly detection and pixel-level anomaly localization. Experimental results on MV dataset can be seen in Table 7 MMRR achieved average 0.865 AUROC for pixel-level anomaly detection and 0.844 AUROC for image-level anomaly detection, which is close SOTA methods. We found that among the test defect-free data in the screw class, there were samples with different distribution in terms of brightness compared to the train defect-free data. Therefore, we trained MMRR by applying brightness augmentation to the train data, and a result of 0.95 AUROC was obtained in the image-wise wagee-wie anomaly detection. However, we did not report the performance because we assumed that we do not know the distribution of the test data.
271
272
273
274
275
278
277
278
279
280
282
Hy perparaaeee sensitivity. As we mentioned earlier, most of the generative model based methods and deep-one class classification based methods have hyperparameter sensitivity problem. For example, DROCC [16] showed how sensitively the performance changes according to the radius value, which a hyperparameter that they used to obtain negative samples. Anomaly detection performace of DROCC CIFARIO 10 dataset fluctuates between 0.7-0.8 for airplane, 0.5-0.7 for deer, and 0.7-0.8 for trucks in terms of AUROC depending on the radius value. There- fore, they carefully searched for the radius value to obtain optimal anomaly detection performance. In addition to this, Akçay et al. [1] showed that the performance of their proposed model is sensitively changed according to the values of three hyperparameters that balance their losses in the CIFARIO dataset. Also, Hou et al. (20) showed that the anomaly detection performance in MVTecAD dataset fluctuates between 0.716-0.82. based on the value of division rate(r,&er, that determines the size of the query.
0.55
283
285
0.75
AUROC
287
0.65
289
Doga
0.6
291
0.55
293
08 Musking- Level (Pm))
Figure 2: Illustration of AUROC with re-
295
spects to the number of Masking Level (Ho.) used for MMRR CIFARIO dataset.
297
298
289
However, MMRR uses only loss for each training phase. And we provide a clear criterion for model design: selecting the model with the lowest loss on validation data. Furthermore, we show how the performance of MMRR changes according to the only hyperparameter that significantly affects our performance in the Fig. 2 From Fig. 2 can be seen that the performance of anomaly detection improves as the number of masking levels used for evaluation increases.
301
303
Prior knowledge. It has been shown in Goyal et al. [66] that the internnformation based meth- ods mentioned in Section 2 relies heavily on the
GEOM
MMRR w/o ref.
MMRR
306
w/o aug.
0.86
0.676 0.682
0.737
307
308
prior knowledge they used. To prove this, they applied flips and small rotations of angle + to CIFARIO data during training. As can be seen
w/ aug. 0.691
0.7
309
Table 1: Comparing AUROC against GEOM on CIFARIO dataset with training data augmentations (rotation +330 and flips).
310
311
in Table there was a large decline in the perfor-
312
mance(0.86 to 0.691) 1) of the Golan and El- Yaniv
313
that used prior knowledge. On the other hand, mance (0.676 to 0.682), and MMRR showed 0.037 lower performance (0.737 to 0.7).
MMRR w/o refine showed rather good perfor-
314
315
4.3
Ablation Study
7

---

MNIST
OC-SVM
KDE
AnoGAN
DSVDD
OC-GAN
CAVGA
MMRR w/o ref.
MMRR
0.988
0.885
0.966
0.98
0.998
0.994
0.9857
0,9941
0.999
0.996
0.992
0.997
0.999
0.997
0.999
0,9982
0.902
0.71
0.85
0.917
0.942
0.989
0.8981
0,94
0,95
0.693
0.887
0.919
0.963
0,983
0,9246
0,955
0,955
0.844
0.894
0.949
0.975
0.977
0,9309
0,9352
0,968
0.776
0.883
0.885
0.98
0.968
0.9173
0,971
0.978
0.861
0.947
0.983
0.991
0.988
0,9765
0,989
0,965
0.884
0.935
0.946
0.981
0.986
0,9539
0,966
0.853
0.669
0.849
0.939
0.939
0.988
0,906
0,945
0.955
0.825
0.924
0.965
0.981
0.991
0,9511
0.98
Table 5: Image-level AUROC for one-vs-all anomaly detection on MNIST.
316
Embedding. To prove the effectiveness of using embedding e, directly masked the data As can be seen from the table, got an average AUROC of 0.6449 the CIFAR 10 dataset when not used. And 0.6449 AUROC is far lower than 0.737 AUROC, which is the performance obtained when e is used. Through these results, it can be seen that fE learned salient features for normal data in the training process of generating e, which is most helpful for restoration even though it is restricted by masking. And the embedding e generated from fø can be seen to have a positive effect on the anomaly detection performance by widening the restoration gap between normal data and abnormal data.
w/o ref.
w/ ref.
318
w/o emb. w/ eeebb
0.642
0.6449
0.676
0.737
320
Table 2: AUROC performance of MMRR w/o and w/ embedding net- work CIFARIO.
322
323
324
325
326
Mask generation method. We proved the effective- ness of our learnable mask by comparing it with other simple masks which are unable to learn. The first mask mask in which all elements have the same constant value jm. and we will call a constant mask. The second mask to be compared is mask generated by bernoulli sampling with a probability of pmm- When we
328
330
(a) From left to right, data, constant mask, bemoulli mask, our mask
332
Constant
Bernoulli
Ours
used the constant mask, we got AUROC performance of 0.667, and when we used the bernoulli mask, we got 0.6478. These are lower performances when compared to 0.737 obtained by our mask generation method. From the experimental results, it can be seen that the use of
w/o ref.
0.619
0.612
0.676
334
335
w/ ref.
0.674
0.648
0.737
336
Table 3: AUROC according to mask genera- tion method on CIFARIO
337
338
a our multi-level mask that can learn to leave informa-
339
tion which most helpful for restoration at a specific
340
masking level during the masking process also helps anomaly detection.
Refinement. As can be seen from the Table 4 there is a big difference 342
MNIST
FMNIST
CIFAR10
MVTecAD
between MMRR with refinement and MMRR without refinement. In the case
w/o ref. w/ ref. 0.967
0.944
0.928
0.676
0.825 / 0.861
344
0.93
0.737
0.840 / 0.865
of MNIST dataset, average auroc im- proved by 0.033 from 0.944 to 0.967.
Table 4: AUROC w/o and w/ refinement module on MNIST, FMNIST, CIFARIO, and MVTecAD, Image-wise / Pixel-wise AUROC performance reported on MVTecAD.
346
And for CIFAR10 dataset, average rOC improved by 0.067 from 0.68 to
348
0.747. Experimental results show another interesting phenomenon besides performance improve- ment. For example, data that has already had good anomaly detection performance in MMRR w/o refinement, such data beloning to airplane, deer, ship classes, does not improve significantly when refinement is applied as can be seen in Table 5 However, the data that performed poorly in the MMRR w/o refinement, such data belonging automobile, truck, showed remarkably large performance improvement. These results show that the intrinsic complexity difference between classes well resolved through the refinement as intended. However, as MVTecAD dataset the data proposed to detect local defect areas, the difference in intrinsic complexity between normal data and abnormal data is not large. Therefore, as can be seen from the Table the performance improvement due to refinement was insignificant.
350
352
353
354
355
356
357
358
8

---

CIFARTO
OC-VV
KDE
AnoGAN
ISVDD
OC-GAN
7-AEE
CAVGA
DROCC
MMRR W/O rel
MATR
Airplame
0.63
0.757
0.702
0.817
0.7778
Automobile
0.44
0.52
0.547
0.639
0.531
0.663
0.784
0.767
0.6065
0.7377
0.667
Cat
0.487
0.497
0.545
0.591
0.62
0.713
0.747
0.671
0.6076
0.6595
Dog
0.5
0.496
0,603
0.637
0.62
0.689
0.552
0.744
0.6143
0.6739
Frog
Horse
0.533
0.564
0.625
0.673
0.575
0.588
0.745
0.714
0.626
0.7037
Truck
0.508
0.54
0.665
0.731
0.554
0.744
0.741
0.762
0.6229
0.7325
Table 6: Image-level AUROC for one-vs-all anomaly detection CIFARIO.
Methol
carpet
gr50
leatber
Gice
wood
bottle
<ible
capsale
hazetout
:sctalior
toctibrrsk
transsserr
Ripper
AP
0.5%
01:00
0.751
0:476
0363
12:939
01:11
11:755
0.976
0.38
0.585
0.579
0.971
0.905
0.65
VAE
0.58
0.885
0.554
0.465
0.695
0.902
0.528
0.862
0.977
0.581
0.588
0.558
0.97]
0.851
RUS14
7- VAE,
0.72
0.919
0.581
MIMRR
0.7858
Gonomaay
0.66
0.708
0.843
0.754
0.334
03992
0.732
0.785
0.7
0.58
0.346
0.633
0.242
AE
0411
0.841
0.515
0.696
0.61
0.955
0.688
0
0.844
0.565
0.82
0.556
0.977
0.776
0.578
MomA
0454
11946
0.11
0.3
0.97
01964
0.664
0881
0.591
0.57
0.81
0.822
0972
0.793
0.871
AErskip
0.385
0.875
0.57
0966
0.77
0.713
0.579
0.747
0.528
0.538
0.53
0.742
0.749
0,996
0835
0.989
0.14
Enuge
0.566
0.522
0352
0.21
0.758
0.76
MEMER
0.981
MMMERR
0.466
0.908
0.7999
0.7693
0 9316
0.9995
09861
0.7535
0.187
03162
0,775
0.9794
0.960
0.70
Table 7: Pixel-level and Image-level anomaly detection on MVTecAD dataset.
.2
MN
PANNSTT
CIARID
MYRocl
MIST
RMNNST
Almormal
CIARIO
MV
Figure Qualitative results for normal and abnormal samples.
Conclusion
359
360
We proposed Multi-Level Masking and Restoration with Refinement (MMRR), which started from the motivation to perform anomaly detection through series of processes of information limitation and restoration. The most noteworthy point of this study that it presented the hyperparameter sensitivity problem for the first time, a problem that had been overlooked in existing anomaly detection studies. MMRR solved the hyperparameter sensitivity problem through ensemble at multiple masking levels with novel mask generation method. To empirically demonstrate the robustness to hyperparameter and prior knowledge- free properties of MMRR, we compared the performance as varying the number of masking level and augmentations. Additionally, we solved the problem of not considering the intrinsic complexity of data owing to the novel mask generation method through the refinement
362
363
384
385
366
367
368
369
module, and achieved comparable performance on MNIST, FMNIST, CIFAR 10, and MVTecAD
datasets. However, since we have to forward several times for ensemble in multi-level masking, it
has the disadvantage of being computationally expensive. We will go further here and try to find lightweight anomaly detection method without suffering from hyperparameter sensitivity problems.
371

---

References
373
[1] S. Akçay, A. Mtapour-Abarhhen and T. P. Breckon. Skip-ganomaly: Skip connected and adversarially trained encoder-docoder anomaly detection. In 2019 International Joint Conference on Neural Nenworks (IJCNN) pages 1-8, IEEE, 2019.
375
376
377
[2] J. An and S. Cho. Variational autoencoder based anomaly detection using reconstruction probability. Special Lecture on IE, 2(1):1-18, 2015.
378
[3] L. Bergman and Y. Hoshen. Classification based anomaly detection for general data. In International Conference on Learning Representations. 2019.
380
[4] L. Bergman, N. Cohen, and Y. Hoshen. Deep nearest neighbor anomaly detection. arXiv preprint arXiv:2002. 10445, 2020.
382
[5] P. Bergmann, M. Fauser, D. Sattlegger, and C. Steger. Uninformed students: Student-teacher anomaly detection with discriminative latent embeddings. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition. pages 4183 4192, 2020.
384
385
[6] Т. Chen, S. Kornblith, M. Norouzi, and G. Hinton. A simple framework for contrastive learning of visual representations. International conference on machine learning, pages 1597-1607, PMLR, 2020.
387
[7] R. A. Davis, K.S. Lii, and D. N. Politis. Remarks some nonparametric estimates of density function. In Selected Works of Murray Rosenblatt, pages 95-100. Springer, 2011.
389
390
[8] L. Deecke, R. Vandermeulen, L. Ruff, S. Mandt, and M. Kloft. Image anomaly detection with generative adversarial networks. In Joint european conference on machine learning and knowledge discovery in databases, pages 3-17. Springer. 2018
392
[9] T. Defard, A. Setkov, A. Loesch, and R. Audigier. Padim: patch distribution modeling framework for anomaly detection and localization. International Conference on Pattern Recognition, pages 475 489. Springer, 2021.
394
385
396
[10] D. Dehaene, O. Frigo, S. Combrexelle, and Elinc. Iterative energy- projectionn on a normal data manifold for anomaly localization. CoRR, abs/2002.03734, 2020. URLhhttss:/arxiiv.om org/ wttp:rg///aaaaaaaaee 2002
397
398
03734
399
[11] Y. Fei, C. Huang, C. Jinkun, M. Li. Y. Zhang, and C. Lu, Auribute restoration framework for anomaly detection. IEEE Transactions on Multimedia, 2020.
400
401
[12] S. Gidaris, P. Singh, and Komodakis. Unsupervised representation learning by predicting image rotations, arXiv preprint | 2018
402
[13] I. Golan and R. El-Yaniv. Deep anomaly detection using geometric transformations. Advances in neural information processing systems, 31, 2018
404
[14] D. Gong, L. Liu, V. Le. B. Saha, M. R. Mansour, S. Venkatesh, and A. v. d. Hengel. Memorizing normality to detect anomaly: augmented deep autoencuder for unsupervised anomaly detection. In Proceedings of the IEEÉ/CVF International Conference on Computer Vision, pages 1705-1714, 2019,
406
407
[15] I. Goodfellow, J. Pouget-Abadie, M. Mirza, Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio. Generative adversarial nets. Advances in neural information processing systems, 27, 2014.
409
[16] S. Goyal, A. Raghunathan, M. Jain, H. V. Simhadri, and Jain. Droce: Deep robust one-class classification. In International Conference on Machine Learning, pages 3711-3721. PMLR, 2020.
411
[17] D. Hendrycks, M. Mazeika, and T. Dietterich. Deep anomaly detection with outlier exposure. In International Conference on Learning Representations. 2018
413
[18] D. Hendrycks, M. Mazeika, S. Kadavath, and D. Song. Using supe-sup learning improve model robustness and uncertainty. Advances Neural Information Processing Systems, 32, 2019.
415
416
[19] H. Hoffmann. Kernel pea for novelty detection. Pattern recognition, 40(3):863-874, 2007.
417
[20] J. Hou, Y. Zhang. Q. Zhong. D. Xie, S. Pu, and H. Zhou. hivide-anddassemblle Learning block- wise memory for unsupervised anomaly detection. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 8791-8800. 2021.
418
420
[21] W. Hu, M. Wang, Q Qin, J. Ma, and B. Liu. Hr:: A holistic approach to one class learning. Advances in Neural Information Processing Systems, 33:19111- 13:19911-11124,,, 2020.
10

---

422
[22]
A. Krizhevsky, G. Hinton, et al. Learning multiple layers of features from tiny images. 2009.
423
[23] G. Kwon, M. Prabhushankar, D. Temel, and G. AlRegib. Backpropagated gradient representations for
424
anomaly detection. In European Conference Computer Vision, pages 206-226. Springer, 2020.
425
(24] Y. LoCun, L. Bottou, Y. Bengio, and P. Haffner. Gradient-based lcarning applicd to document recognition. Proceedings of the IEEE, 86( 1):228222324, 1998.
426
427
[25] C.-I..Li, K. Sohn, J. Yoon, and T. Pfister. Cutpaste: Self-supervised learning for anomaly detection and localization. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 9664-9674, 2021.
428
430
[26] F. T. Liu, K. M. Ting, and Z.-H. Zhou, Isolation forest. In 2008 eighth ieee international conference on data mining, pages 413-422. IEEE, 2008.
[27] L. Liu, H. Jiang, P. Hc, W. Chen, X. Liu, J. Gao, and J. Han. On the variance of the adaptive learning rate and beyond. CoRR, abs/1908.03266 03265, 2019. URL http://///aax weriiiiooo///ooe argg/bs///00 1t///999800000 03265
433
[28] P. Liznerski, I.. Ruff, R. A. Vandermeulen, J. Franks, M. Kloft, and K.-R. Müller. Explainable deep one-class classification. arXiv preprint arXiv:2007.01 2020.
435
[29] E. Parzen, On estimation of probability density function and mode. The annals of mathematical statistics, 33(3):1065-1000 1076, 1962.
437
[30] P. Perera, R. Nallapati, and B. Xiang. Ocgan: One-class novelty detection using gans with constrained latent representations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2898-2906, 2019.
439
440
411
[31] S. Pidhorskyi, R. Almohsen, and G. Doretto. Generative probabilistic novelty detection with adversarial autoencoders. Advances in neural information processing systems, 31, 2018.
442
443
[32] M. Pourreza, B. Mohammadi, M. Khaki, Bouindour, H. Snoussi, and M. Sabokrou. G2d: generate to detect anomaly. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 2003-2011 2021.
444
445
446
[33] T. Reiss, N. Cohen, L. Bergman, and Y. Hoshen. Panda: Adapting pretrained features for anomaly detection and segmentation. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 2806-2811 14, 2021.
447
419
[34] O. Ronneberger, P. Fischer, and T. Brox. U-net: Convolutional networks for biomedical image segmentation. CoRR, ab4/1505.04497, 2015. URI http://////axxii http:://arxiv.org/aba. aba/ 1100.1100.000 045997
451
[35] L. Ruff, R. Vandermeulen, N. Goernitz, L. Deccke, S. A. Siddiqui, A. Binder, E. Müller, and M. Kloft, Deep one-class classification. In International conference on machine learning, pages 2018.
PMLR,
453
[36] M. Sabokrou, M. Khalooei, M. Fathy, and E. Adeli. Adversarially learned one-class classifier for novelty detection. In Proceedings of the IEFE conference on computer vision and pattern recognition, pages 3379-3388, 2018.
455
456
457
[37]
Sakurada and T. Yairi. Anomaly detection using autoencoders with nonlinear dimensionality reduction. In Proceedings of the MLSDA 2014 2nd workshop on machine learning for sensory data analysis, pages 4-11, 1-1,,2115 2014.
458
459
460
[38] M. Salchi, N. Sadjadi, S. Baselizadeh, M. H. Rohban, and H. R. Rabicc. Multiresolution knowledge distillation for anomaly detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14902-14912, 2021.
461
462
463
[39] T. Schlegl, P. Sceböck, S. M. Waldstein, U. Schmidt-Frfuutth and G. Langs. Unsupervised anomaly detection with generative adversarial networks to guide marker discovery. In International conference on information processing medical imaging, pages 146-157. Springer, 2017.
465
466
[40] B. Schölkopf, J. C. Platt, J. Thywe-Taylor, A. J. Smola, and R. C. Williamson. Estimating the support of a high-dimensional distribution. Neural computation, 13(7):1443-1471, 2001.
467
[41] J. Tack, S. Mo, J. Jeong, and J. Shin. Csi: Novelty detection via contrastive learning distributionally shifted instances. Advances in neural information processing systems, 33:11833 111111889118822.2 11852, 2020.
469
[42] D. M. Tax and R. P. Duin. Support vector data description. Machine learning, 54(1):45 66, 2004.

---

471
[43] S. Venkataramanan. K.-C. Peng, R. V. Singh, and A. Mahalanobis. Attention guided anomaly localization in images. In European Conference on Computer Vision, pages 485-503. Springer. 2020.
472
[44] H. Xiao, K. Rasul, and R. Vollgraf. Fashion-mnist: novel image dataset for benchmarking machine learning algorithms. arXiv preprint 07470:7700.07747,, 2017.
474
475
[45]
Yi and S. Yoon. Patch svdd: Patch-level svdd anomaly detection and segmentation. In Proceedings of the Asian Conference on Computer Vision, 2020.
476
477
[46]
S. Zagoruyko and N. Komodakis. Wide residual networks. arXiv preprint 1rXiv:1605.07146, 2016.
478
[47] V. Zavrtanik, M. Kristan, and D. Skočaj. Draem-a discriminatively trained reconstruction embedding for surface anomaly dctection. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pages 1330-8339 2021.
479
[48] V. Zavrtanik, M. Kristan, and D. Skočaj. Reconstruction by inpainting for visual anomaly detection. Pattern Recognition, 112:107706, 2021.
482
Checklist
494
The checklist follows the references. Please read the checklist guidelines carefully for information on how to answer these questions. For each question, change the default [TODO] to [Yes]. [No], or [N/A] . You are strongly encouraged to include a justification to your answer, either by referencing 486
487
the appropriate section of your paper or providing a brief inline description. For example:
488
Did you include the license to the code and datasets? [Yes] See Section ??.
489
Did you include the license to the code and datasets? [No] The code and the data are proprietary.
490
Did you include the license to the code and datasets? [N/A]
Please do not modify the questions and only use the provided macros for your answers. Note that the Checklist section does not count towards the page limit. In your paper, please delete this instructions block and only keep the Checklist section heading above along with the questions/answers below.
482
493
494
1. For all authors...
485
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes] The contributions of our work are thoroughly are included in both abstract and introduction.
486
497
496
(b) Did you describe the limitations of your work? [Yes] The limitations of our work are mentioned in Section 5
(c) Did you discuss any potential negative societal impacts of your work? [N/A]
501
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
503
2. If you are including theoretical results...
504
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
506
3. If you ran experiments..
507
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either the supplemental material or as a URL)? [Yes] We include all
508
509
the information required to reproduce our experimental results in Section 4
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] All the training details are mentioned Section 4
511
512
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [Yes] We reported error bars in the tables in Section
513
514
(d) Did you include the total amount of compute and the type of resources used (c.g., type of GPUs, internal cluster, cloud provider)? [Yes] They are reported in Appendix.
4. If you using existing assets (c.g., code, data, models) or curating/releasnng new assets...
12

---

517
(a) If your work uses existing assets, did you cite the creators? [Yes] We cited all the creators for the assets we use throughout our puper.
518
519
(b) Did you mention the license of the assets? [N/A]
520
(C) Did you include any new assets either in the supplemental material or as URL? [No]
(d) Did you discuss whether and how consent was obtained from people whose data you're 're using/curating? [Yes] We use the data widely used the field related our work. (c) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
522
623
5. If you used crowdsourcing conducted research with human subjects...
526
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
528
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals. if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
531
13


Reviews:
[{"review_id": "JKmKig1LuO", "reviewer": "Reviewer_7z6Y", "summary": "This paper introduces a reconstruction-based anomaly detection algorithm. The algorithm is based on autoencoders, and noise is added in the form of masking of one of the embedding layers, which then has to be reconstructed to match the original input; input-specific masks are computed using a masking network, with a given average masking level. A further refinement network is implemented to correct the residual error from the reconstruction. At test time, predictions from various masking levels are ensembled and the reconstruction error post refinement is used as the anomaly score. Experiments performed on benchmark image anomaly datasets show competitive results with recent methods.\n", "questions": "- Does the root-finding algorithm find a solution to the equation in practice? Is the assumption used to derive the gradient in Eq (2) satisfied?\n\n- How does the method perform on tabular or other non-image datasets (e.g. ODDS datasets)?\n\n- How does the method perform relative to other masking-based autoencoders?\n\n- What is the performance with a uniform mask (see above)? \n", "limitations": "While I agree that the proposed method does eliminate the choice of the masking level hyperparameter, the autoencoder architecture to use and training hyperparameters still have to be tuned for a specific dataset, so I think it is a bit of an overreach to claim that the proposed method solves the tuning issue. To be fair this is an issue with many other methods as well.\n", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Overall, the paper presents a promising approach to anomaly detection but some aspects of the algorithm are not well motivated or validated, and the evaluation could be improved. Detailed comments follow.\n\n**Originality:**\n\nThis work extends a long line of reconstruction-based anomaly detection methods using autoencoders, with recent ones including noise and even masking but at the input layers (see references below). The key difference here is in implementing the masking at the inner embedding layers, and the use of a refinement network to further improve the reconstructions. From this perspective, the proposed method seems incremental. \n\n- One-Class Learned Encoder-Decoder Network with Adversarial Context Masking for Novelty Detection. WACV 2022.\n- Self-Supervised Masking for Unsupervised Anomaly Detection and Localization. IEEE Transactions on Multimedia 2022.\n\n\n**Quality:**\n\nOverall the approach and evaluation are reasonable, though there are several aspects that need to be addressed:\n\n- The paper does not motivate well the need to mask embeddings versus input layers (I did not understand the justification provided on L142), nor are comparisons provided to input masking autoencoder methods provided which could help to motivate the method.\n \n- The paper differentiates through the mask generation step by assuming that the average masking level condition is satisfied (Line 169). However, it is not shown whether this condition is indeed satisfied in the paper - this should be verified to ensure that the method is technically sound. \n\n- Experiments in the paper are limited to images, though the framework appears to be generic. It would be informative to include tabular datasets (e.g. from ODDS - http://odds.cs.stonybrook.edu/) so the generality of the masking strategy can be assessed. Moreover, \"side-information\" is not usually available for such data and would be better able to demonstrate the practical utility of the proposed method.\n\n- Given that the mask is in [0,1], the ablation study (L326) should also include masks with entries uniformly generated over [0,1].\n\n\n**Clarity:**\n- The paper is largely clear but there are several typos scattered across the paper (e.g. L23 real-word, L154 successfullt, L158 owingto, L173 non-differential).\n\n- It would be helpful to provide the percentage of anomalies in each dataset.\n\n- Just a suggestion: it may be clearer to revise the definition of masking to $\\tilde{e} = e \\odot (1-m) + \\epsilon \\odot m$, such that a higher m represents a higher masking level in that more of the input is removed.\n\n\n**Significance:**\n\nOverall, while the method is interesting and has some novel elements, the significance of the results is not clear:\n\n- In terms of the method, the refinement module is independent of the masking approach, and could also be applied to other autoencoder methods; the notion of masking is somewhat interesting and broadly applicable, but the lack of good performance makes its significance unclear.\n\n- In terms of performance, given that other better performing methods exist on the image datasets evaluated on, it is unclear if the method will be practically useful as other training/model hyperparameters still need to be tuned in practice. As mentioned above, it would be useful to demonstrate good performance on other datasets for which side-information is not available.\n\n- The claim that this is the first paper presenting the \"hyperparameter sensitity\" is not quite true, as this has been reported in the following workshop paper.\n\nThe Effect of Hyperparameter Tuning on the Comparative Evaluation of Unsupervised Anomaly Detection Methods. 6th Outlier Detection and Description Workshop @ KDD 2021.\n\n\n**Other comments:**\n\n- Given that some results are pretty close, it would be informative to provide results from multiple runs along with standard deviations.\n\n- In addition, providing results with the more appropriate AUPRC metric for anomaly detection due to the class imbalance would be informative.\n\n** Post rebuttal update: **\nAs described in the comment below, the response has addressed some of my key concerns regarding comparison to other masked-based anomaly detection methods and I am raising my score to a 5.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "GR-3oZurMIE", "reviewer": "Reviewer_UWD3", "summary": "This paper presents an unsupervised learning based anomaly detection method, which masks the images at multi-levels for restoration and then refine the restored images with refinement network. To overcome hyperparameter sensitivity, this work ensembles multiple restoration results from different level of masks and propose a novel mask generation and refinement method to achieve hyperparameter robustness.  Experiments are conducted on  several  benchmarks ", "questions": "What is the advantage of the proposed mask design method in comparison with the masking design in other works? Although the hypermeter issue is relieved to some degree, the restoration quality of designed mask scheme is worse than the peer works. \n\n", "limitations": "The paper didn't address the limitation and potential negative societal impact of the work.\n", "rating": 2, "confidence": 5, "soundness": 1, "presentation": 2, "contribution": 1, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths \n1.This work proposed a multi-level masking scheme for image restoration and ensemble restoration results to overcome hyperparameter sensitivity. \n2.The experiments are evaluated on multiple datasets.\n\nWeaknesses\n1.The novelty of this work is very limited. The mask-based restoration has been studied by search works, and they employed the ensembles of restoration results of different scale masks or different pattern masks.\n2.Some important related works are missing in the paper, and this work does not compare the AD performance with these works. E.g. “CFLOW-AD: Real-Time Unsupervised Anomaly Detection with Localization via Conditional Normalizing Flows”, “Learning Semantic Context from Normal Samples for Unsupervised Anomaly Detection” etc.\n3.The proposed method adopts a complicated framework including multiple restoration and refinement networks, but the AD performance on MVTEC and Cifar-10 dataset is far below the state-of-the-art.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "45s-v4cXP0j", "reviewer": "Reviewer_rzLV", "summary": "This work targeted the unsupervised anomaly detection problem and proposed a reconstruction-based method. Namely, by masking out features of the input data, and adding also noise to the features, the reconstruction and refinement network are only trained on the normal data to achieve small reconstruction errors. Therefore, any anomaly would generally suffer from high reconstruction errors. The evaluation is carried out in the one-vs-rest setup of the multi-class datasets MNIST, FMNIST, CIFAR10, and the defect detection benchmark MVTecAD. These benchmarks cover both semantic and non-semantic anomaly patterns, showing the covered use scenarios of the method.\n\nThe method's gain over SoTAs is quite diverse on the benchmarks. The authors argued an important advantage of the proposed method is its less sensitivity to hyperparameters, e.g., through an ensemble of different levels of masking, and the performance generally increases with more levels and is saturated at a certain level.", "questions": "1. AUPR evaluation: for an imbalanced evaluation set (the ratio of normal and abnormal samples is not balanced, AUPR is a more insightful metric to evaluate than AUROC.\n\n2. Analysis of the learned masking patterns: how they behave differently on different datasets. For instance, in the visual example Fig. 3, while the model is trained on ship of CIFAR10, it can still reconstruct the deer class. This is a known observation of reconstruction-based methods. The proposed method does not seem to provide a different observation. The reconstruction difference is mainly on the local pixels. This leads to the question if the network really exploits the semantic structure to mask and reconstruct. This may explain why the performance on CIFAR10 is quite diverse across different classes.\n\n3. On the MVTec benchmark, there are more competitive baselines, e.g., from https://paperswithcode.com/sota/anomaly-detection-on-mvtec-ad. \n\n4. On the F/MNNIST and CIFAR10, the authors mainly evaluated on the one-vs-rest setting. how this approach can handle multi-class settings, namely, the inlier dataset is multi-class.\n\n5. How well this approach can scale up to more diverse and higher-resolution datasets?\n\n6. More discussion or empirical analysis of the effectiveness of feature masking. For instance, on which feature dimension, how does it change over different input resolutions or types of anomaly patterns, the importance of adding noise vs. just putting zeros, why the learned mask does not select a shortcut, e.g., masking out every n pixels (like a downsampling) to achieve good reconstruction. ", "limitations": "The limitation discussion of this work is primarily focused on cases where the method is performing on a par or worse than baselines, and also on the ablation study part. However, as provided in the previous feedback sessions (weakness and questions), I think it is still unclear on which conditions/reasons the model is particularly good and in which scenarios it is expected to be underperforming. This is very important as so far the empirical results are quite diverse in terms of gains over SoTAs.", "rating": 3, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Reconstruction is one type of method to address anomaly detection. It is an unsupervised approach, thus requiring little knowledge of anomalies. However, compared to other kinds of methods, it is generally underperforming. One common observation is the reconstruction network can also reconstruct the anomalies. For instance, Masked autoencoders can reconstruct the input with a masking ratio of more than 75% on the input space (surely the work is proposed for self-supervised learning, not anomaly detection). The authors' method showed reasonable performance, through feature-level masking, adding noise, and additive refinement. \n\nHowever, the achieved performance is not very strong. First, it does not provide consistent improvement over different cases. On average, it is on a par with some existing methods. It can be important and interesting to discuss why the method is effective in some cases and less effective in others. \n\nSecond, the authors argued prior work is more sensitive to hyperparameter settings. It is not a strong argument in my opinion. All these methods are thresholding-based methods, including the proposed one. Thus, they all somewhat need some validation data to set the threshold before deployment, e.g., finding the value based on the expected false positive or true positive ratio. AUROC and AUPR are threshold-free evaluation metrics, but a threshold is still needed to really make a decision. In practice, it is probably not difficult to obtain some anomalies for validation. The important question is if the choice of hyperparameters is good for unseen anomalies at test time.\n\nThird, at the training time, the model does not exploit any prior knowledge about anomaly patterns to train the reconstruction. In other words, the model decides on its own which cue to use for learning the best reconstruction strategy, largely depending on the given dataset. In other words, it is also hard to predict the model reacts to which type of anomalies. The evaluated benchmarks mainly considered either semantic or non-semantic anomalies. What if construct both semantic and non-semantic anomalies for the same in-distribution dataset, i.e., CIFAR10-dog as the inliers, CIFAR10-others as outliers, CIFAR10-corruption or some other non-semantic shift as another type of outliers. \n\n ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Reject