Title: ReContrast: Domain-Specific Anomaly Detection via Contrastive Reconstruction
Venue: NeurIPS 2023
Accepted: yes
Keywords: Unsupervised Anomaly Detection

Contrastive Learning

Medical Anomaly Detection

Transfer Learning
Abstract: Most advanced unsupervised anomaly detection (UAD) methods rely on modeling feature representations of frozen encoder networks pre-trained on large-scale datasets, e.g. ImageNet. However, the features extracted from the encoders that are borrowed from natural image domains coincide little with the features required in the target UAD domain, such as industrial inspection and medical imaging. In this paper, we propose a novel epistemic UAD method, namely ReContrast, which optimizes the entire network to reduce biases towards the pre-trained image domain and orients the network in the target domain. We start with a feature reconstruction approach that detects anomalies from errors. Essentially, the elements of contrastive learning are elegantly embedded in feature reconstruction to prevent the network from training instability, pattern collapse, and identical shortcut, while simultaneously optimizing both the encoder and decoder on the target domain. To demonstrate our transfer ability on various image domains, we conduct extensive experiments across two popular industrial defect detection benchmarks and three medical image UAD tasks, which shows our superiority over current state-of-the-art methods.

ReContrast: Domain-Specific Anomaly Detection via Contrastive Reconstruction

Abstract

Most advanced unsupervised anomaly detection (UAD) methods rely on modeling feature representations of frozen encoder networks pre-trained on large-scale datasets, e.g. ImageNet. However, the features extracted from the encoders that are borrowed from natural image domains coincide little with the features required in the target UAD domain, such as industrial inspection and medical imaging. In this paper, we propose a novel epistemic UAD method, namely ReContrast, which optimizes the entire network to reduce biases towards the pre-trained image domain and orients the network in the target domain. We start with a feature reconstruction approach that detects anomalies from errors. Essentially, the elements of contrastive learning are elegantly embedded in feature reconstruction to prevent the network from training instability, pattern collapse, and identical shortcut, while simultaneously optimizing both the encoder and decoder on the target domain. To demonstrate our transfer ability on various image domains, we conduct extensive experiments across two popular industrial defect detection benchmarks and three medical image UAD tasks, which shows our superiority over current state-of-the-art methods. Code is available at: https://github.com/guojiajeremy/ReContrast

1 Introduction

Unsupervised anomaly detection (UAD) aims to recognize and localize anomalies based on the training set that contains only normal images. UAD has a wide range of applications, e.g., industrial defect detection and medical disease screening, addressing the difficulty of collection and labeling of all possible anomalies.

Efforts on UAD attempt to learn the distribution of available normal samples. Most state-of-the-arts utilize networks pre-trained on large-scale datasets, e.g. ImageNet [1], for extracting discriminative and informative feature representations. Feature reconstruction [2; 3; 4] and feature distillation methods [5; 6] are proposed to reconstruct features of pre-trained encoders as alternatives to pixel reconstruction [7; 8], because learned features provide more informative representations than raw pixels [2; 3; 9]. The above approaches can be categorized as epistemic methods, under the hypothesis that the networks trained on normal images can only construct normal regions, but fail for unseen anomalous regions. Feature memory & modeling methods [10; 11; 12; 13; 14] memorize and model all anomaly-free features extracted from pre-trained networks in training, and compare them with test features during inference. In feature reconstruction, the parameters of encoders should be frozen to prevent the networks from converging to a trivial solution, i.e., pattern collapse [2; 3]. In feature distillation and feature memory & modeling methods, there are no gradients for optimizing

data flow of input

: gradient backprop.

K:stopgradient

GAP

global averagepooling

x'(another view of x)

x

x

x

x

！

！

Encoder

Encoder

pretrained

pretrained

pretrained

Encoder

Encoder

Encoder

GAP

GAP

Z

z'(another view of=)

Z

Decoder

Decoder

Decoder

Predictor

Predictor

regional

g

distance

h

Z

representation distance

global distance

(a) ContrastiveLearning

(b) Feature Reconstruction UAD

(c) ReContrast (Ours)

feature encoders. Unfortunately, poor transfer ability is caused by the semantic gap between natural images on which the frozen networks were pre-trained and various UAD image modalities.

Most recently, sparse studies have addressed the transfer problem in UAD settings. CFA [14] and SimpleNet[15] unanimously utilize a learnable linear layer to adapt the output features of the encoder to task-oriented features, while still keeping the encoder frozen. However, the layer after a frozen encoder may be insufficient for adaptation as it can hardly recover the domain-specific information already lost during extraction, especially when the target domain is far from natural images, e.g. medical images. More related works are presented in Appendix A.

In this work, we focus on orienting the whole UAD network in target domains. We propose Re constructive Contrast ive Learning (ReContrast) for domain-specific anomaly detection by optimizing all parameters end-to-end. We start with epistemic UAD methods that detect anomalies by reconstruction errors between the encoder and the reconstruction network (decoder) [2; 9]. By inspecting the behavior of encoder, we find that direct optimization of the encoder causes diversity degradation of encoder features and thus harms the performance. Accordingly, we introduce three key elements of contrastive learning (Figure 1(a)) into feature reconstruction UAD (Figure 1(b)), building a 2-D feature map contrast paradigm to train the encoder and decoder jointly on target images without obstacles. First, to mimic the role of global average pooling (GAP), a new optimization objective, namely global cosine distance, is proposed to make the contrast in a global manner to stabilize the optimization. Second, we bring in the essential operation, i.e., stop-gradient, which prevents positive-pair contrastive learning [16; 17] from pattern collapse. Third, as image augmentation can introduce potential anomalies, we propose to utilize two encoders in different domains to generate representations in two views from the same input image. In addition, we propose a hard-mining strategy for optimizing normal regions that are hard to reconstruct, to magnify the difference between intrinsic reconstruction error and epistemic reconstruction error. It is noted that ReContrast is NOT a self-supervised learning method (SSL) for pre-text pre-training like SimSiam [16] and SimCLR [18], but an end-to-end anomaly detection approach.

To validate our transfer ability on various domains, we evaluate ReContrast with extensive experiments on two widely used industrial defect detection benchmarks, i.e., MVTec AD [19] and VisA [20], and three medical image benchmarks, including optical coherence tomography (OCT) [21], color fundus image [22], and skin lesion image [23]. Without elaborately designing memory banks or handcrafted pseudo anomalies, our proposed method achieves superior performance compared with previous state-of-the-arts. Notably, we present an excellent image-level AUROC of 99.5% on MVTec AD, which reduces the error of the previous best by nearly half. The contributions of this study can be summarized as:

We propose a simple yet effective UAD method to handle the poor transfer ability of the frozen ImageNet encoder when applied to UAD images with a large domain gap. Targeting the similarity and difference between contrastive learning and feature reconstruction, three key elements are elegantly introduced into feature reconstruction UAD for optimizing the entire network without obstacles.

Inspired by the GAP in contrastive learning, we design a new objective function, introducing globality into point-by-point cosine distance for improving training stability.

We utilize the key operation of positive-pair contrastive learning, i.e., stop-gradient, to prevent the encoder from pattern collapse.

We propose a novel way of generating contrastive pairs to avoid potential anomalies introduced by image augmentation.

2 Method: From Reconstruction to ReContrast

In the following subsections, we present an exploration journey starting from a conventional feature reconstruction baseline to our ReContrast. We propose three cohesive components inspired by the elements of contrastive representation learning and present them in a "motivation, connection, mechanism" manner. We believe such progressive exploration would illustrate the motivation of our method better.

encoder layer 1

regional cosine

decoder layer 1

distance

Neck

cosine distance

regional

Neck

cosine distance

global

cosine distance

global

encoder layer 2

decoderlayer 2

十

Config.C:B+ optimize encoder

encoder layer 3

decoder layer 3

X

X

encoderlayer

女

X

: decoder layer

upsample

Neck

global

Neck

:bottleneck

cosine distance

Neck

: data flow

global

global

cosine

cosine

distance

distance

gradient

backprop.

Smap

Config.D : C + stop gradient

Config.E: D + contrastive pairs (ReContrast)

(a)

(b)

2.1 Preliminary: Reverse Distillation

First, we revisit a simple yet powerful epistemic UAD method based on feature reconstruction, i.e., Reverse Distillation (RD4AD) [2]. The method is illustrated in Config.A of Figure 2. RD4AD consists of an encoder (teacher), a bottleneck, and a reconstruction decoder (student). Without loss of generality, the first three layers of a standard 4-layer network are used as the encoder. The encoder extracts informative feature maps with different spatial sizes. The bottleneck is similar to the fourth layer of the encoder but takes the feature maps of the first three layers as the input. The decoder is the reverse of the encoder, i.e., the down-sampling operation at the beginning of each layer is replaced by up-sampling. During training, the decoder learns to reconstruct the output of the encoder layers by maximizing the similarity between feature maps. During inference, the decoder is expected to reconstruct normal regions of feature maps but fails for anomalous regions as it has never seen such samples.

Let f k E , f k D ∈ R C k × H k × W k denote the output feature maps from the k th layer of the encoder and decoder respectively, where C k , H k , and W k denote the number of channels, height, and width of the k th layer, respectively. Regional cosine distance [2; 6; 24] is used to measure the difference between f k E and f k D point by point, obtaining a 2-D distance map M k ∈ R H k × W k :

feature map of encoder

feature mapfrom decoder

Lregion

cosine distance after GAP

flatten

global

'cosinedistance

flatten

regional

cosine distance

Lglobal

Lglobal-hm

(a)

(b)

where ∥·∥ is ℓ 2 norm, and a set of ( h, w ) locates a spatial point in the feature map. The calculation is depicted in Figure 3(a). A large value in M k indicates anomaly at the corresponding position. Final anomaly score map S map is obtained by up-sampling all M k to image size and a sum operation. The maximum value of S map is taken as the anomaly score for the image, denoted as S img . During training, the anomaly maps are minimized on normal images by the regional cosine distance loss, given as:

The optimization of L region is illustrated in the upper-left of Figure 3(b). Notably, though L region was proposed clearly as the loss function of RD4AD [2], the official code 1 implemented a different function that makes an extra dimension flatten operation (maybe by coding bug). However, we observe extreme instability during the training with L region , producing degenerated I-AUROC of 95.3% on MVTec AD compared with 98.5% reported in the paper. The I-AUROC of 95.3% is more consistent with other methods using L region [6; 24]. In the next subsection, we will show that the seemingly unreasonable "bug" in the code improves the performance by explicitly introducing globality into optimization.

2.2 Global Cosine Similarity

It is a convention to use global average pooling (GAP) to pool the 2-D feature map of the last network layer to a 1-D representation, in both supervised classification and self-supervised contrastive learning. In UAD, GAP will simply mess up all feature points together, losing the ability to distinguish normal and anomalous regions, as shown in the upper-right of Figure 3(b). Therefore, we try to directly bring globality into the loss function while maintaining the point-to-point correspondence between f k E and f k D . Let F denote a flatten operation that casts a 2-D feature map f ∈ R C × H × W to a vector v ∈ R CHW . Our proposed global cosine distance loss is given as:

where the number of cosine dimension is C · H · W instead of C in L region , as depicted in Figure 3(a). Intuitively, M k is also minimized along with the minimization of L global . It is easy to prove that minimizing L global to 0 equals to minimizing M k to 0. Though losses are not completely optimized to 0 in neural network training, the regional S map still works as the anomaly map. The model trained with L global is named as Config.B .

As shown in Figure 5(a), the performance of Config.B (blue) is much more favorable and stable than Config.A (purple) during training. Here, we briefly analyze the underlying mechanism by plotting the landscape of S map (average of all samples) against model parameters near optimized minima in Figure 4. We observe two differences. First, the overall landscape of the model optimized by L global is flatter than that by L region , which implies S map is more stable during training with L global . The instability of L region can be caused by the point-by-point distance, where the fitting of one region may cause under-fitting of the others. L global can be regarded as the distance between manifolds of feature points, as depicted in the lower-left of Figure 3(b), which measures the consistency globally instead of excessively focusing on individual regions. Second, the landscape around the final minima of the model trained by L global is sharper than that by L region . Previous explorations on the geometry of loss landscapes suggest that flat minima generalize better on unseen samples [25; 26], which is unwanted in UAD settings. The real mechanism can be more complicated and is still under further investigation.

sharper

flatter

2.5

2.5

average

2.0

average

flatter

Smap

sharper

1.0

Smap

0.5

0.0

0.0

parameter direction x

parameter directionx

parameter direction y

parameter directiony

Landscape of average Smap

Landscape of average Smap

(a)

(b)

optimized by Lregion

optimized by Lglobal

2.3 Feature Degradation of Optimized Encoder

It seems easy to adapt the network in the target domain by jointly optimizing the encoder and decoder, as illustrated in Figure 2 Config.C . However, previous attempts [9] suggest that training encoders would lead encoders to extract indiscriminative features that are easy to reconstruct, which is also described as pattern collapsing or converging to trivial solutions [2]. We observe a similar phenomenon that the training loss quickly decreases to nearly zero, as shown by the orange dotted line in Figure 5(a).

To inspect the behavior of the encoder, we probe the diversity of encoder feature maps by the standard deviation ( std ) of ℓ 2 normalized feature points f k E ( h, w ) / ∥ ∥ f k E ( h, w ) ∥ ∥ 2 . The larger the std , the more discriminative the feature, and the more distinctive between different regions. We plot the feature diversity of outputs of the 2 nd encoder layer during training in Figure 5(b). The std of Config.C drops quickly during training, while the std of Config.B keeps constant as a comparison. In addition, the std of Config.C does not completely collapse to zero, which explains the passable performance. The outputs of other middle layers follow the same trend. The observation indicates that the degeneration of performance is attributed to the degradation of encoder features. It is noted that the std starts from different values because of the different modes of batch normalization (BN) [28] during training. 2

anomalous

smap with

image

Lglobal-hm

1.00

1.5

0.95

0.032

0.90

AUROC

！

std

Config.A& B

(overlap)

smap with Lglobal

Loss

easy-normal

hard-normal

0.026

anomaly

0.50

0:0

0

iterations

1500

0

iterations

1500

(a)I-AUROC&loss

(b) std of f2(h, w)/f2(h, w)Il-

(c) visualization of smap

2.4 Stop Gradient

Contrastive learning for SSL maximizes the similarity between the representations of two augmentations (views) of one image while trying to avoid collapsing solutions in which all inputs are encoded to a constant value. SimSiam [16] indicated that the stop-gradient operation plays an essential role in preventing collapsing for SSL methods with positive pairs only [16; 17]. Intriguingly, we found that contrastive SSL can be explained from the perspective of feature reconstruction. The predictor (a multi-layer perceptron, MLP) can be regarded as a reconstruction network that constructs the representation of one view x to match the representation of another view x ′ , as in Figure 1. Vice versa, feature reconstruction UAD can be transformed into contrastive learning as well, regarding the reconstruction network as a predictor.

To this end, we introduce the stop-gradient operation into feature reconstruction, transforming it into a contrastive-like paradigm. The network is optimized by contrasting the feature maps of the encoder and decoder, as shown in Config.D of Figure 2. The gradients do not propagate directly into the encoder, but back into the encoder through the decoder. It is implemented by modifying (3) as:

where sg denotes the stop-gradient operation. The training of this configuration can be intuitively explained as a "mutual reinforcement". The optimization of f k D drives the encoder to be more specific and informative in the target domain by the gradients from the decoder, while the more domainspecific f k E of encoder requires further optimization of f k D . There are previous works trying to explain the deeper mechanism of stop gradient in contrastive learning. In [16], the authors hypothesize this configuration as an implementation of an Expectation-Maximization (EM) algorithm that implicitly solves two underlying sub-problems with two sets of variables. In [29], the authors argue that there are flaws in the hypothesis of [16] and suggest that the decomposed gradient vector (center gradient) helps prevent collapse via the de-centering effect.

As shown in Figure 5(a) and (b), Config.D can constantly extract more discriminative features without degeneration, boosting it to perform better than Config.C at the beginning of training. Previous works [9; 3] suggested that the reconstruction decoder may learn an "identical shortcut" that well reconstructs both seen and unseen samples, which explains the performance drop of Config.D after reaching the peak in Figure 5(a). To handle this issue, contrastive pairs are introduced.

2.5 Contrastive Pairs

Contrastive learning methods make use of (positive) contrastive image pairs, based on the intuition that the semantic information of different augmented views of the same image should be the same. Without image augmentations ( x = x ′ ), contrastive learning degrades into a self-reconstruction paradigm that the predictor predicts the input of itself, which may also collapse into an "identical shortcut" just like Config.D . In epistemic UAD methods, it is impossible to construct contrastive pairs by image augmentation, because 1) any augmentation (flip, cutout, color jitter, etc.) could be potential

anomalies; and 2) the feature maps of two augmented views lost the regional correspondence after spatial augmentations (shift, rotate, scale, etc.). Therefore, we propose an augmentation-free method to construct two views of one image.

In previous explorations, the encoder is adapted to the target image domain and generates feature representations in a domain-specific view. We additionally introduce a frozen encoder network without any adaptation throughout the training, which perceives images in the view of pre-train image domain. As shown in Config.E of Figure 2, the decoder and bottleneck take the features of the domain-specific encoder to reconstruct the frozen encoder; vice versa, they simultaneously take the features of the frozen encoder to reconstruct the domain-specific encoder, building a complete two-view contrastive paradigm. This configuration can also be interpreted as cross-reconstruction, as the decoder reconstructs the information of the frozen encoder given the input of the adapted encoder; and reconstructs the information of the adapted encoder given the input of the frozen encoder. With six pairs of feature maps, the S map is obtained by adding up the six up-sampled cosine distance maps. As shown in Figure 5(b), the encoder feature diversity of Config.E not only holds but slightly increases during training.

2.6 Magnify Epistemic Error by Hard-Normal Mining

Due to network information capacity, intrinsic reconstruction errors exist in available normal regions. The errors of normal details and edges (hard-normal) are generally higher than other plain regions (easy-normal), leading to confusion between the intrinsic error of hard-normal regions and the epistemic error of unseen anomalies, as demonstrated in the lower-left of Figure 5(c). It was proposed to hard-mine the hard-normal regions by simply discarding the "easy" pairs of f k E ( h, w ) , f k D ( h, w ) with small cosine distance in L region [24]. However, arbitrarily discarding feature points in f k D breaks the global feature point manifolds in L global . Therefore, we propose a hard mining strategy, namely L global -hm , focusing on the optimization of hard-normal regions to widen the gap between the epistemic errors caused by unseen anomalies and the errors of all normal regions. The gradients of easy f k D ( h, w ) are discarded instead of eliminating f k D ( h, w ) itself, by modifying f k D as:

where µ ( M k ( h, w ) ) is the average regional distance on the dataset (approximated on mini-batch), σ ( M k ( h, w ) ) is the standard deviation of the regional distance, and α is a hyper-parameter to control the discarding rate. The optimization of L global -hm is depicted in the lower-right of Figure 3(b). As shown in Figure 5(c), the model trained by L global -hm activates fewer normal regions than the model trained by L global . The usage of hard mining strategy can also mitigate the over-fitting of easy training samples.

3 Experiments

In this section, we evaluate the proposed method on two popular UAD applications, i.e., industrial defect inspection and medical disease screening. We refer to our complete approach (ReContrast+ L global -hm ) as ReContrast and Ours in the Experiments section for simplicity.

3.1 Experimental Settings

Datasets . MVTec AD is the most widely used industrial defect detection dataset, containing 15 categories of sub-datasets (5 textures and 10 objects). VisA is a challenging industrial defect detection dataset, containing 12 categories of sub-datasets. OCT2017 is an optical coherence tomography dataset [21]. APTOS is a color fundus image dataset, available as the training set of the 2019 APTOS blindness detection challenge [22]. ISIC2018 is a skin disease dataset, available as task 3 of ISIC2018 challenge [23]. Detailed information is presented in Appendix B.

Implementation . WideResNet50 [30] pre-trained on ImageNet [1] is utilized as the encoder by default. AdamW optimizer [31] is utilized with β =(0.9,0.999) and weight decay=1e-5. The learning rates of new (decoder and bottleneck) and pre-trained (encoder) parameters are 2e-3 and 1e-5, respectively. The network is trained for 3,000 iterations on VisA, 2,000 on MVTec AD and ISIC2018, and 1,000 on APTOS and OCT2017. The α in equation (5) linearly rises from -3 to 1 in the first

one-tenth iterations and keeps 1 for the rest training. More experimental details and environments are presented in Appendix C.

Metrics . Image-level anomaly detection performance is measured by the Area Under the Receiver Operator Curve (I-AUROC). F1-score (F1) and accuracy (ACC) are adopted for medical datasets following [32]. The operating threshold of F1 and ACC is determined by the optimal value of F1. For anomaly segmentation, pixel-level AUROC (P-AUROC) and Area Under the Per-Region-Overlap (AUPRO) [33] are used as the evaluation metrics. AUPRO is more meaningful than P-AUROC because of the unbalanced amount of normal and anomalous pixels [20].

3.2 Anomaly Detection and Segmentation on Industrial Images

Anomaly detection results on MVTec AD are shown in Table 1. Our approach achieves a superior average I-AUROC of 99.5% . In the aspect of error (100% -I-AUROC), we achieve only 0.5%, reducing the previous SOTA error of PatchCore (0.9%) by relatively 44% . Anomaly segmentation results are shown in Table 2 (average of 15 categories). Our ReContrast achieves SOTA performance measured by both P-AUROC and AUPRO. In AUPRO, we produce 95.2% , exceeding the previous SOTA RD4AD by 1.3%. We also present SOTA performances on texture ( Text. Avg. ) and object ( Obj. Avg. ) categories, respectively.

Anomaly detection and segmentation results on VisA are shown in Table 3 (average of 12 categories). Our approach achieves a superior average I-AUROC of 97.5% , exceeding previous SOTA RD4AD by 1.5%. In AUPRO, we produce 92.6% , outperforming previous SOTA PatchCore by 1.4%.

3.3 Multi-Class Anomaly Detection with A Unified Model

Conventional UAD settings train separate models for different object categories. In UniAD [9], the authors proposed to train a unified model for multiple classes. We evaluate our ReContrast under the unified setting on MVTec AD and VisA. Our model is trained for 5,000 iterations on all images containing all 15 MVTec AD categories (or 12 categories in VisA) and evaluated on each category following [9]. The results are presented in Table 4. On MVTec AD, our method produces an I-AUROC of 98.2% , exceeding the previous SOTA 96.5% of UniAD by a large 1.7% . On VisA, both our ReContrast ( 95.1% ) and baseline RD4AD (92.7%) outperform the previous multi-class SOTA UniAD (91.5%), while the proposed method exceeds RD4AD by 2.4%.

3.4 Anomaly Detection on Medical Images

The experimental results on three medical image datasets are presented in Table 5. Apart from industrial UAD methods, we also compare our ReContrast with approaches designed for medical images (f-AnoGan [7], GANomaly [8], AE-flow [32]), and contemporaneous methods proposed for better domain adaption ability (CFA [14], SimpleNet [15]). We reproduce the methods that were not evaluated on the corresponding dataset if they are open-sourced. Our method achieves the best results in all metrics on APTOS and ISIC2018, and comparable best results on the relatively easy OCT2017, demonstrating that ReContrast can directly adapt to various medical image modalities.

3.5 Ablation Study

To verify the effect of the proposed elements, we conduct thorough ablation studies on MVTec AD and APTOS. Because some configurations are unstable during training on some categories, we report the last iteration's performances and those of the best checkpoint during training (best by I-AUROC, evaluate every 250 iterations). The results are reported in Table 6. The use of L global ( Config.B ) boosts the performance of Config.A trained by L region . Directly optimizing the encoder and decoder together ( Config.C ) causes a great degeneration because of pattern collapse, while either stop-gradient ( Config.D ) or contrastive pairs ( Config.C + cp ) recovers some of it. By introducing both ( Config.E ), the performance is further improved to exceed the frozen encoder baseline. Finally, training with L global -hm yields SOTA performances (Ours). Meanwhile, L global -hm can also improve the baseline reconstruction method alone ( Config.B + hm ). More ablation studies, experimental results with error bars, limitations, and qualitative visualization are presented in Appendix.

4 Conclusion

In this study, we propose a novel contrastive learning paradigm, namely ReContrast, for domainspecific unsupervised anomaly detection. It addresses the transfer ability of the pre-trained encoder by jointly optimizing all parameters end-to-end. The key elements of contrastive learning are elegantly embedded in epistemic UAD method to avoid pattern collapse, training instability, and identical shortcut. Extensive experiments on MVTec AD, VisA, and three medical image datasets demonstrate our superiority. The idea of optimizing encoders can further boost the application of UAD methods on more image modalities that are far from natural image domain.

5 Limitations

Scope of Application In this work, we mainly focus on UAD that detects regional defects (most common in practical applications like industrial inspection and medical disease screening), which is distinguished from one-class classification (OCC, or Semantic AD). In our UAD, normal and anomalous samples are semantically the same objects except local detects, e.g. good cable v.s. spoiled cable. In OCC, normal samples and anomalous samples are semantically different, e.g. cat v.s. other animals. More details are discussed in Appendix E.

Training Instability Our method still suffers some extent of training instability. Because of the absence of validation sets in UAD settings, whether the last epoch (for reporting results) is in the middle of a loss spike and performance dip is related to random seeds. We found that a number of UAD methods (RD4AD[2], CFA[14], SimpleNet[15]) are also subject to training instability when running their code. We discuss the reason and solutions to mitigate this effect in Appendix E.

Acknowledgments and Disclosure of Funding

This research work is supported by the National Natural Science Foundation of China (NSFC) (GrantNo. 82072007), Beijing Natural Science Foundation (Grant No. IS23112), and Beijing Institute of Technology Research Fund Program for Young Scholars.

References

O. Russakovsky et al. , 'ImageNet Large Scale Visual Recognition Challenge,' International Journal of Computer Vision , vol. 115, no. 3, 2015.

H. Deng and X. Li, 'Anomaly Detection via Reverse Distillation from One-Class Embedding,' in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pp. 9737-9746, 2022.

Z. You, K. Yang, W. Luo, L. Cui, Y. Zheng, and X. Le, 'ADTR: Anomaly Detection Transformer with Feature Reconstruction,' arXiv preprint arXiv:2209.01816 , 2022.

Y. Shi, J. Yang, and Z. Qi, 'Unsupervised anomaly segmentation via deep feature reconstruction,' Neurocomputing , vol. 424, 2021.

M. Salehi, N. Sadjadi, S. Baselizadeh, M. H. Rohban, and H. R. Rabiee, 'Multiresolution knowledge distillation for anomaly detection,' in Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition , 2021.

G. Wang, S. Han, E. Ding, and D. Huang, 'Student-teacher feature pyramid matching for anomaly detection,' in The British Machine Vision Conference (BMVC) , 2021.

T. Schlegl, P. Seeböck, S. M. Waldstein, G. Langs, and U. Schmidt-Erfurth, 'f-AnoGAN: Fast unsupervised anomaly detection with generative adversarial networks,' Medical Image Analysis , vol. 54, 2019.

S. Akcay, A. Atapour-Abarghouei, and T. P. Breckon, 'GANomaly: Semi-supervised Anomaly Detection via Adversarial Training,' in Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics) , vol. 11363 LNCS, 2019.

Z. You et al. , 'A unified model for multi-class anomaly detection,' in Advances in Neural Information Processing Systems (S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, eds.), vol. 35, pp. 4571-4584, Curran Associates, Inc., 2022.

T. Defard, A. Setkov, A. Loesch, and R. Audigier, 'PaDiM: A Patch Distribution Modeling Framework for Anomaly Detection and Localization,' in Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics) , vol. 12664 LNCS, 2021.

K. Roth, L. Pemula, J. Zepeda, B. Schölkopf, T. Brox, and P. Gehler, 'Towards total recall in industrial anomaly detection,' in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pp. 14318-14328, 2022.

J. Yi and S. Yoon, 'Patch SVDD: Patch-Level SVDD for Anomaly Detection and Segmentation,' in Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics) , vol. 12627 LNCS, 2021.

O. Rippel, P. Mertens, and D. Merhof, 'Modeling the distribution of normal data in pre-trained deep features for anomaly detection,' in Proceedings - International Conference on Pattern Recognition , 2020.

S. Lee, S. Lee, and B. C. Song, 'Cfa: Coupled-hypersphere-based feature adaptation for target-oriented anomaly localization,' IEEE Access , vol. 10, pp. 78446-78454, 2022.

Z. Liu, Y. Zhou, Y. Xu, and Z. Wang, 'Simplenet: A simple network for image anomaly detection and localization,' arXiv preprint arXiv:2303.15140 , 2023.

X. Chen and K. He, 'Exploring simple Siamese representation learning,' in Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition , 2021.

J. B. Grill et al. , 'Bootstrap your own latent a new approach to self-supervised learning,' in Advances in Neural Information Processing Systems , vol. 2020-Decem, 2020.

T. Chen, S. Kornblith, M. Norouzi, and G. Hinton, 'A simple framework for contrastive learning of visual representations,' in 37th International Conference on Machine Learning, ICML 2020 , vol. PartF16814, 2020.

P. Bergmann, M. Fauser, D. Sattlegger, and C. Steger, 'Mvtec ad-a comprehensive real-world dataset for unsupervised anomaly detection,' in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pp. 9592-9600, 2019.

Y. Zou, J. Jeong, L. Pemula, D. Zhang, and O. Dabeer, 'Spot-the-difference self-supervised pre-training for anomaly detection and segmentation,' in Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXX , pp. 392408, Springer, 2022.

D. S. Kermany et al. , 'Identifying Medical Diagnoses and Treatable Diseases by Image-Based Deep Learning,' Cell , vol. 172, no. 5, 2018.

Asia Pacific Tele-Ophthalmology Society, 'APTOS 2019 blindness detection,' 2019.

N. Codella et al. , 'Skin lesion analysis toward melanoma detection 2018: A challenge hosted by the international skin imaging collaboration (isic),' arXiv preprint arXiv:1902.03368 , 2019.

Y. Cao, Q. Wan, W. Shen, and L. Gao, 'Informative knowledge distillation for image anomaly segmentation,' Knowledge-Based Systems , vol. 248, p. 108846, 2022.

N. S. Keskar, D. Mudigere, J. Nocedal, M. Smelyanskiy, and P. T. P. Tang, 'On large-batch training for deep learning: Generalization gap and sharp minima,' in International Conference on Learning Representations , 2017.

P. Foret, A. Kleiner, H. Mobahi, and B. Neyshabur, 'Sharpness-aware minimization for efficiently improving generalization,' in International Conference on Learning Representations , 2021.

H. Li, Z. Xu, G. Taylor, C. Studer, and T. Goldstein, 'Visualizing the loss landscape of neural nets,' Advances in neural information processing systems , vol. 31, 2018.

S. Ioffe and C. Szegedy, 'Batch normalization: Accelerating deep network training by reducing internal covariate shift,' in 32nd International Conference on Machine Learning, ICML 2015 , vol. 1, 2015.

C. Zhang, K. Zhang, C. Zhang, T. X. Pham, C. D. Yoo, and I. S. Kweon, 'How does simsiam avoid collapse without negative samples? a unified understanding with self-supervised contrastive learning,' in International Conference on Learning Representations , 2021.

S. Zagoruyko and N. Komodakis, 'Wide residual networks,' arXiv preprint arXiv:1605.07146 , 2016.

I. Loshchilov and F. Hutter, 'Decoupled weight decay regularization,' in 7th International Conference on Learning Representations, ICLR 2019 , 2019.

Y. Zhao, Q. Ding, and X. Zhang, 'AE-FLOW: Autoencoders with Normalizing Flows for Medical Images Anomaly Detection,' in The Eleventh International Conference on Learning Representations , 2023.

P. Bergmann, M. Fauser, D. Sattlegger, and C. Steger, 'Uninformed students: Student-teacher anomaly detection with discriminative latent embeddings,' in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pp. 4183-4192, 2020.

D. Gudovskiy, S. Ishizaka, and K. Kozuka, 'Cflow-ad: Real-time unsupervised anomaly detection with localization via conditional normalizing flows,' in Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision , pp. 98-107, 2022.

C. L. Li, K. Sohn, J. Yoon, and T. Pfister, 'CutPaste: Self-Supervised Learning for Anomaly Detection and Localization,' in Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition , 2021.

V. Zavrtanik, M. Kristan, and D. Skoˇ caj, 'DRÆM - A discriminatively trained reconstruction embedding for surface anomaly detection,' in Proceedings of the IEEE International Conference on Computer Vision , 2021.

N. Cohen and Y. Hoshen, 'Sub-image anomaly detection with deep pyramid correspondences,' arXiv preprint arXiv:2005.02357 , 2020.

A. S. Collin and C. de Vleeschouwer, 'Improved anomaly detection by training an autoencoder with skip connections on images corrupted with Stain-shaped noise,' in Proceedings International Conference on Pattern Recognition , 2020.

P. Perera and V. M. Patel, 'Learning Deep Features for One-Class Classification,' IEEE Transactions on Image Processing , vol. 28, no. 11, 2019.

D. P. Kingma and M. Welling, 'Auto-encoding variational bayes,' in 2nd International Conference on Learning Representations, ICLR 2014 - Conference Track Proceedings , 2014.

F. Milkovic, B. Filipovic, M. Subasic, T. Petkovic, S. Loncaric, and M. Budimir, 'Ultrasound anomaly detection based on variational autoencoders,' in International Symposium on Image and Signal Processing and Analysis, ISPA , vol. 2021-September, 2021.

H. Zhao et al. , 'Anomaly Detection for Medical Images Using Self-Supervised and TranslationConsistent Features,' IEEE Transactions on Medical Imaging , vol. 40, no. 12, 2021.

T. Reiss, N. Cohen, L. Bergman, and Y. Hoshen, 'Panda: Adapting pretrained features for anomaly detection and segmentation,' in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pp. 2806-2814, 2021.

K. Sohn, C.-L. Li, J. Yoon, M. Jin, and T. Pfister, 'Learning and evaluating representations for deep one-class classification,' in International Conference on Learning Representations , 2021.

X. Gui, D. Wu, Y. Chang, and S. Fan, 'Constrained adaptive projection with pretrained features for anomaly detection,' in Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence, IJCAI-22 (L. D. Raedt, ed.), pp. 2059-2065, International Joint Conferences on Artificial Intelligence Organization, 7 2022. Main Track.

G. Xie, J. Wang, J. Liu, J. Lyu, Y . Liu, C. Wang, F. Zheng, and Y . Jin, 'Im-iad: Industrial image anomaly detection benchmark in manufacturing,' arXiv preprint arXiv:2301.13359 , 2023.

B. Schölkopf, J. C. Platt, J. Shawe-Taylor, A. J. Smola, and R. C. Williamson, 'Estimating the support of a high-dimensional distribution,' Neural Computation , vol. 13, no. 7, 2001.

D. M. Tax and R. P. Duin, 'Support Vector Data Description,' Machine Learning , vol. 54, no. 1, 2004.

P. Bergmann, K. Batzner, M. Fauser, D. Sattlegger, and C. Steger, 'Beyond dents and scratches: Logical constraints in unsupervised anomaly detection and localization,' International Journal of Computer Vision , vol. 130, no. 4, pp. 947-969, 2022.

I. Molybog, P. Albert, M. Chen, Z. DeVito, D. Esiobu, N. Goyal, P. S. Koura, S. Narang, A. Poulton, R. Silva, et al. , 'A theory on adam instability in large-scale machine learning,' arXiv preprint arXiv:2304.09871 , 2023.

S. J. Reddi, S. Kale, and S. Kumar, 'On the convergence of adam and beyond,' arXiv preprint arXiv:1904.09237 , 2019.

Appendices

First, we give the formulation of unsupervised anomaly detection (UAD). Let D train = { ( x i , y i ) } N i =1 , y i ∈ {-} be the training set of normal samples, where x i is the i th sample with its label y i . During test, D test = { ( x i , y i ) } i M =1 , y i ∈ {-, + } ( -for normal, + for anomalous) contains both normal and anomalous images. Our goal is to train a model using D train to identify ( x, +) from ( x, -) in D test , while localizing anomalous regions at the same time.

A Related Work

Epistemic methods are based on the assumption that the networks respond differently during inference between seen input and unseen input. Within this paradigm, pixel reconstruction methods assume that the networks trained on normal images can reconstruct anomaly-free regions well, but poorly for anomalous regions [38]. Auto-encoder (AE) [39; 38], variational auto-encoder (VAE) [40; 41], or generative adversarial network (GAN) [42; 7; 8] are used to restore normal pixels. However, pixel reconstruction models may also succeed in restoring unseen anomalous regions if they resemble normal regions in pixel values or the anomalies are barely noticeable [2; 9]. Therefore, feature reconstruction is proposed to construct features of pre-trained encoders instead of raw pixels [2; 3; 4]. To prevent the whole network from converging to a trivial solution, the parameters of the encoders are frozen during training [9]. In feature distillation [2; 5; 6], the student network is trained from scratch to mimic the output features of the pre-trained teacher network with the same input of normal images, also based on the similar hypothesis that the student trained on normal samples only succeed in mimicking features of normal regions.

Pseudo-anomaly methods generate handcrafted defects on normal images to imitate anomalies, converting UAD to supervised classification [35] or segmentation task [36]. Specifically, CutPaste [35] simulates anomalous regions by randomly pasting cropped patches of normal images. DRAEM [36] constructs abnormal regions using Perlin noise as the mask and another image as the additive anomaly. SimpleNet [15] introduces anomaly by injecting Gaussian noise in the pre-trained feature space. These methods deeply rely on how well the pseudo anomalies match the real anomalies [2], which makes it hard to generalize to different datasets.

Feature memory & modeling methods [10; 11; 12; 13; 14] memorize all (or their modeled distribution) normal features extracted by networks pre-trained on large-scale datasets and match them with test samples during inference. Since these methods require memorizing, processing, and matching nearly all features from training samples, they are computationally expensive in both training and inference, especially when the training set is large. There is a considerable semantic gap between large-scale natural images on which the frozen networks were pre-trained and the target UAD images in various modalities. Therefore, methods based on pre-trained networks struggle in transferring as the feature encoders are not optimized in the target domain.

There are a few studies addressing the adaptation problem in UAD settings. CFA [14] and SimpleNet[15] unanimously utilize a learnable linear layer to transform the output features of the frozen encoder. However, the linear layer may be insufficient for adaption and can hardly recover the domain-specific information in pre-trained features, especially when the target domain, e.g. CT and MRI, is remote from natural image domain. In OCC settings, a number of works focus on utilizing self-supervised pre-training approaches to learn a compact representation space for normal samples [43; 44; 45]. However, the performances are far from satisfactory in UAD tasks (I-AUROC under 90% on MVTec AD).

B Datasets

MVTec AD [19] is an industrial defect detection dataset, containing 15 sub-datasets (5 texture categories and 10 object categories) with a total of 3,629 normal images as the training set and 1,725 normal and anomalous images as the test set. Pixel-level annotations are available for anomalous images for evaluating anomaly segmentation. All images are resized to 256 × 256.

VisA [20] is an industrial defect detection dataset, containing 12 sub-datasets with a total of 9,621 normal and 1200 anomalous images. Pixel-level annotations are available for anomalous images for evaluating anomaly segmentation. We split it into training and test sets following the setting in [20]. All images are resized to 256 × 256.

OCT2017 is an optical coherence tomography (OCT) dataset [21]. The dataset is labeled into four classes: normal, drusen, DME (diabetic macular edema), and CNV (choroidal neovascularization). The latter three classes are considered anomalous. The training set contains 26,315 normal images and the test set contains 1,000 images. All images are resized to 256 × 256.

APTOS is a color fundus image dataset, available as the official training set of the 2019 APTOS blindness detection challenge [22]. The dataset contains 3,662 images with annotated grade 0-4 to indicate different severity of diabetic retinopathy, yielding 1,805 normal images (grade 0) and 1857 anomalous images (grade 1-4). We randomly selected 1,000 normal images as our training set and the rest 2,662 images as the test set. Images are preprocessed to crop the fundus region, and then resized to 256 × 256.

ISIC2018 is a skin disease dataset, available as task 3 of ISIC2018 challenge [23]. It contains seven classes. NV (nevus) is taken as the normal class and the rest of classes are taken as anomaly, following [32]. The training set contains 6705 normal images. The official validation set is used as our test set, which includes 193 images. Images are resized to 256 × 256 and then center cropped into 224 × 224 to remove redundant background. In addition, because the normal and anomalous images of ISIC2018 are different objects (lesions) instead of healthy objects and unhealthy objects, the task is more like one-class classification. Therefore, we take the mean (instead of maximum) value of S map as the S img in ReContrast and other reproduced methods.

C Complete Implementation Details

WideResNet50 [30] pre-trained on ImageNet [1] is utilized as the encoder by default. The decoder is the upside-down version of the encoder, exactly the same as RD4AD [2], i.e., the 3 × 3 convolution with stride 2 at the beginning of each layer is replaced by a 2 × 2 transposed convolution with stride 2. The bottleneck is also the same as RD4AD [2]. For each decoder layer, two 1x1 convolutions are used to project the feature map to reconstruct the frozen encoder and trained encoder, respectively. AdamW optimizer [31] is utilized with β =(0.9,0.999) and weight decay=1e-5. The learning rates of the new (decoder and bottleneck) and pre-trained (encoder) parameters are 2e-3 and 1e-5 respectively. The network is trained for 3,000 iterations on each sub-dataset in VisA, 2,000 on each sub-dataset in MVTec AD and ISIC2018, and 1,000 on APTOS and OCT2017. The batch size is 16 for industrial datasets and 32 for medical datasets. The α in (5) linearly rises from -3 to 1 in the first one-tenth iterations and keeps 1 for the rest of the training. By default, the batchnorm (BN) layers of encoder are set to train mode during training. Because training instability and performance drop are observed for some categories 3 , the BN of encoder is set to eval mode (use pre-trained statistics) for such datasets. The reason is discussed in Appendix E. A library 4 is used to plot the landscape in Figure 4. The distance and steps arguments are set to 1 and 50 respectively.

On MVTec AD and VisA, the results of comparison methods are taken from the original papers or the benchmark paper [46] that report their performances. On medical datasets, we take the results from [32] if available; otherwise, we reproduce the method with official code. For RD4AD, we use their official code, in which L global is already implemented instead of L region , as discussed in Section 2.1. Codes are implemented with Python 3.8 and PyTorch 1.12.0 cuda 11.3. Experiments are run on NVIDIA GeForce RTX3090 GPUs (24GB).

D Additional Experiments

D.1 Additional Experimental Results

In addition to the averaged segmentation performances in Table 2, we present the P-AUROC and AUPRO of each categories in MVTec AD in Table A1. The results in the main paper are reported

with a single random seed following our baseline [2]. In Table A2, we report the mean and standard deviation of three runs with three different random seeds (1, 11, and 111). In addition to the averaged performances in Table 3, we present the I-AUROC, P-AUROC, and AUPRO of each category in VisA in Table A3.

D.2 Qualitative Visualization

The qualitative anomaly segmentation results of ReContrast trained by L global and ReContrast trained by L global -hm are presented in Figure A1. It is shown that the hard-normal regions are less activated with L global -hm . The results of VisA and medical images are shown in Figure A2 and Figure A3, respectively. Each S map is min-max-normalized to 0-1 for clearer visualization.

D.3 Additional Ablation Study

Ablation study is conducted on the value of α in L global -hm , as shown in Table A4. Assuming the regional cosine distances M k ( h, w ) conforms Gaussian distribution, the discarding rates of feature

point are 2.3%, 15.9%, 50%, 69.1%, 84.1%, 93.3% and 97.7% for α = -2, -1, 0, 0.5, 1, 1.5 and 2, respectively. L global -hm is not sensitive to the discarding rate within the range from 50% to 93%. Though α can be tuned for each dataset, we find α = 1 works just well for most circumstances.

We test a variety of encoder backbones in Table A5, i.e., ResNet18, ResNet50, and WideResNet50 (default), and report their performances, model parameters, as well as multiply-accumulate operations (MACs). The corresponding decoder is the reversed version of the encoder. Different encoders may favor different training hyper-parameters such as learning rate and iteration. Though we do not further tune each backbone, all backbones produce excellent results with default hyper-parameters, suggesting the generality of our method. With each backbone, our method outperforms the corresponding feature reconstruction counterpart RD4AD [2]. Notably. our method with ResNet18 is comparable to RD4AD with WideResNet50.

The utilization of two encoders (one frozen, one trained domain-specific) in our method can function as an ensembling, which may also contribute to performances. We conduct ablation experiments using different encoder-decoder feature map pairs to generate anomaly maps in inference. The results are shown in Table A6. On MVTec, using only the feature of domain-specific encoder produces an I-AUROC of 99.38%, which is comparable to the 99.45% of the ReContrast default and outperforms the 98.86% of baseline Config. B. In addition, we train an ensembling version of Config. B using two different encoders (ResNet50 and WideResNet50), producing an I-AUROC of 98.94% which is nearly identical to the baseline. On APTOS, using only the feature of domain-specific encoder produces an I-AUROC of 97.73%, which outperforms both 97.51% of the ReContrast default and 92.49% of the baseline Config. B. The results indicate that the improvement due to ensembling is small and the domain-specific encoder is vital.

E Limitations

Scope of Application. In this work, we mainly focus on UAD that detects regional defects (most common in practical applications), which is distinguished from one-class classification (OCC, or

Semantic AD). In our UAD, normal and anomalous samples are semantically the same objects except local detects, e.g. good cable v.s. spoiled cable. In OCC, normal samples and anomalous samples are semantically different, e.g. cat v.s. other animals. The slight difference in task setting makes the focus of corresponding methods different. Most methods for UAD attempt to detect anomalies by segmenting anomalous local regions, while methods for OCC detect anomalies based on the representation deviation of the whole image [47; 48]. Though methods of these two tasks can be used for each other to a certain extent, we mainly focus on UAD in this work.

Logical Anomaly . MVTec LOCO is a recently released dataset for evaluating the detection performance of logical anomalies. It contains 5 sub-datasets, each comprised of both structural anomalies, e.g. dents and scratches as in MVTec AD, and logical anomalies, e.g. dislocation and missing parts. We find our method performing less favorably on logical anomalous images than GCAD [49] which is specially designed for logical anomalies, as presented in Table A7. We still outperform other vanilla UAD methods that are not specially designed for such anomalies.

Batch Normalization and Training Instability . As discussed in Appendix C, training instability and performance drops are observed for a few categories when setting BN mode of the encoder to train . The phenomenon of training instability is observed in many deep learning tasks. However, validation sets are often allowed, which counteracts training instability and helps the choice of BN mode.

On the one hand, it can be caused by the limited feature diversity of the one-class UAD dataset. First, we recall the formulation of BN layer. Giving a d -dimensional feature x = ( x (1) , ..., x ( d ) ) , each dimension (channel) is normalized in training:

where the expectation E and variance V ar are computed over the batch, and ϵ is a small constant for numerical stability. In most computer vision datasets, an input batch contains a variety of categories. Each dimension is more or less activated by different image features so that V ar [ x ( k ) ] is not zero. However, because a UAD dataset contains only one type of category, there is a chance that a channel of pre-trained feature x ( k ) is barely activated or equally activated, e.g. a channel sensitive to animals is dead on industrial objects. Thus, the denominator √ V ar [ x ( k ) ] + ϵ is nearly zero, causing malfunction of batch normalization. This problem can be fixed by using pre-trained statistical running_mean and running_var ( eval mode), which loses the effect of BN.

On the other hand, this instability can be attributed to the feature of Adam optimizer. We found that a number of UAD methods [2; 14; 15] are also subject to training instability when running their code. Whether the last epoch (for reporting results) is in the middle of a loss spike and performance dip is strongly related to random seed. In some works [50; 51], the authors attribute the training instability to the historical estimation of squared gradients in Adam optimizer.

In our according explorations, we replace the batch variance V ar [ x ( k ) ] smaller than min(5e-4, running_var ) by pre-trained running_var and we reset the historical state buffer (gradient momentum and second-order gradient momentum) of Adam every 500 iters. Without manually selecting BN mode, such tricks enable training with more stability. It yields I-AUROC of 99.41% on MVTec and 97.28% on VisA, comparable to the 99.45% and 97.5% of our reported result and still outperforms previous SOTAs. In the future, approaches to eliminating the problem of BN and optimizer can be further investigated.

smap by

smap by

smap by

smap by

ground

ground

image

truth

Lglobal

Lglobal-hm

image

truth

Lglobal

Lglobal-hm

500

F

image

ground truth

Smap

image

ground truth

Smap

image

ground truth

Smap

normal

(healthy)

anomalous

(unhealthy)

anomalous

(unhealthy)

APTOS

OCT2017

ISIC2018

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents an approach, called ReContrast, for domain-specific anomaly detection and segmentation. The authors aim to overcome the transfer limitations of existing methods by addressing and mitigating the challenges associated with the use of pre-trained encoders from natural image domains. They propose a combination of the reverse distillation approach and contrastive learning to optimize the entire network and align it with the target domain.\n\nIn the paper, the authors systematically explain their approach, highlighting three key elements: the introduction of a new optimization objective called global cosine distance to enhance training stability, the utilization of stop-gradient operation to prevent pattern collapse, and the integration of contrastive learning (without the use of negative samples). By integrating these elements into their feature reconstruction framework, the authors improve the network's transfer ability for anomaly detection tasks.\n\nSTRENGTHS:\n1. The authors address and analyze the limitations of the reverse distillation approach by incorporating simple yet effective modifications. This results in an improved anomaly detection method.\n2. The paper is well-written, and provides a clear and concise description of the proposed ReContrast framework and its underlying principles.\n3. The experimental evaluations conducted across diverse datasets and models demonstrate the practicality and effectiveness of the approach.\n\nWEAKNESSES:\n1. The authors handle training instability by selectively changing the batch normalization (BN) mode of the encoder based on performance on specific datasets or categories. This raises concerns about the proposed method's reliability and generalizability. Using the test set to modify the training process of problematic categories can introduce unintended biases and compromise the evaluation integrity. It would be more desirable to provide a systematic and principled approach to utilizing BN layers without relying on a separate validation set. This would ensure consistent and unbiased training across all datasets and categories. Consequently, it is important to report the performance of the proposed approach when utilizing the train and eval modes consistently for all datasets. Such analysis would provide valuable insights into the observed performance drops that the authors mentioned and help assess the robustness of the proposed method.\n\n2. Another limitation of the paper is the narrow focus on structural anomalies, as evidenced by the results presented in Table A6. While the proposed method performs well at detecting structural anomalies, it falls short when it comes to detecting logical anomalies. It is important to highlight this limitation in the main paper since unsupervised anomaly detection encompasses various anomaly types, including global anomalies like those found in datasets such as CIFAR-10 ([24-26] in the appendix). Clarifying the scope and limitations of the method for detecting different anomaly types will provide a more comprehensive understanding of its applicability in real-world scenarios.\n\n\\#### Update #### \\\nI've increased my rating slightly in light of the response.\n\nQUESTIONS:\nThe paper lacks of reported results in the multi-class setting, where anomaly detection is performed across multiple target classes simultaneously. This setting ([9] in the manuscript), reflects a more realistic scenario of anomaly detection in real-world applications. By learning the data distribution of multiple classes jointly, the detection task becomes more complex and challenging. However, the paper does not provide insights into the performance of the proposed ReContrast approach in this specific setting. What is the performance of the proposed ReContrast approach in this setting?\n\nI am willing to improve my score once these points (including weaknesses) are clarified.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-20"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes an epistemic method, namely ReContrast, for unsupervised anomaly detection (UAD). The proposed method is deeply based on UniAD and RD4AD, with some technical improvement to adapt to the data from different domains. Experiments are conducted on two popular industrial defect detection benchmarks and three medical image UAD tasks.\n\nSTRENGTHS:\n1. The technical details are well illustrated in the paper.\n2. Ablation studies show the effectiveness of the proposed components.\n\nWEAKNESSES:\n1.\tThe paper lacks novelty and resembles more of a technical report than a research paper. The authors summarize their contributions in Lines 68-78, which primarily consist of detailed technical modifications. The proposed method heavily relies on existing methods UniAD and RD4AD, with some specific improvements such as generating contrastive pairs and utilizing a GAP-mimicking objective function.\n2.\tThis paper has room for improvement in terms of its writing. In Lines 42-59, the authors introduce several components aimed at enhancing performance. However, the relationships between these components are not clearly established, and the motivations behind their inclusion are not adequately explained.\n\nQUESTIONS:\n1.\tUnclear contributions compared to existing methods: The paper does not clearly differentiate its contributions from those of existing methods. Both feature reconstruction [9] and contrastive learning [a,b] have been extensively explored in the anomaly detection literature. Although the authors mention the usage of stop-gradient as their main contribution (Lines 75-76), it has already been employed in [a,b].\n2.\tMissing important experimental comparisons: While this paper heavily relies on UniAD [9], it fails to provide any comparisons with UniAD in Table 1, Table 2, and Table 3. Including such comparisons would be crucial for evaluating the proposed method against the baseline.\n3.\tPoor organization: The organization of Section 2.2-2.6 is confusing and lacks coherence. These subsections appear to be assembled randomly without a clear structure, and there is no apparent connection between them. It seems that existing methods and the authors' proposed methods are randomly mixed together, leading to a lack of clarity and logical flow.\n4.\tDomain-specific claims and evaluation: The authors assert that their method is domain-specific, claiming that the proposed components could help adapt the model to industrial or medical domains. However, it is unclear whether the trained model can be used simultaneously on both industrial datasets, i.e, MVTec and Visa. The authors should clarify why they separately evaluate these datasets. They are suggested to follow the approach of UniAD [9] to demonstrate if one model can effectively detect anomalies across multiple categories within a single dataset.\n\nReferences: \n[a] Focus your distribution: Coarse-to-fine non-contrastive learning for anomaly detection and localization. ICME 2022.\n[b] Registration-based Few-Shot Anomaly Detection. ECCV 2022.\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-23"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a novel unsupervised anomaly detection (UAD) method called ReContrast. The method addresses the issue of poor transferability of pre-trained encoders from natural image domains to target UAD domains such as industrial inspection and medical imaging. ReContrast optimizes the entire network to reduce biases towards the pre-trained image domain and aligns the network with the target domain. It combines feature reconstruction and contrastive learning to prevent training instability and pattern collapse. The method is evaluated on industrial defect detection benchmarks and medical image UAD tasks, demonstrating its superiority over current state-of-the-art methods.\n\nSTRENGTHS:\n1. The paper is well-written and easy to read.\n2. This paper propose a new unsupervised anomaly detection method, ReContrast. According to the suggested setup, it is technically valid and should work well.\n3. Sufficient experiments are done to demonstrate the effectiveness of the method, and the robustness of the method is proved by experiments on multiple data sets\n4. The results were strong. Performance is excellent in most cases.\n\nWEAKNESSES:\n1. In the ablation experiments of MVTec AD and APTOS, for example, the experiment on Global Cosine Similarity, the author conducted experiments on the original RD4AD, namely Config A and Config B. This is not sufficient to prove the validity of Global Cosine Similarity on ReContrast, and may be a special case on RD4AD Settings.\n2. The realistic contribution of this paper is relatively weak. Compared with other methods, the reconstructed method does not rely on the pre-trained model or the memory library in anomaly detection. The method in this paper relies on the pre-trained model to maintain the feature extraction capability of the encoder, which I think may affect the practical application of the model.\n\nQUESTIONS:\nI really appreciate the structure of this article. The author has done a lot of work in the demonstration experiment, but the method innovation and contribution are relatively limited, but the overall is worth learning. Here are some suggestions.\n1.Regarding the experiment of Global Cosine Similarity, the author conducted the experiment on the original RD4AD, namely Config A and Config B. It is suggested to add the experimental setting using Region Cosine Similarity to the experimental setting of Config E. To demonstrate the validity of Global Cosine Similarity on ReContrast, rather than a special case on RD4AD Settings.\n2.In the experimental part, I noticed that the P-AUROC/AUPRO index lacked two model indexes compared with I-AUROC, ADTR and CFlow AD, in the experiment conducted on MVTec AD dataset. I hope the author could provide completion to ensure the completeness of the experiment.\n3.It is hoped that the author can visualize how the model gets the Anomaly score map in the visual RD4AD in Figure 2 to ReContrast, so as to facilitate readers' understanding.\n\nLIMITATIONS:\nLimitations are clearly described and I don't expect negative societal impact of this work.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work proposes an unsupervised anomaly detection method using contrastive learning method to reduce the biases between the pre-trained image domain and the target domain. In the paper, different designs of reconstruction networks are introduced and compared in terms of detection performance, model coverage and feature diversity. Finally, authors adopt a contrastive learning method equipped with global cosine loss, hard example mining, as well as additional frozen pre-trained encoder branch. This work conducts extensive experiments across two industrial defect detection benchmarks and three medical image anomaly detection tasks to demonstrate the transfer ability on various image domains.\n\nSTRENGTHS:\n+ This work analyzes the domain gap between the pretrained image domain and the dataset image domain and tries a variety of modifications on distillation network to reduce the domain gap.\n+ Extensive experiments are conducted on MVTec AD, VisA, and three medical image datasets to demonstrate the transfer ability on various image domains.\n\nWEAKNESSES:\n- During both training and inference, the feature loss is calculated using six feature pairs from two encoders. The utilization of two encoders in the methods leads to an increase in parameters and inference time. However, this approach functions as an ensemble, contributing to improved performance. It is important to note that the ablation study does not provide results solely generated by the domain-specific encoder.\n- The training process for configuration E lacks explicit description in the text. In accordance with the information provided in Figure 2, the encoders are switched and repeated during training. However, the text does not provide a detailed explanation of this process.\n- Figure 4 displays the landscapes of S_map, which allows for a comparison between the two losses. According to the description, S_map represents the average anomaly score map of the APTOS dataset. However, the paper or reference [27] does not provide an introduction on how to interpret or visualize the figures, nor does it explain why average anomaly score maps from different images are used, considering that they may have different anomaly locations.\n\nQUESTIONS:\n- As the work follows the SimSiam’s stop-gradient operation to prevent model collapsing. How about using momentum encoder in Moco to train the domain-specific encoder?\n- In this framework, the encoder must be switched during training, why not merge the features from flexible encoder and frozen encoder and then fed to the neck and decoder? In this way a unified network can be achieved.\n\nLIMITATIONS:\nAuthors have addressed limitations of the method on training instability and logical anomaly detection.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-05"}

comments: SUMMARY:
This paper introduces a feature reconstruction model, namely ReContrast, for anomaly detection. By incorporating a set of techniques including feature global similarity, stop gradient, contrastive pair optimizations, the authors shows good performance of ReContrast on both industrial and medical datasets.

STRENGTHS:
1. The paper is easy to follow. The logic behind some technical improvements is well motivated.

2. Extensive experimentation on diverse anomaly detection datasets is conducted. The numerical results of ReContrast outperforms prior arts.

3. The analysis of limitation of RD4AD, especially the Global Cosine Similarity, is impressive.

WEAKNESSES:
1. The paper presentation should be improved. The current version is in a lab report format, outlining the observed issues and employing trial-and-error strategies to address them. It would be preferable to present the methodology in a more cohesive manner.

2. I find it unclear why Config. D would prevent model collapse. As the feature similarity loss can be back-propagated from the decoder to the encoder, the encoder and decoder may collaborate to achieve a trivial solution.

3. The motivation behind Config. E is also unclear to me. If the decoder's objective is to reconstruct the pre-trained encoder, what is the purpose of the trainable encoder? What is the objective function in Config. E?

4. Figure 4 illustrates the model landscape with respect to model's trainable parameters. A flatter surface indicates a more stable training process, but it does not necessarily indicate better generalization concerning input data.

QUESTIONS:
Please refer to question 2-4 in the weakness section.

LIMITATIONS:
The appendix discuses the limitations of this method.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-11