SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Jia Guo<sup>1</sup> Shuai Lu<sup>2</sup> Lize Jia<sup>2</sup> Weihang Zhang<sup>2</sup> Huiqi Li<sup>1,2*</sup>  
<sup>1</sup>School of Information and Electronics, Beijing Institute of Technology, China  
<sup>2</sup>School of Medical Technology, Beijing Institute of Technology, China  
guojia.jeremy@gmail.com, lushuaie@163.com  
{3220212096, zhangweihang, huiqili}@bit.edu.cn
Most advanced unsupervised anomaly detection (UAD) methods rely on modeling feature representations of frozen encoder networks pre-trained on large-scale datasets, e.g. ImageNet. However, the features extracted from the encoders that are borrowed from natural image domains coincide little with the features required in the target UAD domain, such as industrial inspection and medical imaging. In this paper, we propose a novel epistemic UAD method, namely ReContrast, which optimizes the entire network to reduce biases towards the pre-trained image domain and orients the network in the target domain. We start with a feature reconstruction approach that detects anomalies from errors. Essentially, the elements of contrastive learning are elegantly embedded in feature reconstruction to prevent the network from training instability, pattern collapse, and identical shortcut, while simultaneously optimizing both the encoder and decoder on the target domain. To demonstrate our transfer ability on various image domains, we conduct extensive experiments across two popular industrial defect detection benchmarks and three medical image UAD tasks, which shows our superiority over current state-of-the-art methods. Code is available at: https://github.com/guojiajeremy/ReContrast
Unsupervised anomaly detection (UAD) aims to recognize and localize anomalies based on the training set that contains only normal images. UAD has a wide range of applications, e.g., industrial defect detection and medical disease screening, addressing the difficulty of collection and labeling of all possible anomalies.
Efforts on UAD attempt to learn the distribution of available normal samples. Most state-of-the-arts utilize networks pre-trained on large-scale datasets, e.g. ImageNet [1], for extracting discriminative and informative feature representations. Feature reconstruction [2; 3; 4] and feature distillation methods [5; 6] are proposed to reconstruct features of pre-trained encoders as alternatives to pixel reconstruction [7; 8], because learned features provide more informative representations than raw pixels [2; 3; 9]. The above approaches can be categorized as epistemic methods, under the hypothesis that the networks trained on normal images can only construct normal regions, but fail for unseen anomalous regions. Feature memory & modeling methods [10; 11; 12; 13; 14] memorize and model all anomaly-free features extracted from pre-trained networks in training, and compare them with test features during inference. In feature reconstruction, the parameters of encoders should be frozen to prevent the networks from converging to a trivial solution, i.e., pattern collapse [2; 3]. In feature distillation and feature memory & modeling methods, there are no gradients for optimizing
Figure 1: Comparison of architectures. (a) Contrastive learning [16; 17]. (b) Feature reconstruction UAD [2; 9]. (c) Proposed ReContrast. Two decoders share the same weights.  $\mathbf{z}$  is the output of encoder.  $\mathbf{q}$  is the output of predictor or decoder.  $\mathbf{z}'$  and  $\mathbf{q}'$  is another view of  $\mathbf{z}$  and  $\mathbf{q}$ , respectively.
feature encoders. Unfortunately, poor transfer ability is caused by the semantic gap between natural images on which the frozen networks were pre-trained and various UAD image modalities.
Most recently, sparse studies have addressed the transfer problem in UAD settings. CFA [14] and SimpleNet[15] unanimously utilize a learnable linear layer to adapt the output features of the encoder to task-oriented features, while still keeping the encoder frozen. However, the layer after a frozen encoder may be insufficient for adaptation as it can hardly recover the domain-specific information already lost during extraction, especially when the target domain is far from natural images, e.g. medical images. More related works are presented in Appendix A.
In this work, we focus on orienting the whole UAD network in target domains. We propose Reconstructive Contrastive Learning (ReContrast) for domain-specific anomaly detection by optimizing all parameters end-to-end. We start with epistemic UAD methods that detect anomalies by reconstruction errors between the encoder and the reconstruction network (decoder) [2; 9]. By inspecting the behavior of encoder, we find that direct optimization of the encoder causes diversity degradation of encoder features and thus harms the performance. Accordingly, we introduce three key elements of contrastive learning (Figure 1(a)) into feature reconstruction UAD (Figure 1(b)), building a 2-D feature map contrast paradigm to train the encoder and decoder jointly on target images without obstacles. First, to mimic the role of global average pooling (GAP), a new optimization objective, namely global cosine distance, is proposed to make the contrast in a global manner to stabilize the optimization. Second, we bring in the essential operation, i.e., stop-gradient, which prevents positive-pair contrastive learning [16; 17] from pattern collapse. Third, as image augmentation can introduce potential anomalies, we propose to utilize two encoders in different domains to generate representations in two views from the same input image. In addition, we propose a hard-mining strategy for optimizing normal regions that are hard to reconstruct, to magnify the difference between intrinsic reconstruction error and epistemic reconstruction error. It is noted that ReContrast is NOT a self-supervised learning method (SSL) for pre-text pre-training like SimSiam [16] and SimCLR [18], but an end-to-end anomaly detection approach.
To validate our transfer ability on various domains, we evaluate ReContrast with extensive experiments on two widely used industrial defect detection benchmarks, i.e., MVTec AD [19] and VisA [20], and three medical image benchmarks, including optical coherence tomography (OCT) [21], color fundus image [22], and skin lesion image [23]. Without elaborately designing memory banks or handcrafted pseudo anomalies, our proposed method achieves superior performance compared with previous state-of-the-arts. Notably, we present an excellent image-level AUROC of  $99.5\%$  on MVTec AD, which reduces the error of the previous best by nearly half. The contributions of this study can be summarized as:
- We propose a simple yet effective UAD method to handle the poor transfer ability of the frozen ImageNet encoder when applied to UAD images with a large domain gap. Targeting the similarity and difference between contrastive learning and feature reconstruction, three key elements are elegantly introduced into feature reconstruction UAD for optimizing the entire network without obstacles.
- Inspired by the GAP in contrastive learning, we design a new objective function, introducing globality into point-by-point cosine distance for improving training stability.
- We utilize the key operation of positive-pair contrastive learning, i.e., stop-gradient, to prevent the encoder from pattern collapse.
- We propose a novel way of generating contrastive pairs to avoid potential anomalies introduced by image augmentation.
2 Method: From Reconstruction to ReContrast
In the following subsections, we present an exploration journey starting from a conventional feature reconstruction baseline to our ReContrast. We propose three cohesive components inspired by the elements of contrastive representation learning and present them in a "motivation, connection, mechanism" manner. We believe such progressive exploration would illustrate the motivation of our method better.
Figure 2: From RD4AD to ReContrast. (a) Training configurations. (b) Calculation of anomaly map. The calculation and optimization of regional and global cosine distance is presented in Figure 3.
2.1 Preliminary: Reverse Distillation
First, we revisit a simple yet powerful epistemic UAD method based on feature reconstruction, i.e., Reverse Distillation (RD4AD) [2]. The method is illustrated in Config.A of Figure 2. RD4AD consists of an encoder (teacher), a bottleneck, and a reconstruction decoder (student). Without loss of generality, the first three layers of a standard 4-layer network are used as the encoder. The encoder extracts informative feature maps with different spatial sizes. The bottleneck is similar to the fourth layer of the encoder but takes the feature maps of the first three layers as the input. The decoder is the reverse of the encoder, i.e., the down-sampling operation at the beginning of each layer is replaced by up-sampling. During training, the decoder learns to reconstruct the output of the encoder layers by maximizing the similarity between feature maps. During inference, the decoder is expected to reconstruct normal regions of feature maps but fails for anomalous regions as it has never seen such samples.
Let  $f_{E}^{k}, f_{D}^{k} \in \mathbb{R}^{C^{k} \times H^{k} \times W^{k}}$  denote the output feature maps from the  $k^{th}$  layer of the encoder and decoder respectively, where  $C^{k}, H^{k}$ , and  $W^{k}$  denote the number of channels, height, and width of the  $k^{th}$  layer, respectively. Regional cosine distance [2; 6; 24] is used to measure the difference between  $f_{E}^{k}$  and  $f_{D}^{k}$  point by point, obtaining a 2-D distance map  $\mathcal{M}^{k} \in \mathbb{R}^{H^{k} \times W^{k}}$ :
Figure 3: Cosine distances. (a) Calculation of regional and global cosine distance. (b) Optimization of distance losses. The black arrows denote optimization direction. The black and red dotted circles indicate the range of whether the feature point pair is close enough to be selected as easy-normal.
$$
\mathcal {M} ^ {k} (h, w) = 1 - \frac {f _ {E} ^ {k} (h , w) ^ {T} \cdot f _ {D} ^ {k} (h , w)}{\left\| f _ {E} ^ {k} (h , w) \right\| \left\| f _ {D} ^ {k} (h , w) \right\|}, \tag {1}
$$
where  $\| \cdot \|$  is  $\ell_2$  norm, and a set of  $(h,w)$  locates a spatial point in the feature map. The calculation is depicted in Figure 3(a). A large value in  $\mathcal{M}^k$  indicates anomaly at the corresponding position. Final anomaly score map  $S^{map}$  is obtained by up-sampling all  $\mathcal{M}^k$  to image size and a sum operation. The maximum value of  $S^{map}$  is taken as the anomaly score for the image, denoted as  $S^{img}$ . During training, the anomaly maps are minimized on normal images by the regional cosine distance loss, given as:
$$
\mathcal {L} _ {\text {r e g i o n}} = \sum_ {k = 1} ^ {3} \frac {1}{H ^ {k} W ^ {k}} \sum_ {h = 1} ^ {H ^ {k}} \sum_ {w = 1} ^ {W ^ {k}} 1 - \frac {f _ {E} ^ {k} (h , w) ^ {T} \cdot f _ {D} ^ {k} (h , w)}{\left\| f _ {E} ^ {k} (h , w) \right\| \left\| f _ {D} ^ {k} (h , w) \right\|}. \tag {2}
$$
The optimization of  $\mathcal{L}_{\text {region }}$  is illustrated in the upper-left of Figure 3(b). Notably, though  $\mathcal{L}_{\text {region }}$  was proposed clearly as the loss function of RD4AD [2], the official code<sup>1</sup> implemented a different function that makes an extra dimension flatten operation (maybe by coding bug). However, we observe extreme instability during the training with  $\mathcal{L}_{\text {region }}$ , producing degenerated I-AUROC of  $95.3\%$  on MVTec AD compared with  $98.5\%$  reported in the paper. The I-AUROC of  $95.3\%$  is more consistent with other methods using  $\mathcal{L}_{\text {region }}$  [6; 24]. In the next subsection, we will show that the seemingly unreasonable "bug" in the code improves the performance by explicitly introducing globality into optimization.
2.2 Global Cosine Similarity
It is a convention to use global average pooling (GAP) to pool the 2-D feature map of the last network layer to a 1-D representation, in both supervised classification and self-supervised contrastive learning. In UAD, GAP will simply mess up all feature points together, losing the ability to distinguish normal and anomalous regions, as shown in the upper-right of Figure 3(b). Therefore, we try to directly bring globality into the loss function while maintaining the point-to-point correspondence between  $f_{E}^{k}$  and  $f_{D}^{k}$ . Let  $\mathcal{F}$  denote a flatten operation that casts a 2-D feature map  $f \in \mathbb{R}^{C \times H \times W}$  to a vector  $v \in \mathbb{R}^{CHW}$ . Our proposed global cosine distance loss is given as:
$$
\mathcal {L} _ {\text {g l o b a l}} = \sum_ {k = 1} ^ {3} 1 - \frac {\mathcal {F} \left(f _ {E} ^ {k}\right) ^ {T} \cdot \mathcal {F} \left(f _ {D} ^ {k}\right)}{\left\| \mathcal {F} \left(f _ {E} ^ {k}\right) \right\| \left\| \mathcal {F} \left(f _ {D} ^ {k}\right) \right\|}, \tag {3}
$$
<sup>1</sup>https://github.com/hq-deng/RD4AD
where the number of cosine dimension is  $C \cdot H \cdot W$  instead of  $C$  in  $\mathcal{L}_{region}$ , as depicted in Figure 3(a). Intuitively,  $\mathcal{M}^k$  is also minimized along with the minimization of  $\mathcal{L}_{global}$ . It is easy to prove that minimizing  $\mathcal{L}_{global}$  to 0 equals to minimizing  $\mathcal{M}^k$  to 0. Though losses are not completely optimized to 0 in neural network training, the regional  $S^{map}$  still works as the anomaly map. The model trained with  $\mathcal{L}_{global}$  is named as Config.B.
As shown in Figure 5(a), the performance of Config.B (blue) is much more favorable and stable than Config.A (purple) during training. Here, we briefly analyze the underlying mechanism by plotting the landscape of  $S^{map}$  (average of all samples) against model parameters near optimized minima in Figure 4. We observe two differences. First, the overall landscape of the model optimized by  $\mathcal{L}_{global}$  is flatter than that by  $\mathcal{L}_{region}$ , which implies  $S^{map}$  is more stable during training with  $\mathcal{L}_{global}$ . The instability of  $\mathcal{L}_{region}$  can be caused by the point-by-point distance, where the fitting of one region may cause under-fitting of the others.  $\mathcal{L}_{global}$  can be regarded as the distance between manifolds of feature points, as depicted in the lower-left of Figure 3(b), which measures the consistency globally instead of excessively focusing on individual regions. Second, the landscape around the final minima of the model trained by  $\mathcal{L}_{global}$  is sharper than that by  $\mathcal{L}_{region}$ . Previous explorations on the geometry of loss landscapes suggest that flat minima generalize better on unseen samples [25; 26], which is unwanted in UAD settings. The real mechanism can be more complicated and is still under further investigation.
(a)
Landscape of average  $S^{map}$  optimized by  $\mathcal{L}_{region}$
Figure 4:  $S^{map}$  landscape in parameter space (reduced to 2-dimension), on APTOS [22], visualized using [27].
(b)
Landscape of average  $S^{map}$  optimized by  $\mathcal{L}_{global}$
2.3 Feature Degradation of Optimized Encoder
It seems easy to adapt the network in the target domain by jointly optimizing the encoder and decoder, as illustrated in Figure 2 Config.C. However, previous attempts [9] suggest that training encoders would lead encoders to extract indiscriminative features that are easy to reconstruct, which is also described as pattern collapsing or converging to trivial solutions [2]. We observe a similar phenomenon that the training loss quickly decreases to nearly zero, as shown by the orange dotted line in Figure 5(a).
To inspect the behavior of the encoder, we probe the diversity of encoder feature maps by the standard deviation (std) of  $\ell_2$  normalized feature points  $f_{E}^{k}(h,w) / \| f_{E}^{k}(h,w)\|_{2}$ . The larger the std, the more discriminative the feature, and the more distinctive between different regions. We plot the feature diversity of outputs of the  $2^{nd}$  encoder layer during training in Figure 5(b). The std of Config.C drops quickly during training, while the std of Config.B keeps constant as a comparison. In addition, the std of Config.C does not completely collapse to zero, which explains the passable performance. The outputs of other middle layers follow the same trend. The observation indicates that the degeneration of performance is attributed to the degradation of encoder features. It is noted that the std starts from different values because of the different modes of batch normalization (BN) [28] during training.
$^{2}$ Encoder BN is in eval mode in Config.A using pre-trained running mean and variation, and it is in train mode using batch mean and variation if optimized.
(a) I-AUROC & loss
(b)  $std$  of  $f_{E}^{2}(h,w) / \left\| f_{E}^{2}(h,w)\right\|_{2}$
Figure 5: (a) I-AUROC and loss during training on APTOS[22]. (b) Feature diversity during training on APTOS[22]. (c) Example  $S^{map}$  of a Hazelnut, optimized by  $\mathcal{L}_{global}$  or  $\mathcal{L}_{global-hm}$ .
(c) visualization of  $S^{map}$
2.4 Stop Gradient
Contrastive learning for SSL maximizes the similarity between the representations of two augmentations (views) of one image while trying to avoid collapsing solutions in which all inputs are encoded to a constant value. SimSiam [16] indicated that the stop-gradient operation plays an essential role in preventing collapsing for SSL methods with positive pairs only [16; 17]. Intriguingly, we found that contrastive SSL can be explained from the perspective of feature reconstruction. The predictor (a multi-layer perceptron, MLP) can be regarded as a reconstruction network that constructs the representation of one view  $x$  to match the representation of another view  $x'$ , as in Figure 1. Vice versa, feature reconstruction UAD can be transformed into contrastive learning as well, regarding the reconstruction network as a predictor.
To this end, we introduce the stop-gradient operation into feature reconstruction, transforming it into a contrastive-like paradigm. The network is optimized by contrasting the feature maps of the encoder and decoder, as shown in Config.D of Figure 2. The gradients do not propagate directly into the encoder, but back into the encoder through the decoder. It is implemented by modifying (3) as:
$$
\mathcal {L} _ {\text {g l o b a l}} = \sum_ {k = 1} ^ {3} 1 - \frac {s g \left(\mathcal {F} \left(f _ {E} ^ {k}\right)\right) ^ {T} \cdot \mathcal {F} \left(f _ {D} ^ {k}\right)}{s g \left(\| \mathcal {F} \left(f _ {E} ^ {k}\right) \|\right) \| \mathcal {F} \left(f _ {D} ^ {k}\right) \|} \tag {4}
$$
where  $sg$  denotes the stop-gradient operation. The training of this configuration can be intuitively explained as a "mutual reinforcement". The optimization of  $f_{D}^{k}$  drives the encoder to be more specific and informative in the target domain by the gradients from the decoder, while the more domain-specific  $f_{E}^{k}$  of encoder requires further optimization of  $f_{D}^{k}$ . There are previous works trying to explain the deeper mechanism of stop gradient in contrastive learning. In [16], the authors hypothesize this configuration as an implementation of an Expectation-Maximization (EM) algorithm that implicitly solves two underlying sub-problems with two sets of variables. In [29], the authors argue that there are flaws in the hypothesis of [16] and suggest that the decomposed gradient vector (center gradient) helps prevent collapse via the de-centering effect.
As shown in Figure 5(a) and (b), Config.D can constantly extract more discriminative features without degeneration, boosting it to perform better than Config.C at the beginning of training. Previous works [9; 3] suggested that the reconstruction decoder may learn an "identical shortcut" that well reconstructs both seen and unseen samples, which explains the performance drop of Config.D after reaching the peak in Figure 5(a). To handle this issue, contrastive pairs are introduced.
2.5 Contrastive Pairs
Contrastive learning methods make use of (positive) contrastive image pairs, based on the intuition that the semantic information of different augmented views of the same image should be the same. Without image augmentations  $(x = x^{\prime})$ , contrastive learning degrades into a self-reconstruction paradigm that the predictor predicts the input of itself, which may also collapse into an "identical shortcut" just like Config.D. In epistemic UAD methods, it is impossible to construct contrastive pairs by image augmentation, because 1) any augmentation (flip, cutout, color jitter, etc.) could be potential
anomalies; and 2) the feature maps of two augmented views lost the regional correspondence after spatial augmentations (shift, rotate, scale, etc.). Therefore, we propose an augmentation-free method to construct two views of one image.
In previous explorations, the encoder is adapted to the target image domain and generates feature representations in a domain-specific view. We additionally introduce a frozen encoder network without any adaptation throughout the training, which perceives images in the view of pre-train image domain. As shown in Config.E of Figure 2, the decoder and bottleneck take the features of the domain-specific encoder to reconstruct the frozen encoder; vice versa, they simultaneously take the features of the frozen encoder to reconstruct the domain-specific encoder, building a complete two-view contrastive paradigm. This configuration can also be interpreted as cross-reconstruction, as the decoder reconstructs the information of the frozen encoder given the input of the adapted encoder; and reconstructs the information of the adapted encoder given the input of the frozen encoder. With six pairs of feature maps, the  $S^{map}$  is obtained by adding up the six up-sampled cosine distance maps. As shown in Figure 5(b), the encoder feature diversity of Config.E not only holds but slightly increases during training.
2.6 Magnify Epistemic Error by Hard-Normal Mining
Due to network information capacity, intrinsic reconstruction errors exist in available normal regions. The errors of normal details and edges (hard-normal) are generally higher than other plain regions (easy-normal), leading to confusion between the intrinsic error of hard-normal regions and the epistemic error of unseen anomalies, as demonstrated in the lower-left of Figure 5(c). It was proposed to hard-mine the hard-normal regions by simply discarding the "easy" pairs of  $f_{E}^{k}(h,w)$ ,  $f_{D}^{k}(h,w)$  with small cosine distance in  $\mathcal{L}_{\text{region}}$  [24]. However, arbitrarily discarding feature points in  $f_{D}^{k}$  breaks the global feature point manifolds in  $\mathcal{L}_{\text{global}}$ . Therefore, we propose a hard mining strategy, namely  $\mathcal{L}_{\text{global-hm}}$ , focusing on the optimization of hard-normal regions to widen the gap between the epistemic errors caused by unseen anomalies and the errors of all normal regions. The gradients of easy  $f_{D}^{k}(h,w)$  are discarded instead of eliminating  $f_{D}^{k}(h,w)$  itself, by modifying  $f_{D}^{k}$  as:
$$
f _ {D} ^ {k} (h, w) = \left\{ \begin{array}{c} s g \left(f _ {D} ^ {k} (h, w)\right), i f \mathcal {M} ^ {k} (h, w) <   \mu \left(\mathcal {M} ^ {k} (h, w)\right) + \alpha * \sigma \left(\mathcal {M} ^ {k} (h, w)\right), \\ f _ {D} ^ {k} (h, w), e l s e \end{array} \right. \tag {5}
$$
where  $\mu\left(\mathcal{M}^k(h,w)\right)$  is the average regional distance on the dataset (approximated on mini-batch),  $\sigma\left(\mathcal{M}^k(h,w)\right)$  is the standard deviation of the regional distance, and  $\alpha$  is a hyper-parameter to control the discarding rate. The optimization of  $\mathcal{L}_{global-hm}$  is depicted in the lower-right of Figure 3(b). As shown in Figure 5(c), the model trained by  $\mathcal{L}_{global-hm}$  activates fewer normal regions than the model trained by  $\mathcal{L}_{global}$ . The usage of hard mining strategy can also mitigate the over-fitting of easy training samples.