SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Table A2: Performance on MVTec AD over three runs  $(\%)$
I-AUROC	P-AUROC	AUPRO
Carpet	99.72 ± 0.27	99.27 ± 0.05	98.55 ± 0.88
Grid	100 ± 0.00	99.26 ± 0.05	97.79 ± 0.03
Leather	100 ± 0.00	99.48 ± 0.02	99.17 ± 0.04
Tile	99.66 ± 0.17	96.18 ± 0.11	92.95 ± 0.66
Wood	99.05 ± 0.05	95.94 ± 0.03	92.61 ± 0.10
Bottle	100 ± 0.00	99.00 ± 0.01	97.13 ± 0.08
Cable	99.58 ± 0.15	98.92 ± 0.02	95.63 ± 0.02
Capsule	97.86 ± 0.36	98.40 ± 0.00	95.34 ± 0.04
Hazelnut	100 ± 0.00	99.08 ± 0.03	95.87 ± 0.08
MetalNut	100 ± 0.00	98.73 ± 0.04	94.49 ± 0.09
pill	98.94 ± 0.24	99.13 ± 0.03	97.75 ± 0.05
Screw	97.80 ± 0.14	99.57 ± 0.02	98.50 ± 0.07
Toothbrush	99.44 ± 0.45	99.17 ± 0.02	94.99 ± 0.02
Transistor	99.65 ± 0.11	95.38 ± 0.03	82.61 ± 0.22
Zipper	99.68 ± 0.15	98.16 ± 0.04	95.12 ± 0.25
All Avg	99.42 ± 0.05	98.39 ± 0.01	95.21 ± 0.06
Table A3: Performance of 12 subset (categories) of VisA  $(\%)$ .
candle	capsules	cashew	chewinggum	fryum	macaroni1
I-AUROC	97.20	93.55	98.14	99.28	97.56	98.84
P-AUROC	99.15	99.46	97.41	97.36	91.96	99.01
AUPRO	94.77	94.45	94.28	86.63	79.11	93.63
macaroni2	pcb1	pcb2	pcb3	pcb4	pipe_fryum
I-AUROC	91.79	97.86	97.84	98.18	99.82	99.96
P-AUROC	99.04	99.79	98.99	99.05	98.70	98.30
AUPRO	97.36	96.76	92.47	95.14	91.29	95.57
D.2 Qualitative Visualization
The qualitative anomaly segmentation results of ReContrast trained by  $\mathcal{L}_{global}$  and ReContrast trained by  $\mathcal{L}_{global-hm}$  are presented in Figure A1. It is shown that the hard-normal regions are less activated with  $\mathcal{L}_{global-hm}$ . The results of VisA and medical images are shown in Figure A2 and Figure A3, respectively. Each  $S^{map}$  is min-max-normalized to 0-1 for clearer visualization.
D.3 Additional Ablation Study
Ablation study is conducted on the value of  $\alpha$  in  $\mathcal{L}_{global-hm}$ , as shown in Table A4. Assuming the regional cosine distances  $\mathcal{M}^k (h,w)$  conforms Gaussian distribution, the discarding rates of feature
point are  $2.3\%$ ,  $15.9\%$ ,  $50\%$ ,  $69.1\%$ ,  $84.1\%$ ,  $93.3\%$  and  $97.7\%$  for  $\alpha = -2, -1, 0, 0.5, 1, 1.5$  and 2, respectively.  $\mathcal{L}_{global-hm}$  is not sensitive to the discarding rate within the range from  $50\%$  to  $93\%$ . Though  $\alpha$  can be tuned for each dataset, we find  $\alpha = 1$  works just well for most circumstances.
We test a variety of encoder backbones in Table A5, i.e., ResNet18, ResNet50, and WideResNet50 (default), and report their performances, model parameters, as well as multiply-accumulate operations (MACs). The corresponding decoder is the reversed version of the encoder. Different encoders may favor different training hyper-parameters such as learning rate and iteration. Though we do not further tune each backbone, all backbones produce excellent results with default hyper-parameters, suggesting the generality of our method. With each backbone, our method outperforms the corresponding feature reconstruction counterpart RD4AD [2]. Notably, our method with ResNet18 is comparable to RD4AD with WideResNet50.
The utilization of two encoders (one frozen, one trained domain-specific) in our method can function as an ensembling, which may also contribute to performances. We conduct ablation experiments using different encoder-decoder feature map pairs to generate anomaly maps in inference. The results are shown in Table A6. On MVTec, using only the feature of domain-specific encoder produces an I-AUROC of  $99.38\%$ , which is comparable to the  $99.45\%$  of the ReContrast default and outperforms the  $98.86\%$  of baseline Config. B. In addition, we train an ensembling version of Config. B using two different encoders (ResNet50 and WideResNet50), producing an I-AUROC of  $98.94\%$  which is nearly identical to the baseline. On APTOS, using only the feature of domain-specific encoder produces an I-AUROC of  $97.73\%$ , which outperforms both  $97.51\%$  of the ReContrast default and  $92.49\%$  of the baseline Config. B. The results indicate that the improvement due to ensembling is small and the domain-specific encoder is vital.
Table A4: Ablation on the values of  $\alpha$  in  ${\mathcal{L}}_{\text{global-hm }}$  on MVtec AD (%)
α	-inf (Lglobal)	-2	-1	0	0.5	1	1.5	2
discard rate	0%	2.3%	15.9%	50%	69.1%	84.1%	93.3%	97.7%
I-AUROC	99.13	99.13	99.14	99.32	99.39	99.45	99.38	98.85
P-AUROC	98.09	98.11	98.13	98.30	98.31	98.37	98.39	98.25
AUPRO	94.59	94.62	94.65	94.97	95.00	95.20	95.28	95.13
Table A5: Ablation on the encoder backbones on MVtec AD  $(\%)$
Backbone	ResNet18	ResNet50	WResNet50
Method	RD4AD	Ours	RD4AD	Ours	RD4AD	Ours
Params(M)	18.7	21.7	63.6	74.9	117	145
MACs(G)	4.95	10.1	19.4	42.0	36.0	75.2
I-AUROC	97.9	98.7	98.4	99.1	98.5	99.5
P-AUROC	97.1	97.9	97.7	98.3	97.8	98.4
AUPRO	91.2	94.2	93.1	95.0	93.9	95.2
Table A6: Ablation on the use of different encoder feature maps for calculating  $S^{map}(\%)$ .
encoder feature maps for calculating S^map	MVTec AD	APTOS
Config. B	3 frozen	98.86	92.49
Config. B (R50+WR50)	6 frozen	98.94	92.14
Ours	3 frozen only	99.16	95.32
Ours	3 trained only	99.38	97.73
Ours	3 trained + 3 frozen	99.45	97.51
Scope of Application. In this work, we mainly focus on UAD that detects regional defects (most common in practical applications), which is distinguished from one-class classification (OCC, or
Semantic AD). In our UAD, normal and anomalous samples are semantically the same objects except local detects, e.g. good cable v.s. spoiled cable. In OCC, normal samples and anomalous samples are semantically different, e.g. cat v.s. other animals. The slight difference in task setting makes the focus of corresponding methods different. Most methods for UAD attempt to detect anomalies by segmenting anomalous local regions, while methods for OCC detect anomalies based on the representation deviation of the whole image [47; 48]. Though methods of these two tasks can be used for each other to a certain extent, we mainly focus on UAD in this work.
Logical Anomaly. MVtec LOCO is a recently released dataset for evaluating the detection performance of logical anomalies. It contains 5 sub-datasets, each comprised of both structural anomalies, e.g. dents and scratches as in MVtec AD, and logical anomalies, e.g. dislocation and missing parts. We find our method performing less favorably on logical anomalous images than GCAD [49] which is specially designed for logical anomalies, as presented in Table A7. We still outperform other vanilla UAD methods that are not specially designed for such anomalies.
Table A7: Anomaly Detection Performance on MVtec LOCO (%). Structural I-AUROC is calculated on normal images and structural anomalous images. Logical I-AUROC is calculated on normal images and logical anomalous images. Mean I-AUROC is the average of the above two scores.
RD4AD [2]	DRAEM [36]	PaDiM [10]	PatchCore [11]	GCAD [49]	Ours
struct. I-AUROC	88.0	74.4	70.5	82.0	80.6	90.7
logic. I-AUROC	69.4	72.8	63.7	69.0	86.0	73.4
mean I-AUROC	78.7	73.6	67.1	75.5	83.3	82.1
Batch Normalization and Training Instability. As discussed in Appendix C, training instability and performance drops are observed for a few categories when setting BN mode of the encoder to train. The phenomenon of training instability is observed in many deep learning tasks. However, validation sets are often allowed, which counteracts training instability and helps the choice of BN mode.
On the one hand, it can be caused by the limited feature diversity of the one-class UAD dataset. First, we recall the formulation of BN layer. Giving a  $d$ -dimensional feature  $x = (x^{(1)}, \dots, x^{(d)})$ , each dimension (channel) is normalized in training:
$$
\hat {x} ^ {(k)} = \frac {x ^ {(k)} - E [ x ^ {(k)} ]}{\sqrt {V a r [ x ^ {(k)} ] + \epsilon}}
$$
where the expectation  $E$  and variance  $Var$  are computed over the batch, and  $\epsilon$  is a small constant for numerical stability. In most computer vision datasets, an input batch contains a variety of categories. Each dimension is more or less activated by different image features so that  $Var[x^{(k)}]$  is not zero. However, because a UAD dataset contains only one type of category, there is a chance that a channel of pre-trained feature  $x^{(k)}$  is barely activated or equally activated, e.g. a channel sensitive to animals is dead on industrial objects. Thus, the denominator  $\sqrt{Var[x^{(k)}] + \epsilon}$  is nearly zero, causing malfunction of batch normalization. This problem can be fixed by using pre-trained statistical running_mean and running_var (eval mode), which loses the effect of BN.
On the other hand, this instability can be attributed to the feature of Adam optimizer. We found that a number of UAD methods [2; 14; 15] are also subject to training instability when running their code. Whether the last epoch (for reporting results) is in the middle of a loss spike and performance dip is strongly related to random seed. In some works [50; 51], the authors attribute the training instability to the historical estimation of squared gradients in Adam optimizer.
In our according explorations, we replace the batch variance  $Var\left[x^{(k)}\right]$  smaller than min(5e-4, running_yar) by pre-trained running_yar and we reset the historical state buffer (gradient momentum and second-order gradient momentum) of Adam every 500 iters. Without manually selecting BN mode, such tricks enable training with more stability. It yields I-AUROC of  $99.41\%$  on MVtec and  $97.28\%$  on VisA, comparable to the  $99.45\%$  and  $97.5\%$  of our reported result and still outperforms previous SOTAs. In the future, approaches to eliminating the problem of BN and optimizer can be further investigated.
Figure A1:  $S^{map}$  of our ReContrast trained by  $\mathcal{L}_{global}$  or  $\mathcal{L}_{global-hm}$ . Hard-normal regions are less activated with  $\mathcal{L}_{global-hm}$ .
Figure A2:  $S^{map}$  of our method on VisA.
Figure A3:  $S^{map}$  of our method on medical image datasets.