SlowGuess/NeurIPS_2023_OCR huggingface dataset:
STFPM During training, the student feature tries to align the distribution of training dataset with the teacher. During prediction, the input  $x$  is fed into both student and teacher feature extractors, where the student outputs  $f_{s}(x)$  and teacher outputs  $f_{t}(x)$ . For the anomaly scoring, it relies in a traditional density estimation approach. The assumption would be that the normal samples are mapped to the high density area where the student encoder and the teacher encoder are aligned, and the anomalous samples are mapped to the low density area of the student extractor. Therefore, the anomaly score is computed as the distance between  $f_{s}(x)$  and  $f_{t}(x)$ , i.e.  $AS(x) = d(f_s(x),f_t(x))$ . In this case the distance metric  $d$  is the  $l_{2}$ -norm.
Reverse distillation The anomaly scoring function here is derived from the standard anomaly scoring functions used in reconstruction-based anomaly detection algorithm. In particular, the anomaly score is defined as the distance between the encoded features and the reconstructed features from the decoder.  $AS(y) = d(f(y),\nu (\phi (f(y))$  , where  $\nu (\cdot)$  is the decoder,  $\phi (\cdot)$  is the distiller and  $f(\cdot)$  is the encoder. The idea behind this anomaly scoring function is that the decoder has learned to reconstruct normal samples due to training dataset, but isn't able to reconstruct anomalous samples. Therefore, anomalous samples would have a higher anomaly score.
CFA For CFA we use the standard image-level density estimation approach. In particular, the training samples are mapped to a feature map  $f(x)$  and clustered into  $k$  clusters using  $k$ -means. For the prediction, given an input  $y$ , its final feature  $f(y)$  is computed after feeding it into the feature extractor and descriptor. Then the  $d$  nearest cluster centers of  $f(y)$  would be selected, and the anomaly score for that sample is the mean distance to those  $d$  centers.  $AS(y) = \Sigma_{f_i \in N_k(x)} d(f_i, f(y))$ . In this case, the distance metric  $d$  is simply the  $l_2$  distance. The idea behind this approach is to assume that the anomalous samples would be mapped to low-density area in feature space, which are far away from all the cluster centers, while the normal samples would be mapped to high density area in feature space, which is close to the normal samples in training dataset.
Mean-shift and Red PANDA Both contrastive learning based methods used as baselines, namely, mean-shift (Reiss and Hoshen [2021]) and Red Panda (Cohen et al. [2023]) rely on finetuning an
encoder (both pre-trained or not) by grouping the set of feature vectors from images in the training data around a sub-region of the hypersphere centered in the origin. During prediction time, the most common approach to classify a new sample as anomaly or not is through the mean distance of the kNN normal images. Following the original work, in mean-shift,  $k$  was set to 2, and in Red Panda it was set to 1.
CSI CSI (Tack et al. [2020]) relied on only a vanilla contrastive loss between original samples, and highly augmented samples to serve as a proxy for abnormal samples. Similarly to the previous setting of mean-shift and Red Panda, this leads to a feature space that falls around the hypersphere centered in the origin, but not necessarily in the surface. However, operating with the underlying hypothesis that the highly augmented samples match the anomalies, the feature vectors from images in the training data are already being pushed to the diametrically opposite side of the hypersphere when compared to abnormal samples. Additionally, it was empirically verified that the norm of vectors of abnormal samples is much lower than that of in-distribution samples. This leads to a distance criterion that measures the closest training sample through the cosine similarity and the norm of the feature vector of the sample:  $\max_m\sin (f(x_m),f(x))\cdot |f(x)|$  , where  $f$  is the encoder that maps the input object,  $x$  , to its feature space, and sim is the cosine similarity.
I.3 Hyperparameters
As this work considered a novel setting where each anomaly detection method was evaluated for the first time, we modestly optimized hyperparameters. Our approach consisted of two primary steps. The first involved scaling up two key factors: (a) batch size, and (b) learning rate. Subsequently, we methodically scanned through an array of distinct parameters for each baseline model. These included the backbones ResNet18, ResNet34, ResNet50 and WideResNet50, alongside various anomaly scoring methodologies that leverage image-level, density estimation, reconstruction error, and pixel-wise density estimation approaches. An additional aspect of our study was an ablation analysis where the regularization weight was fine-tuned by sweeping through the set of values 0.001, 0.01, 0.1, 1, 10, 100. We adhered to the hyperparameters as depicted in the original works for all other variables and refrained from performing any further optimizations on them.
I.4 Backbone choice
One thing we notice during our experiments is that for models that rely on the pretrained backbones, the choice of backbone matters. For instance, for STFPM, the optimal choice was the simplest feature extractor ResNet18, but for reverse distillation, the optimal choice was WideResNet50. The choices seem to be model dependent more than dataset relevant. For more details on the chosen backbone for each method refer to Tab. 7
I.5 Computational resources
The complete project required 3400 hours of GPU usage throughout all experiments, covering development, testing, and comparisons. The resources supplied were part of a local custer, and consisted of two GPU models: the NVIDIA TITAN RTX and the NVIDIA Tesla V100.
I.6 Code and Licensing
The main Python libraries used in our implementation, were Pytorch, which is under a BSD-3 license<sup>1</sup>, and Pytorch Lightning, which is under Apache 2.0 license<sup>2</sup>.
Methods that were derived from the anomalous library (Akcay et al. [2022]), namely STFPM, reverse distillation, and CFA, were already implemented as a Pytorch Lightning Module, and are all under an Apache 2.0 license<sup>3</sup>. These were incorporated directly in our pipeline.
$^{1}$ https://github.com/pytorch/pytorch/blob/main/License
$^{2}$ https://github.com/Lightning-AI/lightning/blob/master/License
<sup>3</sup>https://github.com/openvinotoolkit/anomalib/blob/main/License
STFPM
Wang et al. [2021]	ReverseDistil
Deng and Li [2022]	CFA
Lee et al. [2022]	MeanShift
Reiss and Hoshen [2021]	CSI
Tack et al. [2020]	Red PANDA
Cohen et al. [2023]
Camelyon17 (3x96x96)
learning rate	10-2	10-2	10-5	10-3	10-3	10-4
optimizer	SGD	Adam	AdamW	Adam	Adam	SGD
batch size	32	32	16	32	64	128
backbone	ResNet18	WideResNet50	ResNet18	ResNet50	ResNet18	ResNet50
pretaining	True	True	True	True	False	True
best reg. weight	100	10	100	10	1	10
anomaly score	density estimation	reconstruction error	density estimation	density estimation	density estimation	density estimation
DiagViB-6 (MNIST) (3x256x256)
learning rate	10-1	10-2	-	10-3	10-3	3·10-4
optimizer	SGD	Adam	-	Adam	Adam	SGD
batch size	32	32	-	32	32	32
backbone	ResNet18	WideResNet50	-	ResNet50	ResNet18	ResNet50
pretaining	True	True	-	True	False	True
best reg. weight	1	10	-	1	1	10
anomaly score	density estimation	reconstruction error	-	density estimation	density estimation	density estimation
DiagViB-6 (Fashion-MNIST) (3x256x256)
learning rate	10-1	10-2	-	10-3	10-3	3·10-4
optimizer	SGD	Adam	-	Adam	Adam	SGD
batch size	32	32	-	32	32	32
backbone	RESNet18	WideRestNet50	-	ResNet50	ResNet18	ResNet50
pretaining	True	True	-	True	False	True
best reg. weight	0.1	1	-	10	1	1
anomaly score	density estimation	reconstruction error	-	density estimation	density estimation	density estimation
Table 7: A detailed summary of the hyperparameters used for each evaluated model across three datasets: Camelyon17, DiagViB-6 (MNIST), and DiagViB-6 (Fashion-MNIST). Parameters include learning rate, scheduler, optimizer, batch size, backbone, pretraining, regularization weight, and mmd kernel type, along with the type of anomaly score. Notably, the CFA model could not be successfully implemented for DiagViB-6 based experiments despite trying an extensive range of hyperparameter combinations. Models are referenced by their respective citations.
DCoDR (Kahana and Hoshen [2022]), which Red Panda is based from, was released under a Software Research License<sup>4</sup>. Our experiments for Red Panda were derived directly from the official repository. Mean-shifted contrastive learning was released under a Software Research License<sup>5</sup>. We re-implemented this method as a Pytorch Lightning Module, loosely following its original official implementation. DiagViB-6 (Eulig et al. [2021]) and the Camelyon17 (Koh et al. [2021]) datasets were also publicly released with a GNU Affero General Public License v3.0<sup>6</sup> and a MIT License<sup>7</sup>, respectively. Our implementation follows directly from its official repository.
<sup>4</sup>https://github.com/jonkahana/DCoDR/blob/main/License
$^{5}$ https://github.com/talreiss/Mean-Shifted-Anomaly-Detection/blob/main/License
$^{6}$ https://github.com/boschresearch/diagvib-6/blob/main/License
$^{7}$ https://github.com/p-lambda/wilds/blob/main/License