SlowGuess/NeurIPS_2023_OCR huggingface dataset:
ACR-DSVDD. We use the standard convolutional neural network architecture used in meta-learning. Specifically, the network contains four convolution layers. Each convolution layer is followed by a batch normalization layer and a ReLU activation layer. The final layer is a fully-connectly layer followed by a batch normalization layer. The center of DSVDD has the same dimension as the output of the fully-connected layer, which is 32. For CIFAR100/CIFAR100-C, each convolution layer has 128 kernels. For MNIST, Omniglot, and OrganA, each convolution layer has 64 kernels. Each kernel's size is  $3 \times 3$ . We use Adam with a learning rate of 0.003 on CIFAR100 dataset and  $1e - 4$  on all the other datasets.
ACR-BCE. We use the same network structure as ACR-DSVDD without the final batch normalization layer and the center. The final fully-connected layer has output dimension of 1. We train the model with binary cross entropy loss. We use Adam with a learning rate of 0.003 on CIFAR100 dataset and  $1e - 4$  on all the other datasets.
these are either classification datasets or datasets where one of the covariates is binned to provide classes.
${}^{9}$  This is the popular "one-vs-rest" testing set-up,which is standard in AD benchmarking. (e.g., [65])
F.2 Implementation Details on MVtec AD for Anomaly Segmentation
Training Protocols. Since the images are roughly aligned, we can use a sliding window over the image to detect local defects in each window. However, this requires pixel-level alignment and is unrealistic for the MVtec AD dataset. We instead first extract informative texture features using a sliding window, which corresponds to the 2D convolutions. The convolution kernel is instantiated with the ones in a pre-trained ResNet. We follow the same data pre-processing steps of Cohen and Hoshen [12], Defard et al. [15], Rippel et al. [62] to extract the texture representations (the third layer's output in our case) of WideResNet-50-2 pre-trained on ImageNet. Second, we detect anomalies in the extracted features in each sliding window position with our ACR method. Specifically, each window position corresponds to one image patch. We stack into a batch the patches taken from a set of images that all share the same spatial position in the image. For example, we may stack the top-left patch of all testing wood images into a batch and use ACR to detect anomalies in that batch. Finally, the window-wise anomaly scores are bilinearly interpolated to the size of the original image, i.e., the pixel-level anomaly scores.
Following the same data pre-processing steps of Cohen and Hoshen [12], Defard et al. [15], Rippel et al. [62], we extract the texture representations (the third layer's output in our case) of WideResNet-50-2 pre-trained on ImageNet. After feature extraction, a batch of  $B$  images leads to a representation of size  $(B, C, H, W) \coloneqq (B, 1024, 14, 14)$ . These representations contain both textual and spatial information. During meta-training, we treat each spatial position as one new class so that there are  $14 \times 14 = 196$  new classes for each original class (e.g., wood), and each new class contains  $B$  data points, each a 1024 long vector. As a result, the model (DSVDD in our usage) takes a batch of vectors of size  $(B, 1024)$  as input and assigns anomaly scores to each vector within the batch. When adding synthetic abnormal data (Eq. (6)), we use Gaussian noise corrupted input vectors rather than new class data, incorporating the fact that local defects result in similar feature vectors instead of globally different textures. Specifically, we add Gaussian noise sampled from  $\mathcal{N}(0, 0.01I)$  and set  $\pi = 0.5$  in Eq. (6). Since there is no During testing, we batch each position  $(h, w)$  of all images and detect anomalies at position  $(h, w)$ . Because the defects are local, and there is a possibility that there is no defect at some position  $(h_0, w_0)$ , we manually add synthetic noisy vectors into the tested batch as the training procedure to ensure the images have a low anomaly score at  $(h_0, w_0)$ . After getting anomaly scores, we remove the synthetic vector results, leading to scores of size  $(B, 14, 14)$ , and then upscale the scores into the original image size by bilinear interpolation. We acknowledge that using CutPaste [45] to generate more realistic synthetic abnormal samples is another option. We leave the investigation to future work.
ACR-DSVDD and Hyperparameter Search. Our model is a five-layer MLP with intermediate batch normalization layers and ReLU activations. The hidden sizes of the perceptrons are [512, 256, 128, 64, 32]. The center is size 32. The statistics of all batch normalization layers are computed on fly on the training/test batches. We average the gradients of 32 randomly sampled tasks for each parameter update. Each task contains 30 normal feature vectors and 30 noise-corrupted feature vectors. We train the model with Meta Outlier Exposure loss. We set the learning rate 0.0003 and iterate 50 updates for each class. We search the hyperparameters on a test subset of bottle class (half of the original test set) and apply the same hyperparameters to all classes afterward.
F.3 Implementation Details on Tabular Data
ACR-NTL has the same model architecture as the baseline NTL, and ACR-DSVDD adds one additional batch normalization layer on top of the DSVDD baseline. Our algorithm is applicable to the existing backbone models without complex modifications.
ACR-DSVDD. ACR is applied to the backbone model DSVDD [63]. The neural network of DSVDD is a four-layer MLP with intermediate batch normalization layers and ReLU activations. The hidden sizes on Anoshift dataset are [128, 128, 128, 32]. The hidden sizes on Malware dataset are [64, 64, 64, 32]. One batch normalization layer is added on the top of the network on Anoshift experiment. The statistics of all batch normalization layers are computed on fly on the training/test batches. We use Adam with a learning rate of  $4e - 4$  on Anoshift dataset and  $1e - 4$  on Malware dataset.
ACR-NTL. ACR is applied to the backbone model NTL [56]. The shared encoder of NTL is a four-layer MLP with intermediate batch normalization layers and ReLU activations. The hidden sizes of the encoder are [128, 128, 128, 32]. The statistics of all batch normalization layers are computed on fly on the training/test batches. We set the number of neural transformations as 19. Each neural transformation is parametrized by a three-layer MLP of hidden size of 128 with ReLU activations. All networks are optimized jointly with Adam with a learning rate of  $4e - 4$ .
G Meta Outlier Exposure Avoids Trivial Solutions.
The benefit of the outlier exposure loss in meta-training is that the learning algorithm cannot simply learn a model on the average data distribution, i.e., without learning to adapt. This failure to adapt is a common problem in meta-learning. Our solution relies on using each training sample  $\mathbf{x}_i$  in different contexts: depending on the sign of  $y_{i,j}$ , data point  $\mathbf{x}_i$  is considered normal (when drawn from  $P_j$ ) or anomalous (when drawn from  $\bar{P}_j$ ). This ambiguity prevents the model from learning an average model over the meta data set and forces it to adapt to individual distributions instead.
For example, DSVDD with its original loss function may suffer from a trivial solution that maps any input data to the origin in the feature space and achieves the optimal zero loss [63]. This trivial solution is also possible in our proposed meta-training procedure. But Meta Outlier Exposure gets rid of this trivial solution because mapping everything to the origin incurs an infinite loss on  $\mathbf{A}_{\theta}$ . Similar reasoning also applies to binary cross entropy loss.
H Connections to Other Areas
Our problem setup and assumptions share similarities with other research areas but differences are also pronounced.
Connection to Batch Normalization-Based Test-time Adaptation (TTA). Many works for TTA feature batch-level predictions [10, 49, 53, 67, 76] assumes its test-time data are corrupted but from the same semantic classes as the training data, but the zero-shot AD's test data can be drawn from a completely new class.
Connection to Unsupervised Domain Adaptation (UDA). Although our approach uses unlabeled data like the UDA setting [38], UDA assumes the unlabeled data from the shifted domain is available during training and can be used to update the model parameters. But our method doesn't rely on the availability of novel data during training time and doesn't require updating the model parameters during test time.
Connection to Zero-shot Classification. Xian et al. [79] explains the nature of zero-shot classification and writes that "the crux of the matter for all zero-shot learning methods is to associate observed and non-observed classes through some form of auxiliary information which encodes visually distinguishing properties of objects." The auxiliary information demands extra human annotations like picture attributes. In contrast, our method assumes a batch of test data without human annotations. The test distribution information is automatically contained in the batch statistics.
Connection to Meta-learning. Although we assume that there is a meta-training dataset available like meta-learning, we don't require a support set for updating the model during both training time and test time. The presence of a support set differentiates our method from meta-learning in many aspects. First, for the most well-known technique (MAML) in meta-learning, training requires second-order derivative information of the support set loss function, which is computationally expensive and slows the optimization. Second, it is unclear how to select the support set size for adaptation. Sometimes, it may require a large support set to achieve good adaptations. For example, OC-MAML needs at least a 10-shot support set to perform on par with our method on CIFAR100-C; Third, the support set requires labeled data. This already adds burdens to practitioners. Fourth, the model parameter updates require additional maintenance and extra cost during testing.
Connection to Contextual AD. Contextual AD considers a changing notion of normality based on context [25, 74]. In contextual AD, the training and testing data are from the same data generating process, which involves (hidden or observed) contextual variables controlling the generation. This is different from our setup. We tackle the problem when the training and testing data are from different data generating processes.
Table 4: AUC (%) with standard deviation for anomaly detection on CIFAR100-C and Omniglot. As an ablation, rather than utilizing outlier exposure, we trained Zero-shot BN only on normal data of each task.
CIFAR100-C (Gaussian Noise)	Omniglot
1%	5%	10%	20%	5%	10%	20%
One-class loss
(data-adapted) ResNet152	72.2±2.2	73.9±1.4	74.2±0.9	73.8±0.3	96.2±1.0	96.4±0.8	96.2±0.8
70.9±2.2	67.6±0.2	67.0±0.7	64.9±0.5	99.2±0.2	99.1±0.1	99.0±0.1
Table 5: The effects of batch normalization for zero-shot AD. The first two columns show different combinations of batch normalization usage during training and testing. The third column answers the question whether the type of batchnorm usage works for zero-shot AD.
BatchNorm (train)	BatchNorm (test)	Work?
✓	✓	Yes
✓	✗	No
✗	✓	No
✗	✗	No
Table 6: The effects of test time batch size on the results of zero-shot AD. We report the test results in AUC when the contamination ratio is set to  $5\%$  and  $10\%$ . The studies are conducted on the Gaussian noise version of CIFAR-100C. On the extreme batch sizes, each batch contains one anomaly.
Batch size	3	6	11	16
One anomaly	66.4±2.3	77.9±2.8	82.3±2.7	84.8±2.0
Batch size	20	40	60	80	100
5%	83.7±1.9	85.3±1.1	85.6±1.2	85.9±0.8	85.6±0.7
10%	84.5±1.5	86.1±0.8	85.7±0.7	85.8±0.5	85.8±0.6
Table 7: The effects of the number of classes used in training on zero-shot AD. We report the test results in AUC when the contamination ratio is fixed to  $10\%$ . The studies are conducted on the Omniglot dataset.
#Training classes	1	2	5	10	15
AUROC	59.0±0.6	71.8±0.6	72.5±0.3	72.2±1.0	75.3±0.4
#Training classes	20	40	80	160	320	640	1200
AUROC	79.0±1.0	90.5±0.5	95.3±0.2	97.6±0.2	98.1±0.2	98.4±0.1	99.1±0.2
I Additional Results
I.1 Ablation Study
Training with Different Losses. We study the benefits of using meta outlier exposure in Eq. (5) and compare to a) using one-class classification loss  $L[\mathbf{S}_{\theta}(\mathbf{x}_{\mathcal{B}})] = \frac{1}{B}\sum_{i\in \mathcal{B}}\mathbf{S}_{\theta}^{i}(\mathbf{x}_{\mathcal{B}})$  with  $\mathbf{S}_{\theta}^{i}(\mathbf{x}_{\mathcal{B}}) = ||\phi_{\theta}^{i}(\mathbf{x}_{\mathcal{B}}) - \mathbf{c}||^{2}$  where  $\phi_{\theta}$  is the feature map, b) (data-adapted) ResNet152. The data-adapted ResNet152 first learns the features by performing a multi-class classification task with the meta-training set. Then a batch normalization layer is applied on the top of penultimate layer representations for zero-shot anomaly detection. We train a 100-class classifier for CIFAR100C and Omniglot separately. Note that for Omniglot, we randomly sub-sample 100 classes from its 1400 training classes and train the classifier. From the results in Tab. 4 we can see that both ablations perform competitive with ACR on the simple Omniglot dataset, but perform much worse compared to ACR on the complex CIFAR100-C dataset. In conclusion, using meta outlier exposure in training is favorable.
Training or Testing Without BatchNorm. We investigate whether training or testing without batch normalization works for zero-shot AD or not. To this end, we employ four different combinations
of batch normalization usage during training and testing and check which combination works and which doesn't. We trained the models with the same meta-training procedure as what we used in Sec. 4.1.1 and tested on CIFAR100-C and Omniglot. We present the results in Tab. 5. In the third column, "Yes" indicates the AUROC metric is significantly larger than 0.5, and therefore learns a meaningful zero-shot AD model; "No" indicates the AUROC performance is around 0.5, which means the predicted anomaly scores are just random guesses and the model cannot be used for zero-shot AD. Tab. 5 shows that only when the batch normalization is used both in training and testing, the zero-shot AD works. Otherwise, the meta-training procedure couldn't result in meaningful zero-shot AD representations.
Moreover, for the DSVDD model, we can theoretically show that training without batch normalization will not work with meta outlier exposure: the optimal loss function has nothing to do with zero-shot AD. Rather, the optimal loss is only related to the mixture weight  $\pi$  during training. Without loss of generality, suppose we have two training distributions  $P_{1}, P_{2}$ . We learn a Deep SVDD model parameterized by  $\theta$  and  $c$  by the meta outlier exposure method,
$$
\begin{array}{l} l (\theta , c) \\ = \mathbb {E} _ {x \sim P _ {1} ^ {\pi}} \left[ (1 - y _ {1}) (f _ {\theta} (x) - c) ^ {2} + \frac {y _ {1}}{(f _ {\theta} (x) - c) ^ {2}} \right] + \mathbb {E} _ {x \sim P _ {2} ^ {\pi}} \left[ (1 - y _ {2}) (f _ {\theta} (x) - c) ^ {2} + \frac {y _ {2}}{(f _ {\theta} (x) - c) ^ {2}} \right] \\ = \mathbb {E} _ {x _ {1} \sim P _ {1}, x _ {2} \sim P _ {2}} \left[ \pi \left(f _ {\theta} \left(x _ {1}\right) - c\right) ^ {2} + \frac {1 - \pi}{\left(f _ {\theta} \left(x _ {2}\right) - c\right) ^ {2}} + \pi \left(f _ {\theta} \left(x _ {2}\right) - c\right) ^ {2} + \frac {1 - \pi}{\left(f _ {\theta} \left(x _ {1}\right) - c\right) ^ {2}} \right] \tag {17} \\ = \sum_ {i = 1} ^ {2} \mathbb {E} _ {x _ {i} \sim P _ {i}} \left[ \pi \left(f _ {\theta} \left(x _ {i}\right) - c\right) ^ {2} + \frac {1 - \pi}{\left(f _ {\theta} \left(x _ {i}\right) - c\right) ^ {2}} \right] \\ \geq 4 \sqrt {\pi (1 - \pi)} \\ \end{array}
$$
where  $0.5 < \pi < 1$  implies the majority assumption and the equality holds when the model parameters  $(\theta$  and  $c)$  are tuned such that  $\left(f_{\theta}(x_i) - c\right)^2 = \sqrt{(1 - \pi)/\pi}$  for any  $x_i$ . All data points will be put at the hypersphere's surface centered around  $c$  with a radius  $\sqrt{(1 - \pi)/\pi}$  in the feature space when the model is trivially optimized. However, the optimal loss has nothing to do with distinguishing different distribution's input data  $x$  in the feature space, which is unlikely to produce useful representations for zero-shot AD.
On the other hand, if we apply batch normalization in the model  $f_{\theta}$ , we will not have the optimal loss function irrelevant to distributions. To see this, note that batch normalization will shift the input toward the origin. Thus  $x_{1}$  in training task  $P_{1}^{\pi}$  and  $x_{2}$  in training task  $P_{2}^{\pi}$  should have similar representations as they both take the majority in each task. Similarly,  $x_{2}$  in task  $P_{1}^{\pi}$  and  $x_{1}$  in task  $P_{2}^{\pi}$  are minorities, thus mapped far away from the origin in the feature space. Therefore, the symmetry breaks in Eq. (17), and the above trivial optimal loss disappears.
Ablation Study on Batch Sizes. To test the batch size effects, we add an ablation study on the Gaussian noise version of CIFAR-100C where we fix the anomaly ratio as  $5\%$  or  $10\%$  and try different batch sizes. The results are summarized in Tab. 6. It shows that larger batch sizes lead to more stable results. The performance is similar when the batch size is larger than or equal to 40. Even with a batch size being 20, our results are still better than the best-performing baseline.
We also test extreme batch sizes being 3, 6, 11, and 16 where each batch contains one anomaly.
Ablation Study on Number of Training Classes. To analyze the effect of the number of training distributions on the zero-shot AD performance, we conducted experiments on Omniglot where we varied the number of available meta-training classes from 1, 2, 5, 10, 15, 20, 40, 80 to 640, 1200. We separately trained ACR-DSVDD on each setup and tested the resulting models on the test set that has a  $10\%$  ground-truth anomaly ratio. We repeated 5 runs of the experiment with random initialization and reported the AUROC results in Tab. 7. It shows that using 320 available classes for this dataset is sufficient to achieve a decent zero-shot AD performance. The results also demonstrate that even though we have only one class in the meta-training set, thanks to the batch norm adaptation, we can still get better-than-random zero-shot AD performance.
Ablation Study on Other Normalization Techniques. As follows, we report new experiments involving LayerNorm, InstanceNorm, and GroupNorm for zero-shot AD.
We stress that, while these methods may have overall benefits in terms of enhancing performance, they do not work in isolation in our zero-shot AD setup. A crucial difference between these methods and batch normalization is that they treat each observation individually, rather than computing normalization statistics across a batch of observations. However, sharing information across the batch (and this way implicitly learning about distribution-level information) was crucial for our method to work.
Our experiments (AUROC results in the table below) with DSVDD on the Omniglot dataset support this reasoning. Using these normalization layers in isolations yields to random outcomes  $(\mathrm{AUROC} = 50)$ :
LayerNorm	InstanceNorm	GroupNorm
50.0±0.9	50.6±0.7	50.2±0.5
We also added a version of the experiment where we combined these methods with batch normalization in the final layer. The results dramatically improve in this case:
BatchNorm (BN)	LayerNorm + BN	InstanceNorm + BN	GroupNorm + BN
99.1±0.2	98.8±0.1	98.8±0.2	98.2±0.2
Experimental details: We use DSVDD as the anomaly detector and experiment on the Omniglot dataset. Each nonlinear layer of the feature extractor for DSVDD is followed by the respective normalization layer. We apply the same training protocol as Tab. 9 in the paper. For GroupNorm, we separate the channels into two groups wherever we apply group normalization.
Effects of BatchNorm (BN) Layer Position. We conducted additional experiments on two visual anomaly detection tasks – anomaly segmentation on the MVTec-AD dataset and object-level AD on CIFAR100C. We used the same DSVDD model architectures as used in Tables 1 and 2 as the backbone model, except that we switched off BN in all but one layer. For anomaly segmentation, there are five possible BN layer positions; and there are four positions for the object-level AD model. We switched off the BN layers in all but one position and then re-trained and tested the model with the same protocol used in our main paper (For CIFAR100C, we tested the model with the test data anomaly ratio of  $10\%$ ). We iterate this procedure across all available BN layer positions. We repeat every experiment with different random seeds five times and report the mean AUROC and standard deviation. The results are summarized in the tables below, where a smaller value of the BN position corresponds to earlier layers (close to the input), and a larger value corresponds to later layers close to the output. The final column is copied from our results in Tables 1 and 2 where BN layers are on all available positions. For both MVTec-AD and CIFAR100C, we average the performance across all test classes.
Results on the two tasks have opposite trends regarding the effects of BN layer positions. Specifically, for anomaly segmentation on MVTec-AD, earlier BN layers are more effective, while for AD on CIFAR100C, later BN layers are more effective. This observation can be explained by the fact that anomaly segmentation is more sensitive to low-level features, while object-level AD is more sensitive to global feature representations. In addition, compared to the results in Tables 1 and 2 (copied to the last column in the table below), our results suggest that using BN layers at multiple positions does help re-calibrate the data batches of different distributions from low-level features (early layers) to high-level features (late layers) and shows performance improvement over a single BN layer.
MVTec-AD
BN Position	1	2	3	4	5	(1,2,3,4,5)
Pixel-level	80.8±1.9	69.6±1.4	73.9±0.9	63.6±1.6	60.9±0.8	92.5±0.2
Image-level	74.7±0.9	59.2±1.6	63.6±1.3	65.5±1.2	65.4±1.3	85.8±0.6
CIFAR100C
BN Position	1	2	3	4	(1,2,3,4)
AUROC	61.4±0.5	61.0±0.9	68.2±0.9	68.9±1.1	85.9±0.4
Figure 3: 2D visualization (after PCA) of the adaptively centered representations for two test tasks in the Omniglot dataset. The same learned DSVDD model adapts with our proposed method and maps samples from the majority class (class 1 (left) and class 2 (right)) to the same center in the embedding space in both tasks.
Robustness of the mixing hyperparameter  $\pi$  in Eq. (6). We conduct the following experiments with varying  $\pi$ . The experiment has the same setup as Table 1 on CIFAR100C with a testing anomaly ratio of 0.1. The results show that all tested  $\pi$ 's results are over  $84\%$  AUC.
CIFAR100C
π	0.99	0.95	0.9	0.8	0.6
AUROC	85.8±0.5	85.4±0.5	84.1±0.4	85.9±0.4	84.4±0.6
I.2 Visualization of ACR.
We provide a visualization of the learned representations from DSVDD on the Omniglot dataset as qualitative evidence in Fig. 3. We observe that even though the normal and abnormal data classes flip in two plots, the model learns to center the samples from the majority class and map the samples from the minority class away to the center in the embedding space. In conclusion, ACR is an easy-to-use zero-shot AD method and achieves superior zero-shot AD results on different types of images. The performance of ACR is also robust against the test anomaly ratios.
I.3 Additional Results on CIFAR100-C
We test all methods on all corruption types of CIFAR100-C. The results are presented in Tab. 8.
I.4 Additional Results on Non-natural Images
Datasets. We further evaluate the methods on two other non-natural datasets—MNIST and Omniglot of hand-written characters. MNIST uses the same split and evaluation protocol as OrganA. On Omniglot, we take the first 1200 classes to form the meta-training set and use the remaining unseen 423 classes for testing.