Zero-Shot Anomaly Detection via Batch Normalization
Authors: Aodong Li
Chen Qiu
Marius Kloft
Padhraic Smyth
Maja Rudolph
Stephan Mandt
Venue: NeurIPS 2023 poster (2023)
Keywords: deep anomaly detection
zero-shot learning
batch normalization
TL;DR: We show how an anomaly detector can be adapted to a new data distribution without retraining

Abstract: Anomaly detection (AD) plays a crucial role in many safety-critical application domains. The challenge of adapting an anomaly detector to drift in the normal data distribution, especially when no training data is available for the "new normal," has led to the development of zero-shot AD techniques. In this paper, we propose a simple yet effective method called Adaptive Centered Representations (ACR) for zero-shot batch-level AD. Our approach trains off-the-shelf deep anomaly detectors (such as deep SVDD) to adapt to a set of inter-related training data distributions in combination with batch normalization, enabling automatic zero-shot generalization for unseen AD tasks. This simple recipe, batch normalization plus meta-training, is a highly effective and versatile tool. Our results demonstrate the first zero-shot AD results for tabular data and outperform existing methods in zero-shot anomaly detection and segmentation on image data from specialized domains.

Zero-Shot Anomaly Detection via Batch Normalization
Aodong Li' UC Irvine
Chen Qiu" Bosch Center for AI
Marius Kloft TU Kaiserslautern
Padhraic Smyth
Maja Rudolph" Bosch Center for Al
Stephan Mandt UC Irvine
UC Irvine
Abstract
Anomaly detection (AD) plays crucial role in many domains. The challenge of adapting an anomaly detector to drift in the normal data distribution, especially when training data is available for the "new normal", has led to the development of zero-shot AD techniques. In this paper, we propose a simple yet effective method called Adaptive Centered Representations (ACR) for zero-shot batch-level AD. Our approach trains off fif-the-shelf deep anomaly detectors (such as deep SVDD) adapt combination with batch normalization, enabling automatic zero-shot generalization for unseen AD tasks. This simple recipe, batch normalization plus meta-tret veta-training, a highly effective and versatile tool. Our theoretical results guarantee the sero-shot generalization for unseen AD tasks; our empirical results demonstrate the first zero-shot AD results for tabular data and outperform existing methods in zero-shot anomaly detection and segmentation on image data from specialized domains. Code is at https: // thub. com/ coo/aodongll.
application
of inter-related training data distributions
shot-shotss -ahot-ad-ao via batch- norm
Introduction
Anomaly detection (AD)-the task of identifying data instances deviating from the norm significant role in numerous application domains, such fake review identification, bot detection in social networks, tumor recognition, and industrial fault detection. AD is particularly crucial in safety-critical applications where failing to recognize anomalies, for example, in chemical plant a self-driving car, can risk lives.
Consider medical setting where anomaly detector encounters a batch of medical images from different patients. The medical images have been recorded with new imaging technology different from the training data, or the patients are from demographic the anomaly detector has not been trained on. Our goal to develop an anomaly detector that can still process such data using batches, assigning low scores to normal images and high scores to anomalies (i.e., images that differ systematically) without retraining. anomalies are rare. Given a new batch of test data, zero-shot AD method [18, 36, 51, 71] has to detect which features are typical of the majority of normal samples and which features are atypical.
achieve this zero-shot adaptation, we exploit the fact that
We propose Adaptive Centered Representations (ACR), a lightweight zero-shot AD method that combines two simple ideas: batch normalization and meta-training. Assuming an overall majority of "normal" samples, randomly-samples batch will typically have more normal samples than anomalies. The effect of batch normalization is then to draw these normal samples closer to the center (in its recentering and scaling operation), while anomalies will end up further away from the center. Notably, this scaling and centering is robust to distribution shift in the input, allowing
Equal contribution Joint supervision.
Correspondence to: aodongl1 @uci.edu
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Festure space
Meta-training Set
Testing Distribution
PT
Traning Testing Norally
Normal
P
PS
Pg
P?
a) Demonstrations concrete examples of a meta- veeaatraining set and testing distribution. not necessary for the veta-training set include the exact types of samples encountered during testing. For instance, when detecting lions within geese, the training data does not need to include lions geese. b) Illustration of zero-shot batch-level AD with ACR using one-class classifier [63]. The approach encounters three tasks (Pi:. Eq. (6)) during training (black arrows) and learns to map each task's majority of samples (i.e., the normal samples) space. At test time (blue arrow), the learned model maps the normal (majority) samples to the same center and the distance from the center serves as AD score.
Figure I:
shared learned center in embedding
self-supervised anomaly detector to generalize to distributions never encountered during training. We propose a meta-training scheme to unlock the power of batch normalization layers for zero-shot AD. During training, the anomaly detector will see many different anomaly detection tasks, mixed from different choices for normal and abnormal examples. Through this variability in the training tasks, the anomaly detector will learn to rely as much as possible on the batch normalization operations in its architecture.
Advantages of ACR include that theoretically grounded, simple, Domain-independern and compatible with various backbone models commonly used in deep AD [56, 63]. Contrary to recent approaches based on foundation models [36], applicable only to images, ACR can be employed on data from any domain, such time series, tabular data, or graphs.
We begin by presenting our assumptions and method in Sec. 2. Next, with the main idea in mind, describe the related work in Sec. 3. We demonstrate the effectiveness our method with experiments in Sec. 4. Finally, we conclude our work and state the limitations and societal impacts.
Our contributions can be summarized as follows:
An effective new method. Our results for the first time show that training off-the-shelf deep anomaly detectors on meta-training set, using batch normalization layers, gives automatic zero- shot generalization for AD. For which we derive a generalization bound on anomaly scores.
Zero-shot AD tabular data. We provide the first empirical study of zero-shot AD on tabular data, where our adaptation approach retains high accuracy.
Competitive results for images. Our results demonstrate not only substantial improvement in zero-shot AD performance for non-natural images, including medical imaging but also establish a new state-of-the- -htttat in anomaly segmentation on the MVTec AD benchmark [5].
2
Method
We begin with the problem statement in Sec. 2.1 and then state the assumptions in Sec. 2.2. Finally we present our proposed solution Sec. 2.3. The training procedure outlined in Alg. 1 in Supp. C.
2.1 Problem Statement and Method Overview
We consider the problem of learning an anomaly detector that is required to immediately adapt (without any further training) when deployed new environment. The main idea is to use batch normalization as a mechanism for adaptive batch-level AD. For any batch of data containing mostly "normal" samples, each batch normalization shifts its inputs to the origin, thereby (1) enabling the discrimination between normal data and outliers/anomalies, and (2) bringing data from different distributions into a common frame of reference. (Notably, we propose applying batch norm in multiple layers for different anomaly scorers.) For the algorithm to generalize to unseen distributions,

---

we train our model on multiple data sets of "normal" data simultancously. making sure each training batch contains majority of related data points (from the same distribution) at time.
Fig. la illustrates this idea, where all distributions are exemplified based on the example of homoge- neous groups of animals (only dogs, only robins, etc.) The goal is to detect lion among geese, where neither geese nor lions have been encountered before. Fig. 1b illustrates the scheme based on the popular example of deep support vector data description (DSVDD) [63], where samples are mapped to pre-specified point in an embedding space and scored based on their distance to this point. All training distributions are mapped to the same point, as enabled through batch normalization.
2.2 Notation and Assumptions
To formalize the notion of a meta-training set, we consider distribution of interrelated data distribu- tions (previously referred to as groups) as commonly studied meta-learning and zero-shot learning [3, 21, 22, 32]. This inter-relatedness can be expressed by assuming that K training distributions Pi...., Pk and a test distribution P. are sampled from a meta-distribution Q:
Pi,",, Pk, P.
i.d.
Q.
We assume that the distributions in Q share some common structure, such that training a model on one distribution has the potential to aid in deploying the model on another distribution. For example, the data x could be radiology images from patients, and each P, or P. could be a distribution of images from specific hospital. These distributions share similarities but differ systematically because of differences in radiology equipment, calibration, and patient demographics" Each of the distributions P E Q defines different anomaly detection task. For each task, we have to obtain an anomaly scoring function Sg that assigns low scores to normal samples x and high scores to anomalies.
We now consider a batch B D of size B, taken from an underlying data D P of size N. The batch can be characterized by indexing data points from D:
B = (i1, ..... iB) Unif((1......)
(2)
We denote the anomaly scores on a butch level by defining vector-valued anomaly score
Sxx) (S (xs),"", Si" (xs));
indicating the anomaly score for every datum in a batch. By thresholding the anomaly scores St we obtain binary predictions of whether data point x; is anomalous in the context of batch Xg-
By conditioning on batch of samples, our approach obtains distributional information beyond single sample. For example, an image of a cat may normal in the context of a batch cat images, but it may be anomalous in the context of batch of otherwise dog images. This is different from current deep anomaly detection schemes that evaluate anomaly scores without referring to a context.
Before presenting a learning scheme of how to combine batch-level information in conjunction with established anomaly detection approaches, we discuss the assumptions that our approach makes. (The empirical or theoretical justifications as well as possibilities of removing or mitigating the assumptions can be found in Supp. A.)
A1 Availability of a meta-training set. As discussed above, we assume the availability of a set of interrelated distributions. The meta-set is used to learn a model that can adapt without re-training.
A2 Batch-level anomaly detection. As mentioned above, we assume we perform batch-level predic- tions at test time, allowing us to detect anomalies based on reference data in the batch.
A3 Majority of normal data. We assume that normal data points take the majority in every i.i.d. sampled test batch.
Due to the absence of anomaly labels (or text descriptions) at test-time, cannot infer the correct anomaly labels without assumptions A2 and A3. Together, they instruct us that given a batch of test examples, the majority of the samples in the batch constitute normal samples.
"The divergence between distributions can be much larger than shown in this example, See our experiments.
3

---

2.3 Adaptively Centered Representations
Batch Normalization as Adaptation Modules. An important component of our method batch normalization, which shifts and re-scales any data batch Xy to have sample mean zero and variance one. Batch normalization also provides a na've parameter-free zero-shot batch-level anomaly detector:
Snivee (x)) ||(x: jxxg 0x l2
where ji and 6² are the coordinate-wise sample mean and sample variance. jú is dominated by the majority of the batch, which by assumption A3, is the normal data. If the x; lie in an informative feature space, anomalies will have a higher-than than-usual distance to the mean, making the approach a simple, adaptive AD method, illustrated in Fig. 2 in Supp. D.
While the example provides proof of concept, in practice, the normal samples typically do not concentrate around their mean in the raw data space. Next, we integrate this idea into neural networks and develop an approach that learns adaptively centered representations for zero-shot AD.
Deep Models with Batch Normalization Layers as Scalable Zero-shot Anomaly Detectors. In deep neural networks, the adaptation ability is obtained for free with batch normalization layers [35]. Batch normalization has become a standard and necessary component to facilitate optimization convergence in training neural networks. In common neural network architectures [27, 33, 60), batch normalization layers are used after each non-linear transformation layer, making a zero-shot adaptation with respect to its input batch. The entire neural network, stacking up many non- non-lineat transformation and normalization layers, has powerful potential in scalable sero-shot adaptation and learning adaptation-neede needed feature representations for complex data forms.
Training Objective. As discussed above, we can instantiate Sg as a deep neural network with batch normalization layers and optimize the neural network weights O. We first provide our objective function and then the rationality. Our approach is compatible with a wide range of deep anomaly detection objectives; therefore we consider a generic loss function L[So(xa (xx) that is function of the anomaly score. For example, in many cases, the loss function to be minimized is the anomaly score itself (averaged over the batch).
The availability of a meta-data set (A1) gives rise to the following minimization problem:
0" arg min 1 Σ Bx- [[So()))
(5)
Typical choices for [[(S(( [x include DSVDD [64] and neural transformation learning (NTL.) [56].
Details and modifications of this objective will follow.
Why does it work? Batch normalization helps re-calibrate the data batches of different distributions into similar forms: normal data will center around the origin. Such calibration happens from granular features (lower layers) to high-level features (higher layers), resulting in powerful feature learning and adaptation ability². We visualize the calibration in Fig. 3 in Supp. I.2. Therefore, optimizing Eq. (5) are able to learn (locally) optimal Sg- that is adaptive to all K different training distributions. Such learned adaptation ability will be guaranteed to generalize to unseen related distributions P..
See Sec. 2.4 below and Supp. B for more details.
Meta Outlier Exposure. While Eq. (5) can be a viable objective, we significantly improve over it while avoiding trivial solutions*. The approach builds on treating samples from other distributions as anomalies during training. The idea is that the synthetic anomalies can be used to guide learning tighter decision boundary around the normal data [30]. Drawing on the notation from Eq. 1, we thus
simulate mixture distribution by contaminating each P, by admixing a fraction (1 -1-a0 T) << 1 of data
from other available training distributions. The resulting corrupted distribution P is thereby
Pj := x (1-
P-=
1 Eies K-
This notation captures the case where the training distribution is free of anomalies (N 1).
Wiitoout batch normalization, optimizing Fq. (5) can be meaningless for some objectives. See Supp. I.I. We explain how this objective avoids trivial solutions in Supp. G and show the benefits in Tab. 4 of Supp. I.I.

---

Next, we discuss constructing an additional loss for the admixed anomalies, whose identity is known at training time. As discussed in [30, 58], many anomaly scores S allow for easily constructing a score Ao(x xg) that behaves inversely. That means, we expect (x to be large when evaluated on normal samples, and small for anomalies. Importantly, both scores share the same parameters. In the context of DSVDD, we define Sş(xE) 1/Ag(XB) but other definitions are possible for alternative losses [58, 63, 64). Using the inverse score, we can construct a supervised AD loss on the meta training set as follows.
We define a binary indicator variable vj. indicating whether data point i normal or anomalous in the context of distribution P; (i.e.. vf = 0 iff x e Pÿ). We later refer to it as anomaly label. A natural choice for the loss in Eq. (5) is therefore
1
[[So(x))] B (111 (xB) y (xx)))
The loss function resembles the outlier exposure loss [30]. but as opposed to using synthetically generated samples (typically only available for images), we use samples from the complement Pj at training time to synthesize outliers. The training pseudo-code is in Alg. 1 of Supp. C.
In addition to DSVDD,
also study backbone models such as binary classifiers and NTL [56]. For NTL, we adopt the Sp and As used by Qiu et al. [58]. For binary classifiers, we set Sx -log (1 - ([o(x)))) and Ax(x) = =- log (Jo(x)).
Batch-level Prediction. After training, we deploy the model in an unseen production environment to detect anomalies in zero-shot adaptive fashion. Similar to the training set, the distribution will be mixture of new normal samples P. and an admixture of anomalies from distribution never encountered before. For the method work, we still assume that the majority of samples be normal (Assumption A3). Anomaly scores are assigned based on batches, as during training. For prediction, the anomaly scores are thresholded at a user-specified value.
Time complexity for prediction depends on the network complexity and is constant 0(1) relative to batch size, because the predictions can be trivially parallelized via modern deep learning libraries.
2.4 Theoretical Results
Having described our method, we now establish a theoretical basis for ACR by deriving a bounded generalization error on an unseen test distribution P.. We define the generalization error in terms of training and testing losses, i.e., we are interested in whether the expected loss generalizes from the meta-training distributions Pi...,, Px to an unseen distribution P..
To prepare the notations, we split So(x) into two parts: feature extractor [x( that spans from the input layer to the last batch norm layer and that performs batch normalization, and an anomaly score S(z) that covers all the remaining layers. We use Pj to denote the data distribution Py transformed by the feature extractor fe. We assume that PJ satisfies Egz- [z] = 0 and Varg- P3 [z]
1 for j == 1,..., K, * because fo ends up with a batch norm layer.
are large enough such that, for batches from each given distribution Pz. the mini-batch means and variances are approximately constant across batches. Furthermore, assume the loss function L[S())] is bounded by C for any z. Let |.lrv denote the total variation. Then, the generalization error upper bounded by
Theorem 2.1. Assume the
E P. (E((((())
1
E x
1
Σ
B M[S(())))]
SC
p
1
K
K
K
j-1
The proof is shown in Supp. B. Note that Thm. 2.1 still holds if P, or P. contaminated distributions PJ or Pč.
Remark. Thm. 2.1 suggests that the generalization error of the expected loss function bounded KK Σ Pj- While we leave a formal bound of the TV distance to future studies, the following intuition holds: since fa contains batch norm layers, the empirical distributions K E P3 and Pa will share the same (zero) mean and (unit) variance. If both distributions are dominated by their first two moments, we can expect the total variation distance to be small, providing an explanation for the approach's favorable generalization performance.
by the total variation distance between P: and 1
5

---

3 Related Work
Deep AD. Many recent advances in AD are built on deep learning methods [65] and early strategies used autoencoder [9, 55, 85] or density- density-based [13, 66] models. Another pioneering stream of research combined one-class classification [70] with deep learning [57, 63]. Many other approaches to deep AD are self-supervised, employing self-suuervised loss function to train the detector and score anomalies [4, 23, 31, 44, 56, 68, 72, 75].
All of these approaches assume that the data distribution will not change too much test time. However, in many practical scenarios, there will be significant shifts in the abnormal distribution and even the normal distribution. For example. Dragoi al. [17] observed that existing AD methods fail in detecting anomalies when distribution shifts occur in network intrusion detection. Another line of work in this context requires test-time modeling for the entire test set, e.g., COPOD [47], ECOD [48], and robust autoencoder [85], preventing real-time deployment.
Few-shot AD. Several recent works have studied adapting an anomaly detector to shifts by fine- tuning a few test samples. One stream of research applies model-agnostic meta learning (MAML) [21] to various deep AD models, including one-class classification [22], generative adversarial networks [52], autoencoder [78), graph deviation networks [16], and supervised classifiers [20, 84). Some approaches extend prototypical networks to few-shot AD [8. 40J. Kozerawski and Turk [39] learn linear SVM with a few samples on top of a frozen pre- tr-irrrneed feature extractor, while Sheynin et al. [73] learn a hierarchical generative model from few normal samples for image AD. Wang et al. [77] learn an energy model for AD. The anomalies are scored by the error of reconstructing their embeddings from set of normal features that are adapted with few test samples. Huang et al. [32] learn -ategory-agnostic model with multiple training categories (a meta set). At test time, a few normal samples from a novel category are used to establish an anomaly detector in the feature space. Huang et al. [32] does not exploit the presence of meta-set to learn stronger anomaly detector through synthetic outlier exposure. While meta-training for object-level anomaly detection (e.g.. [22]) is generally simpler (it is easy to find anomaly examples, i.e., other objects different from the normal one), meta- trat--taining for anomaly segmentation (c.g., [32) poses a harder task since image defects may differ from object to object (e.g.. defects in transistors may not easily generalize to subtle defects in wood textures). Our experiments found that using images from different distributions as example anomalies during training is helpful for anomaly segmentation on MVTec--DD (see Supp. 1.5).
In contrast to all of the existing few-shot AD methods, we propose a zero-shot AD method and demonstrate that the learned AD model can adapt itself to new tasks without any support samples.
Zero-shot AD. Foundation models pre-trained on massive training samples have achieved remark- able results on zero-shot tasks on images [37, (9, 82, 83). For example, contrastive language-image pre-training (CLIP) [59] is a pre-trained language-vision model learned by aligning images and their paired text descriptions. One can achieve zero-shot image classification with CLIP by searching for the best-aligned text description of the test images. Esmaeilpour al. [18] extend CLIP with a learnable text description generator for out-of-distribution detection. Liznerski et al. [51] apply CLIP for zero-shot AD and score the anomalies by comparing the alignment of test images with the correct text description of normal samples. In terms of anomaly segmentation, Trans-MM [7] is an interpretation method for Transformer-basee generate pixel-level masks of input images, which can be applied to CLIP. MaskCLIP [86] directly exploits CLIP's Transformer layer potential in semantic segmentation to generate pixel-level predic- tions given class descriptions. MAEDAY [71] uses the reconstruction error of a pre-trained masked autoencoder [28] to generate anomaly segmentation masks. WinCLIP [36], again using CLIP, slides window over image and inspects each patch to detect local defects defined by text descriptions.
architectures. Trans-MM uses the attention map to
However, foundation models have two constraints that do not exist in ACR. First, foundation models are not available for all data types. Foundation models do not exist for example for tabular data, which occurs widely in practice, for example in applications such as network security and industrial fault detection. Also, existing adaptations of foundation models for AD (e.g.. CLIP) may generalize poorly to specific domains that have not been covered in their massive training samples. For example, I Liznerski et al. [51] observed that CLIP performs poorly on non-natural images, such as MNIST digits. In contrast, ACR does not rely on powerful pre-trained foundation model, enabling zero-shot AD on various data types. Second, human involvement is required for foundation models. While previous

---

pre-trained CLiP-based zero-shot AD methods adapt to new tasks through informative prompts given by human experts, our method enriches the zero-shot AD toolbox with new adaptation strategy without human involvement. Our approach allows the anomaly detector to infer the new task/distribution based on a mini-batch of samples.
Connections to Other Areas. Our problem setup and assumptions share similarities with other research areas but differences are also pronounced. Those areas include test-time adaptation [10, 49, 53, 67, 76], unsupervised domain adapiation [38], zero-shot classification [79], meta-learning [21], and contextual AD [25]. Supp. H details the connections, similarities, and differences.
4 Experiments
We evaluate the proposed method ACR on both image (detection/seemmenttiion) and tabular data, where distribution shifts occur at test time. We compare ACR with established baselines based on deep AD, zero-shot AD, and few-shot AD methods. The experiments show that our method is suitable for different data types, applicable to diverse AD models, robust to various anomaly ratios, and significantly outperforms existing baselines. We report results on image and tabular data in Sec. 4.1 and Sec. 4.2, and ablation studies in Sec. 4.3. Results more datasets are in Supps. 1.3 to L.6.
4.1 Experiments on Images
Visual AD consists of two major tasks: (image-level) anomaly detection and (pixel-level) anomaly segmentation. The former aims to accurately detect images of abnormal objects, e.g., detecting non-dog images; the latter focuses on detecting pixel-level local defects an image, e-g.. marking board wormholes. We test our method on both tasks and compare it to existing SOTA methods.
4.1.1 Anomaly Detection
We evaluate ACR images when applied two simple backbone models: DSVDD [63] and binary classifier. Our method is trained from scratch. The evaluation demonstrates that ACR achieves superior AD results on corrupted natural images, medical images, and other non-natural images.
Datasets. We study four image datasets: CIFARI00-C [29], OrganA [81] (and MNIST [42], and Omniglot [41] in Supp. 1.4). We consider CIFAR 100-C is the noise-corrupted version of CIFAR1000 test data, thus considered as distributionally shifted data. We train using all training images from original CIFARION 100 and test all models CIFAR 100-C. OrganA is medical image dataset with 11 classes (for various body organs). We leave two successive classes out for testing and use the other classes for training. We repeat the evaluation on all combinations of two consecutive classes. Across all experiments, we apply the "one-vs-rest" setting at test time, i.e., one class is treated as normal, and all the other classes are abnormal [65]. We report the results averaged over all combinations.
Baselines, We compare our proposed method with a SOTA stationary deep anomaly detector (anomaly detection with an inductive bias (ADIB) [14]), pre-trained classifier used for batch-level zero-shot AD (ResNet152 [27]). a SOTA zero-shot AD baseline (CLIP-AD [51]), and few-shot AD baseline (one-class model-agnostic meta learning (OC-MAML) [22]). ResNet152-1 and ResNet152-11 differ in the which statistics they use in batch normalization: ResNet152-1 uses the statistics from training and ResNet152-II uses the input batch's 's statistics. See Supp. E for more details.
Implementation Details. We set x 0.8 in Eq. (6) to apply Meta Outlier Exposure. For each approach, we train a single model and test it on different anomaly ratios. Two backbone models are implemented: DSVDD [63] (ACR-DSVDD) and a binary classifier with cross entropy loss (ACR-BCE), More details are given in Supp. F.
Results. We report the results in terms of the AUROC averaged over five independent test runs with standard deviation. We apply the model to tasks with different anomaly ratios to study the robustness of ACR to the anomaly ratio at test time. Our method ACR significantly outperforms all baselines on Gaussian noise-corrupted CIFARI00-C and OrganA in Tab. 1. In Tabs. 8 and 9 in Supp. I, we systematically evaluate all methods on all 19 corrupted versions of CIFAR 100 and on non- nature images (MNIST, Omniglot). The results show that on non-natural images (OrganA, MNIST, Omniglot) ACR performs the best among compared methods, including the large pre-trained CLIP-AD bascline; on corrupted natural images CIFAR-100C), ACR achieves results competitive
7

---

Table 1: AUC (%) with standard deviation for anomaly detection on CIFAR C with Gaussian
noise [29] and medical image dataset, OrganA. ACR with both backbone models perform best.
CIFARI00-C
OrganA
1%
5%
10%
20%
1%
5%
10%
ADIB [14]
50.9+2.4
50.5
50.6a0.9
50.2+0.5
49.9
50.2+ 50.3+2.4
ResNet152-1 [27]
75.6+2.3
73.
73.2+0.8
69.9+0.6
54.2+1.1
53.9+0.5
53.2+0.6
ResNet152-11 [27]
62.5+3.1
61.81.77
61.2+0.6
60.2:0.4
54.2=1.7
53.5+0.8
52.9=0.3
OC-MAML [22]
53.0=3.6
54.1
55.8+0.6
57.1=1.0
73.7+4.7
72.2+2.6
74.2+2.4
CLIP-AD [51]
82.3+1.1
82.6=0.9
82.3+0.9
82.6+0.1
52.6.0.8
51.9#0.6
51.5+0.2
ACR-DSVDD (ouns)
87.7=1.4
8..3
85.9+0.4
8..6+0.4
1..0=1.0
77.7+0.4
76.3+0.3
ACR-BCE (ours)
84.3+2.2
86.0=0.3
86.0
85.7=0.4
81.1
79.5
7..3=
Table 2: Pixel-level and image-level AUC (%) on MVTec AD. On average, method outperforms the strongest baseline WinCLIP by 7.4% AUC in pixel-level anomaly segmentation.
MAEDAY (71)
CLIP (59)
Trans-MM [7]
MaskCLIP [86]
WinCLIP [36]
ACR (ours)
pixel-level 69.4
57.510.0
63.720.0
85. 15.00
9.2
image-level 74.5
74.010.0
91.810.0
85.810.6
with CLIP-AD and significantly outperforms other baselines. ACR is also robust on various anomaly ratios: without any (hyper)paraameter tuning, the results are consistent and don't vary over 3%. The deep AD baseline, ADIB, doesn't have adaptation ability and thus fails to perform the testing tasks, leading to random guess results. Pre-trained ResNet152 armed with batch normalization layers can adapt but with limited ability, which is in contrast with our method that directly learns to adapt. Few-shot OC-MAML suffers because it requires large support set at test time to achieve adaptation effectively. CLIP-AD has a strong performance on corrupted natural images but struggles with non-natural images, presumably because is trained on massive natural images from the internet.
4.1.2 Anomaly Segmentation
We benchmark our method ACR on the MVTec AD dataset [5] in a zero-shot setup. Experiments show that ACR achieves new state-of-the-art anomaly segmentation performance.
Datasets.
MVTec AD comprises 15 classes of images for industrial inspection. The goal is to detect the local defects accurately. To implement our method for zero-shot anomaly segmentation tasks, train on the training sets of all classes except the target one and test on the test set of the target class. For example, when segmenting wormholes on wood boards, we train a model on the other 14 classes' training data except for wood and later test on wood test set. This satisfies the zero-shot definition as the model doesn't 't any wood data during training. We apply this procedure for all classes.
Baselines. We compare our method to four zero-shot anomaly segmentation baselines: Trans-MM [7], MaskCLIP [86], MAEDAY [71], and WinCLIP [36]. The details are described in Sec. 3. We report their results listed in Jeong et al. [36]. Schwartz et al. [71].
Implementation Details. We first extract informative texture features using sliding window, which corresponds to 2D convolutions. The convolution kernel is instantiated with the ones in a pre-trained ResNet. We follow the same data pre-processing steps of Cohen and Hoshen [12]. Defard et al. [15], Rippel al. [62] to extract the features (the third layer's output in our case) of WideResNett50022 pre-trained on ImageNet. Second, we detect anomalies in the extracted features in each window position with our ACR method. Specifically, each window position corresponds to one image patch. We stack into batch the patches taken from set of images that all share the same spatial position. For example, we may stack the top-left patch of all testing wood images into a batch and use ACR to detect anomalies in that batch. Finally, the window-wise anomaly scores are bilinearly interpolated to the original image size to get the pixel-level anomaly scores. In implementing meta outlier exposure, we tried two sourees of outliers: one is corrupedd images, and the other is images of other classes. We report results of the former in the main paper and the latter in Supp. L.S. More implementation details are given in Supp. F.
Results. Similar to common practice, we report both the pixel-level and image-level results in Tab. We use the largest level-level anomaly score the image-level score. All methods are evaluated with
8

---

Table AUC (%) with standard deviation for anomaly detection on Anoshift with different anomaly contamination rations (1% 20%) and on different splitting strategies AVG and FAR [17]. ACR with either backbone model outperforms all baselines. Especially, under the distribution shift occuring in the FAR split, ACR is the only method that is significantly better than random guessing.
1%
5%
10%
20%
FAR
AVG
FAR
AVG
FAR
AVG
FAR
AVG
OC-SVM [69]
49.610.2
62.610.1
49,610.2
62.610.1
49.510.1
62.7.0.1
49.5
6,.610.1
Foorest [50]
25.8+0.4
54.6+0.2
26.1+0.1
54.7+0.1
26.0+0.1
54.6+0.1
26.0=0.1
54.7+0.1
LOF [6]
37.3+0.5
59.6+0.3
37.0=0.1
59.5+0.1
37.0+0.1
59.5+0.1
37.1+0.1
59.5+0.1
KNN [61]
45.0=0.3
70.8=0.1
45.3=0.2
70.9+0.1
45.1
70.8=0.1
45.2=0.1
70.8
DSVDD [63]
34.6+0.3
62.3+0.2
34.7+0.1
62.5+0.1
34.7+0.2
62.5+0.1
34,7=0.1
62.5+0.1
AE [1]
18.640.2
25.340.1
18.740.2
25.540.1
18.740.1
25.5
18.740.1
25.5
LUNAR [24]
24.5+0.4
38.3=0.4
24.6=0.1
38.6+0.2
24.7+0.1
38.7=0.1
24.6=0.1
38.6=0.1
ICL [72]
20.610.3
50.510.2
20.710.2
50.410.1
20.710.1
50.410.1
20.810.1
50,410,1
NTL [56]
40.710.3
57.0.0.1
40.920.2
57.110.1
41.010.1
57.140.1
41.020.1
57.120,1
BERT-AD[I7]
28.6+00.3
64.6#0.2
28.7+0.1
64.6+0.1
28.7+0.1
64.6+0.1
28.7=0.1
64.7+0.1
ACR-DSVDD (ours)
62.010.5
74.0
61.310.1
73.310.1
60.410.1
72.510.1
59.110.1
71.210.1
ACR-NTL (ours)
62.510.2
73.420.1
62.210.1
73.210.1
62.310.1
73.1
62.0-0.1
72.720.1
the AUROC metric. It shows that 1) our method is competitive to the SOTA method in image-level detection tasks, and 2) it surpasses the best baseline WinCLIP by a large margin (7.4% AUC on average) in anomaly segmentation tasks, achieving a new SOTA performance and testifying the potential of our method. We report class-wise results Supp. I.5.
4.2 Experiments on Tabular Data
Tabular data is important data format in many real-world AD applications, e.g, network intrusion detection and malware detection. Distribution shifts in such data occur naturally over time (e.g.. as new malware emerges) and grow over time. Existing zero-shot AD approaches [36, 51] are not applicable to tabular data. We evaluate ACR on tabular AD when applied to DSVDD and NTL. ACR achieves a new SOTA of zero-shot AD performance on tabular data with temporal distribution shifts.
Datasets. We evaluate all methods on two real-world tabular AD datasets Anoshift [17] and Mal- [34] where data shifts over time. Anoshift is data traffic dataset for network intrusion detection collected over ten years 22006-2015. We follow the preprocessing procedure and train/test split suggested in Dragoi et al. [17]. We train the model normal data collected from 2006 to 2010 and test on a mixture of normal and abnormal samples (with anomaly ratios varying from 1% to 20%) collected from 2011 2015. We also apply similar protocols on Malware [34], dataset for detecting malicious computer programs, and provide details in Supp. 1.6.
Baselines. We compare with statc-of-the art deep and shallow detectors for tabular AD [2, 17, 26] and study their performance under test distribution shifts. The shallow AD baselines include OC- SVM [69], IForest [50], LOF [6], and KNN [61]. The deep AD baselines include DSVDD [63], Autoencoder (AE) [1], LUNAR [24], internal contrastive learning (ICL) [72], NTL [56], and BERT- AD [17]. We adopt the implementations from PyOD [26] their official repositories.
Implementation Details. To formulate meta-training sets, we bin the data against their timestamps (year for Anoshift and month for Malware) so each bin corresponds to one training distribution Pj. The training tasks are mixed with normality ratio π = 0.8. To create more training tasks, we augment the data using attribute permutations, resulting in additional training distributions. These attribute permutations increase the variability of training tasks and encourage the model to learn permutation invariant
inntuaaaiooniivaiiant features. At test time, the attributes are not permuted. Details are in Supp. F.
Results. In Tab. 3, we report the results on Anoshift split into AVG (data from 2011 to 2015) and FAR (data from 2014 and 2015). The two splits show how the performance degrades from average (AVG) when strong distribution shifts happen after long time interval (FAR). The results of Malware with varying ratios are in Tab. 12 and Supp. I.6. We report average AUC with standard deviation over five independent test runs. The results on Anoshift and Malware show that ACR outperforms all baselines on all distribution-shiited settings. Remarkably, ACR is the only method
validate mixture of normal and abnormal samples collected from 2006 2010

---

that clearly outperforms random guessing on shifted datasets (the FAR split in Anoshift and the test split in Malware). All baselines perform worse than random on shifted test sets even though they achieve strong results when there are no distribution shifts (see results in Alvarez et al. [2]. Dragoi et al. [17], Han et al. [26]). This worse-than-randonn phenomenon also verified in the benchmark paper AnoShift [17]. The reason is that in cyber-security applications (e.g.. Anoshift and Malware), the attacks evolve adversarially. The anomalies (cyber attacks) are intentionally updated to be as similar to the normal data to spoof the firewalls. That's why static AD methods like KNN flip their predictions during test time and achieve worse than random performance. In terms of robustness, although ACR- RRR-SDDD''S performance degrades little (within 3%) when the anomaly ratio increases, ACR-NTL is fairly robust to high anomaly ratios. The degradation is attributed to the fact that the majority of normal samples get blurred as the anomaly ratio increase, leading to noisy batch statistics.
4.3 Ablation Studies
We perform several ablation studies in Supp. I.1, including 1) demonstrating the benefit of the Meta Outlier Exposure loss, 2) studying the effect of batch normalization, and 3) analyzing the effects of the batch sizes and the number of meta- veta-training classes. To show that Meta Outlier Exposure is favorable option, we compare it against the one-class classification loss and a fine-tuned version of R2 on domain-specific training data. Tab. shows that our approach outperforms the two alternatives on two image datasets. To analyze the effect of batch normalization, we adjust batch normalization usage during training and testing listed in Tab. 5. More details and the studies on the batch size, the number of meta-training classes, other normalization techniques (LayerNorm, InstanceNorm, and GroupNorm), effects of batch norm position, and robustness of the mixing hyperparameter be found in Supp. I.1.
5 Conclusion
We studied the problem of adapting a learned AD method to a new data distribution, where the concept of "normality" changed. Our method is zero-shot approach and requires no training or fine-tuning to a new data set. We developed a new meta-training approach, where we trained an off off-the-shelf deep AD method on (meta-) set of interrelated datasets, adopting batch normalization in every layer, and used samples from the meta set as either normal samples and anomalies, depending on the context. We showed that the approach robustly generalized to new, unseen anomalies.
Our experiments on image and tabular data demonstrated superior zero-shot adaptation performance when no foundation model was available. We stress that this is an important result since many, not most AD applications in the real world rely on specialized datasets: medical images, data from industrial assembly lines, malware data, network intrusion data etc. Existing foundation models often do not capture these data, as we showed. Ultimately, our analysis shows that relatively small modifications to model training (meta-learning, batch normalization, and providing artificial anomalies from the meta-set) will enable the deployment of existing models in zero-shot AD tasks.
I Limitations & Societal Impacts Our method depends on the three assumptions listed in Sec. 2. If those assumptions are broken, zero-shot adaptation cannot be assured.
Anomaly detectors are trained to detect enypical/under-represeett data in a data set. Therefore, deploying an anomaly detector, e.g., in video surveillance, may ultimately discriminate against under-represented groups. Anomaly detection methods should therefore be critically reviewed when deployed on human data.
Acknowledgements
SM acknowledges support by the National Science Foundation (NSF) under an NSF CAREER Award, award numbers 2003237 and 2007719, by the Department of Energy under grant D by the IARPA WRIVA program, and by gifts from Qualcomm and Disney. Part of this work was conducted within the DFG research unit FOR 5359 on Deep Learning on Sparse Chemical Process Data. PS was supported by the US National Science Foundation under awards 1900644 and 1927245 and by the National Institute of Health under awards R01-AG005330-0021 and ROI R01-LM013344, SM and PS were supported by the Hasso Plattner Institute (HPI) Research Center in Machine Learning
10

---

and Data Science at the University of California, Irvine. MK acknowledges support by the Carl-Zeiss Foundation, the DFG awards KL 2698/2-1, KL 2698/5-1, KL 2698/6-1, and KL 2698/7-1, and the BMBF awards 03/B0770F and 011221010C. We thank Eliot Wong- Toi for helpful feedback on the manuscript.
The Bosch Group is carbon neutral. Administration, manufacturing and research activities do no longer leave carbon footprint. This also includes GPU clusters on which the experiments have been performed.
References
[1] Charu C Aggarwal. An introduction to outlier analysis. In Outlier analysis, pages 1-34. Springer, 2017.
[2] Maxime Alvarez, Jean-Charles Verdier, D'Jeff K Nkashama, Mare Frappier, Martin Tardif, and Froduald Kabanza. revealing large-scale evaluation of unsupervised anomaly detection algorithms. arXiv preprint (rXiv:2204.098255 2022.
[3] Jonathan Baxter. A model of inductive bias learning. Journal of artificial intelligence research, 12:149-198 198, 2000.
[4] Liron Bergman and Yedid Hoshen. Classification-based anomaly detection for general data. In International Conference on Learning Representations, 2020.
[5] Paul Bergmann, Michael Fauser, David Sattlegger, and Carsten Steger. Mvtec ad-a comprehen- sive real-world dataset for unsupervised anomaly detection. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 9592-9600, 2019.
[6] Markus M Breunig, Hans-Peter Kriegel, Raymond T Ng. and Jörg Sander. Lof: identifying density-based local outliers. In Proceedings of the 2000 ACM SIGMOD international conference on Management of data, pages 93-104, 2000.
[7] Hila Chefer, Shir Gur, and Lior Wolf. Generic attention-model explainability for interpreting bi-modal and encoder-decoder transformers. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 397-406, 2021.
[8] Bingqing Chen, Luca Bondi, and Samarjit Das. Learning to adapt to domain shifts with few- shot samples in anomalous sound detection. In 2022 26th International Conference Pattern Recognition (ICPR), pages 133-139. IEEE, 2022.
[9] Xiaoran Chen and Ender Konukoglu. Unsupervised detection of lesions in brain mri using constrained adversarial auto-encoders. In MIDL Conference book. MIDL, 2018.
[10] Sungha Choi, Seunghan Yang. Seokeon Choi, and Sungrack Yun. Improving test-time adaptation via agnostic weight regularization and nearest source prototypes. In Computer Vision- ECCV 2022: 17th European Conference, Tel Aviv, Israel. October 23-27, 2022, Proceedings, Part XXXIII, pages 440-458. Springer, 2022.
[11] Gregory Cohen, Saeed Afshar, Jonathan Tapson, and Andre Van Schaik. Emnist: Extending mnist to handwritten letters. In 2017 international joint conference on neural networks (IJCNN), pages
IEEE, 2017.
[12] Niv Cohen and Yedid Hoshen. Sub-image anomaly detection with deep pyramid correspon- dences. arXiv preprint arXiv:2005.02357. 2020.
[13] Lucas Deecke, Robert Vandermeulen, Lukas Ruff, Stephan Mandt, and Marius Kloft. Image anomaly detection with generative adversarial networks. In Joint european conference machine learning and knowledge discovery in databases, pages 3-17. Springer, 2018.
[14] Lucas Deecke, Lukas Ruff, Robert A Vandermeulen, and Hakan Bilen. Transfer-based semantic anomaly detection. International Conference on Machine Learning, pages 2546-2558. PMLR, 2021.
11

---

[15] Thomas Defard, Aleksandr Setkov, Angelique Loesch, and Romaric Audigier. Padim: patch distribution modeling framework for anomaly detection and localization. In Pattern Recognition. ICPR International Workshops and Challenges: Virtual Event, January 10-15, 2021, Proceedings, Part IV, pages 475 Springer, 2021.
[16] Kaize Ding, Qinghai Zhou, Hanghang Tong, and Huan Liu, Few-shot network anomaly detection via cross-network meta-learning. Proceedings of the Web Conference 2021, 2021.
[17] Marius Dragoi, Elena Burceanu, Emanuela Haller, Andrei Manolache, and Florin Brad. Anoshift: distribution shift benchmark for unsupervised anomaly detection. In Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2022.
[18) Sepideh Esmaeilpour, Bing Liu, Eric Robertson, and Lei Shu. Zero-shot out- aut-of-distribution detection based on the pretrained model clip. In Proceedings of the AAAI conference on artificial intelligence, 2022.
[19] Alireza Fallah, Aryan Mokhtari, and Asuman Ozdaglar. Generalization of model-agnostic meta- learning algorithms: Recurring and unseen tasks. Advances in Neural Information Processing Systems, 34:5469-5480, 2021.
[20] Tongtong Feng, Q. Qi, Jingyu Wang. and Jianxin Liao. Few-shot class-adaptive anomaly detec- tion with madel-agnostic meta-learning 2021 IFIP Networking Conference (IFIP Networking), pages 1-9, 2021.
[21] Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adap- tation of deep networks. In International conference muchine learning, pages 1126-1135. PMLR, 2017.
[22] Ahmed Frikha, Denis Krompaß, Hans- Geos-Geerg Köpken, and Volker Tresp. Few-shot one-class classification via meta-learning, Proceedings of the AAAI Conference Artificial Intelligence, 35(8):7448-744 7456, May 2021.
[23] Izhak Golan and Ran Yanya.iv. Deep anomaly detection using geometric transformations. In Advances in Neural Information Processing Systems, pages 9758-9769, 2018.
[24] Adam Goodge, Bryan Hooi, See-Kiong Ng, and Wee Siong Ng. Lunar: Unifying local outlier detection methods via graph neural networks. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 6737-6745, 2022.
[25] Manish Gupta, Jing Gao, Charu C Aggarwal, and Jiawei Han. Outlier detection for temporal data: A survey. IEEE Transactions on Knowledge and data Engineering, 26(9):225--2277, 2013.
[26) Songqiao Han, Xiyang Hu, Hailiang Huang, Minqi Jiang, and Yue Zhao. Adbench: Anomaly detection benchmark. In Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2022.
[27] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[28] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16000-16009, 2022.
[29] Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common corruptions and perturbations. In International Conference on Learning Representations, 2019.
[30] Dan Hendrycks, Mantas Mazeika, and Thomas Dietterich. Deep anomaly detection with outlier exposure. In International Conference on Learning Representations, 2018 8.
[31] Dan Hendrycks, Mantas Mazeika, Saurav Kadavath, and Dawn Song. Using self-supervised learning can improve model robustness and uncertainty. Advances in Neural Information Processing Systems, 32: 32:15663-15674, 2019.
12

---

[32] Chaoqin Huang, Haoyan Guan, Aofan Jiang, Ya Zhang, Michael Spratling, and Yan-Feng 2022: 17th
Wang. Registration based few-shot anomaly detection. In Computer European Conference, Tel Aviv Israel, October 23-27, 2022, Proceedings, Part XXIV, pages 303-319. Springer, 2022.
[33] Gao Huang. Zhuang Liu,, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4700-4708, 2017.
[34] Ngoe Anh Huynh, Wee Keong Ng, and Kanishka Ariyapala. new adaptive learning algorithm and its application to online malware detection. In International Conference Discovery Science, pages 18-32. Springer, 2017.
[35] Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International conference on machine learning, pages 448-456. pmlr, 2015.
[36] Jongheon Jeong, Yang Zou, Taewan Kim, Dongqing Zhang, Avinash Ravichandran, and Onkar Dabeer. Winclip: Zero-/few-shot anomaly classification and segmentation. 2023.
[37] Chao Jia, Yinfci Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoe Le, Yun- Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision language representation learning with noisy text supervision. In International Conference on Machine Learning, pages 4904 4916. PMLR, 2021.
[38] Wouter M Kouw and Marco Loog. A review of domain adaptation without target labels. IEEE transactions on pattern analysis and machine intelligence, 43(3):766-778, 2019.
[39] Jedrzej Kozerawski and Matthew A. Turk. Clear: Cumulative learning for one-shot one-class image recognition, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3446-3455, 2018.
[40] Anna Kruspe. One-way prototypical networks. ArXiv, abs//990.000200 2019.
[41] Brenden M Lake, Ruslan Salakhutdinov, and Joshua B Tenenbaum. Human-level concept learning through probabilistic program induction. Science, 150062666:13332113333 2015.
[42] Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):2278-23)) 1998.
[43] Aodong Li, Alex Boyd, Padhraic Smyth, and Stephan Mandt. Detecting and adapting to irregular distribution shifts in bayesian online learning. Advances in neural information processing systems, 34:68 0--8888222 2021.
[44] Aodong Li, Chen Qiu, Padhraic Smyth, Marius Kloft, Stephan Mandt, and Maja Rudolph. Deep anomaly detection under labeling budget constraints. arXiv preprint arXiv:2332.078.3,, 2023.
[45] Chun- Li, Kihyuk Sohn, Jinsung Yoon, and Tomas Pfister. Cutpaste: Self-supervised learning for anomaly detection and localization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9664 9674, 2021.
[46] Da Li, Yongxin Yang, Yi-Zhe Song, and Timothy Hospedales. Learning to generalize: Meta- learning for domain generalization. In Proceedings of the AAAI conference on artificial intelligence, volume 32, 2018.
[47] Zheng Li, Yue Zhao, Nicola Botta, Cezar lonescu, and Xiyang Hu. Copod: copula-based based outlier detection. In 2020 IEEE international conference data mining (ICDM), pages 11 1118-11 118-112.. IEFE, 2020.
[48] Zheng Li, Yue Zhao, Xiyang Hu, Nicola Botta, Cezar lonescu, and George Chen. Ecod: Unsu- pervised outlier detection using empirical cumulative distribution functions. IEEE Transactions on Knowledge and Data Engineering. 2022.
13

---

[49] Hyesu Lim, Byeonggeun Kim, Jaegul Choo, and Sungha Choi. Tin: A domain- shift aware batch normalization test-time adaptation. In The Eleventh International Conference on Learning Representations, 2023.
[50] Fei Tony Liu, Kai Ming Ting, and Zhi-Hua Zhou. inolation-based anomaly detection. ACM Trunsuctions on Knowledge Discovery from Data (TKDD), 6(1):1-39, 2012.
[51] Philipp Liznerski, Lukas Ruff, Robert Vandermeulen, Billy Joe Franks, Klaus Robert Muller, and Marius Kloft. Exposing outlier exposure: What can be learned from few, one, and zero outlier images. Transactions on Machine Learning Research, 2022.
[52] Yiwei Lu, Frank Yu, Mahesh Kumar Krishna Reddy, and Yang Wang. Few-shot scene-adaptive anomaly detection. In ECCV, 2020.
[53] Zachary Nado, Shreyas Padhy, D Sculley, Alexander D' D'mour, Balaji Lakshminarayanan, and Jasper Snoek. Evaluating prediction-tini batch normalization for robustness under covariate shift. arXiv preprint arXiv:2006. 10963, 2020.
[54] Alex Nichol, Joshua Achiam, and John Schulman. On first-order meta-learning algorithms. arXiv preprint arXiv: 100..800399999 2018.
[55] Emanuele Principi, Fabio Vesperini. Stefano Squartini, and Francesco Piazza. Acoustic novelty detection with adversarial autocncoders. In 2017 International Joint Conference Neural Networks (IJCNN), pages 3324-3330. IEEE, 2017.
[56] Chen Qiu, Timo Pfrommer, Marius Kloft, Stephan Mandt, and Maja Rudolph. Neural transfor- mation learning for deep anomaly detection beyond images. In International Conference Machine Learning, pages 8703-8714. 84. PMLR, 2021.
[57] Chen Qiu, Marius Kloft, Stephan Mandt, and Maja Rudolph. Raising the bar in graph-level anomaly detection. In Proceedings of the Thirty-First International Joint Conference Artificial Intelligence IJCAI-22, pages 2196-2203, 2022.
[58] Chen Qiu, Aodong Li, Marius Kloft, Maja Rudolph, and Stephan Mandt. Latent out- lier exposure for anomaly detection with contaminated data. In Kamalika Chaudhuri, Ste- fanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato, editors, Proceed- ings of the 39th International Conference Machine Learning, volume 162 of Proceed- ings of Machine Learning Research, pages 18153-18167. PMLR, 17-23 Jul 2022. URL https:// https/////oeeeengs....m mlr press/v162/qiu2283 html.
[59] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning. pages 8748-8766 PMLR, 2021.
[60] Ilija Radosavovic, Raj Prateek Kosaraju, Ross Girshick, Kaiming He, and Piotr Dollár. Design- ing network design spaces. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10436, 2020.
[61] Sridhar Ramaswamy, Rajeev Rastogi, and Kyuseok Shim. Efficient algorithms for mining outliers from large data sets. In Proceedings of the 2000 ACM SIGMOD international conference on Management of data, pages 427-438, 2000.
[62] Oliver Rippel, Patrick Mertens, and Dorit Merhof. Modeling the distribution of normal data in pre-trained deep features for anomaly detection. 2020 25th International Conference on Pattern Recognition (ICPR), pages 8726-6733 IEEE, 2021.
[63] Lukas Ruff, Robert Vandermeulen, Nico Goernitz, Lucas Deecke, Shoaib Ahmed Siddiqui, Alexander Binder, Emmanuel Müller, and Marius Kloft. Deep one-class classification. In International conference on machine learning, pages 1393-4402. PMLR, 2018.
[64] Lukas Ruff, Robert A Vandermeulen, Nico Görnitz, Alexander Binder, Emmanuel Müller, Klaus- Robert Müller, and Marius Kloft. Deep semi supervieee anomaly detection. In International Conference Learning Representations, 2019.
14

---

[65] Lukas Ruff, Jacob R Kauffmann, Robert A Vandermeulen, Grégoire Montavon, Wojeiech Samek, Marius Kloft, Thomas G Dietterich, and Klaus-Robert Müller. A unifying review of deep and shallow anomaly detection. Proceedings of the IEEE, 2021.
[66] Thomas Schlegl, Philipp Seeböck, Sebastian M Waldstein, Ursula Schmidt-Erfurth, and Georg Langs. Unsupervised anomaly detection with generative adversarial networks guide marker discovery. In International conference on information processing in medical imaging, pages 146-157. Springer, 2017.
[67] Steffen Schneider, Evgenia Rusak, Luisa Eck, Oliver Bringmann, Wieland Brendel, and Matthias Bethge. Improving robustness against common corruptions by covariate shift adaptation. Advances in Neural Information Processing Systems, 33:1 1539-11551, 2020.
[68] Tim Schneider, Chen Qiu, Marius Kloft, Decky Aspandi Latif, Steffen Staab, Stephan Mandt, and Maja Rudolph. Detecting anomalies within time series using local neural transformations. arXiv preprint 220.::2202.0394444 2022.
[69] Bernhard Schölkopf, Robert C Williamson, Alex Smola, John Shawe-Taylor, and John Platt. Support vector method for novelty detection. Advances neural information processing systems, 12, 1999.
[70] Bernhard Schölkopf, John C Platt, John ShawerTaylor, Alex J Smola, and Robert C Williamson. Estimating the support of -igeeniiensi distribution, Neural computation, 13(7):1443- 1471, 2001.
[71] Eli Schwartz, Assaf Arbelle, Leonid Karlinsky, Sivan Harary, Florian Scheidegger, Sivan Doveh, and Raja Giryes. Maeday: Mae for few and zero shot anomaly-detection. arXiv preprint arXiv:2211. 14307, 2022.
[72] Tom Shenkar and Lior Wolf. Anomaly detection for tabular data with internal contrastive learning. In International Conference Learning Representations. 2021.
[73] Shelly Sheynin, Sagie Benaim, and Lior Wolf. A hierarchical transformation transianaionddicciinaaangg generative model for few shot anomaly detection. In Proceedings of the IEEE/CVE International Conference Computer Vision (ICCV), pages 8495-8504, October 2021.
[74] Yaniv Shulman. Unsupervised contextual anomaly detection using joint deep variational generative models. arXiv preprint arXiv:1904.000
2019.
[75] Kihyuk Sohn, Chun-Liang Li, Jinsung Yoon, Minho Jin, and Tomas Pfister. Learning and evaluating representations for deep one-class classification. In International Conference on Learning Representations, 2020.
[76] Dequan Wang, Evan Shelhamer, Shaoteng Liu, Bruno Olshausen, and Trevor Darrell. Tent: Fully test-time adaptation by entropy minimization. In International Conference on Learning Representations, 2021.
[77] Ze Wang, Yipin Zhou, Rui Wang, Tsung- Yu Lin, Ashish Shah, and Ser-Nam Lim. Few-shot fast-adaptive anomaly detection. In Advances in Neural Information Processing Systems, 2022.
[78] Jhih- -iang Wu, Ding-Jie Chen, Chiou-Shami Fuh, and Tyng- Lyng-Luhhln Liu. Learning unsupervised metaformer for anomaly detection. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 4369-4378, October 2021.
[79] Yongqin Xian, Christoph H Lampert, Bernt Schiele, and Zeynep Akata. Zero-shot learning comprehensive evaluation of the good, the bad and the ugly. IEEE transactions on pattern analysis and machine intelligence, 41(9):251-2266 2018.
[80] Zehao Xiao, Xiantong Zhen, Shengcai Liao, and Cees GM Snock. Energy-based test sample adaptation for domain generalization. In The Eleventh International Conference on Learning Representations, 2023.
[81] Jiancheng Yang, Rui Shi, and Bingbing Ni. Medmnist classification decathlon: A lightweight automl benchmark for medical image analysis. In 202/ IEFE 18th International Symposium on Biomedical Imaging (ISBI), pages 191-195. IEEE, 2021.
15

---

[82] Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Yeung, Mojtaba Seyedhosseini, and Yonghui Wu. Coca: Contrastive captioners are image-text foundation models. Transactions on Machine Learning Research, 2022.
[83] Lu Yuan, Dongdong Chen, Yi-Ling Chen, Noel Codella, Xiyang Dai, Jianfeng Gao, Houdong Hu, Xuedong Huang, Boxin Li, Chumyuan Li, et al. Florence: A new foundation model for computer vision. arXiv preprint arXiv:2111.11432, 2021.
[84] Shen Zhang, Fei Ye, Bingnan Wang, and Thomas G. Habetler. Few-shot bearing anomaly detection via model-agnostic meta-learning. 2020 23rd International Conference on Electrical Machines and Systems (ICEMS), pages 1341-1346, 2020.
[85] Chong Zhou and Randy Paffenroth. Anomaly detection with robust deep autoencoders. In Proceedings of the 23rd ACM SIGKDD international conference on knowledge discovery and data mining, pages 665-674, 2017.
[86] Chong Zhou, Chen Change Loy, and Dai. Denseclip: Extract free dense labels from clip. arXiv preprint aTXi::2112.01077, 2021.
16

---

A Justifications of Assumptions A1-A3
As follows, we provide justifications for assumptions AI-A3, Following the justification, we also discuss possibilities to remove or mitigate the assumptions.
A1 Assuming an available meta-training set is widely adopted in few-shot learning or meta-learning [21, 22, 32, 54] and domain generalization [46, 80]. In practice, the meta- meeaatraining set can be generated using available covariates. For example, for our tabular data experiment, we used the timestamps; in medical data, one could use data collected from different hospitals or different patients to obtain separate sets for meta-training: and in MVTec-AD, we used the other training classes except for the target class to form the training set. We also provided an ablation study on the number of classes in the meta-training set (Tab. 7). We found even in the extreme case where we only have one data class in the training set, the trained model still provides meaningful results.
There are multiple ways to mitigate this assumption. If one does not have a meta- veeta-taiinng set at hand, one can train their model on different but related dataset, e.g., train on Omniglot but test on MNIST (see results below under this setting. We still get decent AUC results on MNIST).
Anomaly ratio
1%
5%
10%
20%
AUROC
84.4+2.4
85.2+2.5
84.3=2.5
82.2+2.4
A2 Batch-level prediction is a common assumption used in robustness literature [10, 49, 53, 67, 76]. In addition, batch-level predictions are widely used in real life. For example, people examine Covid19 test samples at a batch level out of economic and time-efficiency considerations", To relax this assumption, our method can easily be extended score individual data by presetting the sample mean and variance in BatchNorm layers with a collection of data. These moments are then fixed when predicting new individual data. Empirically, understand the impacts of the batch size on the prediction performance, we conducted an ablation study with as small batch size as three in the experiments.
A3 Besides being supported by the intuition that anomalies are rare, this is consistent with most of the data used in the literature. ADBench has 57 anomaly detection datasets (with an average anomaly ratio of 5%), all matching our assumption that the normal data take the majority in each dataset.
We provide a simple mathematical argument for the validity of A3, showing that a mini-batch with majority of anomalies is very unlikely be drawn for sufficiently large mini-batch size B. Let < 1/2 denote the fraction of anomalies among the data and define A 1/2 p > 0. For every data point x; the batch, let y ~ Bernoullip)) encode whether x; normal (y:: = 0) or abnormal (y: = 1). The variable SB == f/1 + ... + YB thus counts the number of anomalies in the batch, so that SB < B/2 means that the majority of the data normal. We want to show that the violation of A3 is unlikely, that is, P(SB B/2) is small. By Hoeffding's 's inequality, since 0 S yı 1 for all i, it follows that PSS 2 B/2)= P(S8-E[ E[SB] s)] B/2 Bp) & cxx ( 2B(0.5 5 -)) exp ( ( 2BA3). which converges to zero exponentially fast when B 00.
B Generalization to an Unseen Distribution P.
This section aims to provide proof for Thm. 2.1. Inspired by Fallah et al. [19], we derive upper bound of the generalization eтror of our meta- veea-training approach on unseen distributions. The error is described in terms of the data distributions transformed by batch-norm-involwee feature extractors.
Definition B.1. Given sample space Ωi and its o-field the total variation distance between two probability measure P; and P, defined on F
P. - P,llTV sup PA() sup E. (e=))
(8)
AEF
f:0%/51
covid- 19- and-nedicall-eviin pooled- seadd-aamll-teettnneeeeee -tt---tteeeiii-------eeiinnnnnni tengg--tttttnggcce covid--
ttteps://githble wgtthhb.cco/Miingg824///eeeee
17

---

Now we split the neural network into two parts: the first part is the layers before (including) the last batch normalization layer, referred as feature extractor = fx),,, and the second part is the layers after the last batch normalization layer, namely the anomaly score map S(z) = (x)) (x). (z)) may involve learnable parameters, we omit the notations for conciseness. The split allows us to separate the effects of batch normalization layers on the generalization error of unseen distributions.
Under the transformation of fe consisting of batch normalization layers, we have the data distribution transformed into the distribution of adaptatively centered representations
(x( (x(x Pz(
(9)
resulting in Pj with Epz[z]
[z]] 0 and Varp;[z]=11
Assume the mini-batch is large enough so that the mini-batch means and variances approximately constant across batches, i.e., the batch statistics batch normalization layers are equal to the population-truth values. Consequently, when x1,.-.,XB Lid P3 which constitutes Xg, their latent representations z; (xB) are
lid. Pj. for i = 1,..,. B. Then the expectation of the batch-level losses
Exp=PP
Σ [[5) B 1-1
E Pa1
Σ L S( (2))] i-1
B
1 E Pp,, [41)(2))]
Σ
B
Py [4[S(2)]]
(10)
Assumption B.2. For any parameters (if any), the loss function L[S(-)] is bounded by C.
We now quantify the generalization error to an unseen distribution P, by the difference between the expected loss of data batches of P. and the one of meta- cett-training distribution.
1 8
K
B
E Exg- P.
[x
E XB-PP P3
LSS (xs)]
(11)
B 1
B
1
En-P:
[[[S(z)]]
Σ
E py [[[S(z]]]
(by Eq. (10))
(12)
j-l
[[S(z)]
1
[[S(z)]
C
Eg- P.
Σ
E P
(by Assumption B.2)
(13)
C
K
C
1=1
[[S())] C
K
[[S()))
SC sup
Ezz-P
E
Eyy P3
(14)
U<L/C'1
K Σ
CC
P.
P
(by Definition B.1)
(15)
K
j-1
TV
This result suggests the generalization error the loss function is bounded by the total variation distance between P and 1 E Pj. The batch normalization re-calibrates all P, such that Pj centers the origin and has unit variance, making the distributions similar. Thus the total variation gets smaller after batch normalization, lowering the generalization error upper bound.
The limitation of this analysis is we assume the batch statistics are population- truthh moments (mean
and variance) in Z; iid. P.. So we cannot analyze the effects of the batch size during training and testing. That said, we provide empirical evaluations on different batch sizes B at test time in Supp. I.I.
C Algorithm
The training procedure of our approach is simple and similar to any stochastic gradient-based optimization. The only modification is to take into account the existence of meta-training set. See Alg. 1.
18

---

Algorithm 1:
Training procedure of ACR
Input
K interrelated training distributions Pi,.., Pk Lid. Mixting rate π
Q
Deep anomaly detector model parameters Sub-sample size M
Mini-batch size [B]
learning rate
Number of training iterations '''
Output: Optimized anomaly detector with parameter 07
Randomly initialize
2 Construct PT. ..... ....Pk (Eq. (6)) for iteration t in [1. do
Sample M tasks {xB... aM
m-1 from all K task distributions (Pf,"., PR) Em --Vo
0, 0--1 aVe
L(Som., (xB,)) )) (Eq. (7))
end
AIRor: 4L
0-2 P7
Figure 2: Illustration of batch normalization for AD with two tasks PT and Pz. The method (eatch-)normalizer the data in PTT separately. each PT consists mainly of normal samples, most samples will be shifted close to the origin (by subtracting the respective task's 's mean). As a result, the samples from all tasks concentrate around the origin in joint feature space (gray area) and thus can be tightly enclosed using, e.g., one-class classification. Samples from the test task are batch normalized in the same way.
D Toy Example with Batch Normalization
An important component of our method is batch normalization, which shifts and re-scales any data batch Xg have sample mean zero and variance one. Batch normalization also provides a basic paramcter-frve zero-shot batch-level anomaly detector (Eq. (4)). In Fig. 2, we show ID case of detecting anomalies mixture distribution. The mixture distribution composes of normal data distribution (the major component) and an abnormal data distribution (the minor component). Eq. (4) adaptively detects anomalies a batch level by shifting the normal data distribution toward the origin and pushing anomalies away. Setting user-specified threshold allows making predictions.
E Baselines
CLIP-AD [51). CLIP (Contrastive Language-Image Pre-training (59]) a pre-trained visual repre- sentation learning model that builds open-source images and their natural language supervision signal. The resulting network projects visual images and language descriptions into the same fea- ture space. The pre-trained model can provide meaningful representations for downstream tasks such as image classification and anomaly detection. When applying CLIP on zero-shot anomaly detection, CLIP prepares pair of natural language descriptions for normal and abnormal data: la "A photo of something"). The anomaly score of
{In "A photo test image is the relative distance between L, and to 1, in the feature space,
exp((f.(x). ((.)))
s(x) Eeelhnd exp(( f((e)))
19

---

where fo and fi are the CLIP image and description feature extractors and (:--) is the inner product. We name this baseline CLIP-AD.
Compared to our proposed method, CLIP-AD requires a meaningful language description for the image. However, this is not always feasible for all image datasets like Omniglot [41], where people can't t name the written characters easily, In addition, CLIP-AD doesn't 't apply to other data types like tabular data or time-series data. Finally, CLIP-AD has limited ability adapt to a different data distribution other than its training one. These limitations are demonstrated in our experiments.
OC-MAML [22]. One-Class Model Agnostic Meta Learning (OC-MAML) is meta-learning algorithm that taylors MAML [21] toward few-shot anomaly detection setup. OC-MAML learns a global model parameterization & that can quickly adapt to unseen tasks with few new task data points S, called a support set. The new-task adaptation takes the global model parameters to a task-specific parameterization St) that has low loss L(Qr; (0,,S)) on the new task L, represented by another dataset Qt, called a query set. OC-MAML uses a one-class support set to update the model parameters 0 with few gradient steps get Φb To learn an easy-to-adapt global parameterization 0, OC-MAMI. directly minimizes the target loss on lots of training tasks. Suppose there are T tasks for training. The following loss function minimized
1
T E s. Qepp (T(Qr:(0,55)) 1
I(0)
(16)
where p'g is task t's support distribution and P/2 is the query set distribution. During training, the support set contains K normal data points where K is usually small, termed K-shot OC-MAML. The query set contains an equal number of normal and abnormal data and provides optimization gradients for 0. During test time, OC-MAML adapts the global parameter 9 on the unseen task's ' support set S*, resulting in a task- sask-specific parameter @(0, S*)) The newly adapted parameters are then used for downstream tasks.
OC-MAML not zero-shot anomaly detector and requires K support data points to adapt compared to our method. Our method is simpler in training as it doesn't 't need to adapt to the support set with additional gradient updates, characterized in the function (8, OC-MAML is also different in batch normalization. Rather than the original batch normalization, OC-MAML first computes the batch moments using the support set and then normalizes both the support and query set with the same moments. However, the computed moments can be noisy when the support set size is small. In our experiments, we adopt a 1-shot OC-MAML for all image data.
ResNet152 [27]. Because batch normalization is an effective tool for zero-shot anomaly detection (see Fig. 1b), we directly apply batch normalization on extracted features from pre-trained model. We then compute the anomaly score as the Euclidean distance between feature vector and the origin in the feature space. Our experiments use a ResNet152 model pre-trained on ImageNet as a feature extractor and extract its 4048-dimension penultimate layer output as the final feature vector. Upon computing the features of an input batch through batch normalization layers in ResNet, two variants are available: using the batch statistics from training re-computing the statistics of the test input batch itself. We name the former variant ResNet152-1 and the latter ResNet152-II. Baseline ResNet152 152 doesn't optimize the feature extractor jointly with the zero-shot detection property of batch normalization. Hence the extracted pre-trained features are not optimal for the zero-shot AD.
ADIB [14]. In addition to zero-shot and few-shot anomaly detectors, we also compare with the oate-of-the-art deep anomaly detector ADIB [14] which use pre-trained image features and additional data for outlier exposure in training. We use "debiased" subset of Tiny
as the outlier
exposure data for CIFAR 100 as suggested in Hendrycks et al. [30], use EMNIST [11] as the outlier exposure data for MNIST as suggested in Liznerski et al. [51], use OrganC and OrganS datasets [81] as outlier exposure data for OrganA, and use half of the training data as normal data and half of the training data as auxiliary outliers for Omniglot.
20

---

F Implementation Details
Practical Training and Testing. On visual anomaly classification and tabular AD, we construct training and test distributions using labeled datasets", where all x from the same class j (e.g.. all O's in MNIST) are considered samples from the same P.. The dataset Q (e.g.. MNIST as whole) is the meta-set of all these distributions.
For training and testing, we split the meta meta-dataset into disjoint subsets. In the MNIST example, we define Po..... P1 as the distributions of images with digits 0 and use them for training. For testing, we select a single distribution of digits not seen during training (e.g., digit 5) as the "new normal" distribution P. to which we adapt the model. The remaining digits (6 9 in this example) are used as test-time anomalies. To reduce variance, we rotate the roles among digits 5 9, using each digit as test distribution once."
F1 Implementation Details on Image Data for Anomaly Detection
Hyperparameter Search. We search the hyperparameters validation set split from the training set, after which we integrate the validation set into the training set and train the model on that. Then test the model the test set.
On CIFAR100-C, we construct the validation set on the training set of the primitive CIFAR100. We randomly select 20 classes as the validation set and set the remaining classes to be the training dataset at validation time, We search the neural network architecture (layers (3,4,5,6), number of convolutional kernels (32, 64, 128), and the output dimensions (8, 16, 32, 64, 128) while fixing the kernel size by 3x3. For the learning rate, we search values 0.1, 0.01, 0.001, 0.0001, and 0.00001, after which we search finer values in a binary search fashion. We also search the mini-batch size B (30, 60) and the number of sampled tasks A (16, 32, 64) at each iteration. We select the combination that trade-off the convergence speed and optimization stability. When selecting the anomaly ratio (Eq. (6)), we test 0.99, 0.95, 0.9, 0.8, 0.6 and find the results are quite robust to these values. So we fix 0.8 across the experiments.
On non- non-natural image datasets (OrganA, Omniglot, and MNIST), we search hyperparameters on the validation set of Omniglot and use the searched hyperparameters on all datasets. Specifically, at validation time, we randomly split the Omniglot into 1200 classes for training and 423 classes for validation. After optimizing the hyperparameters, and the remaining 423 classes for testing. The searched hyperparameters are the same as the ones in CIFAR LIFAR10-CC described above.
constantly use the first 1200 classes for training
Training Protocols. We train the model 6,000 iterations on CIFARI00 data, 10,000 iterations on Omiglot, and 2,000 iterations on MNIST and OrganA. Each iteration contains 32 training tasks; each task mini-batch has 30 (for datasets other than CIFAR100) or 60 (for CIFAR 100) points sampled All 32 training tasks' gradients are averaged and incur one gradient update per iteration.
from po.s.
ACR-DSVDD. We use the standard convolutional neural network architecture used in meta-learning. Specifically, the network contains four convolution layers. Each convolution layer is followed by a batch normalization layer and a ReLU activation layer. The final layer is fully-connectly layer followed by batch normalization layer. The center of DSVDD has the same dimension as the output of the folle-commected layer. which is 32. For CIFARI LOO/CIAAR 100-C, each convolution layer has 128 kernels. For MNIST, Omniglot, and OrganA, each convolution layer has 64 kernels. Each kernel's 's size is 3x3. We use Adam with a learning rate of 0.003 on CIFAR 100 dataset and le - on all the other datasets.
We use the same network structure as ACR-DSVDD without the final batch normaliza- tion layer and the center. The final fully-connected layer has output dimension of 1. We train the model with binary cross entropy loss. We use Adam with a learning rate of 0.003 on CIFAR100 dataset and le 4 on all the other datasets.
ACR-BCE.
$ 'these are either classification datasets or datasets where one of the covariates is binned to provide classes. This is the popular "one-vs-rest" testing set-up. which is standard in AD benchmarking. (e.g.. [65])
21

---

F22 Implementation Details on MVTec AD for Anomaly Segmentation
Training Protocols. Since the images are roughly aligned, we can use sliding window over the image to detect local defects in each window. However, this requires pixel-level alignment and is unrealistic for the MVTee AD dataset. We instead first extract informative texture features using sliding window, which corresponds to the 2D convolutions. The convolution kernel is instantiated with the ones in pre-trained ResNet. We follow the same data pre- pre-preeessing steps of Cohen and Hoshen [12], Defard et al. [15], Rippel et al. [62] to extract the texture representations (the third layer's output in our case) of WideResNet-50-22 pre-trained on ImageNet. Second, we detect anomalies in the extracted features in each sliding window position with our ACR method. Specifically, each window position corresponds one image patch. We stack into batch the patches taken from set of images that all share the same spatial position in the image. For example, we may stack the top-left patch of all testing wood images into a batch and use ACR to detect anomalies in that batch. Finally, the window-wise anomaly scores are bilinearly interpolated to the size of the original image, i.c., the pixel-Ievel anomaly scores.
Following the same data pre-pr pre-procsssing steps of Cohen and Hoshen [12], Defard et al. [15], Rippel et al. [62], extract the texture representations (the third layer's output in our case) of WideResNet- 50-2 pre-trained on ImageNet. After feature extraction, a batch of B images leads to a representation of size (B, C, H, W) (B, 1024, 14, 14). These representations contain both textual and spatial information. During meta-training, we treat each spatial position as one new class so that there are 14 x 14 196 new classes for each original class (e.g., wood), and each new class contains B data points, each a 1024 long vector. As a result, the model (DSVDD in our usage) takes a batch of vectors of size (B, 1024) as input and assigns anomaly scores each vector within the batch. When adding synthetic abnormal data (Eq. (6)). we use Gaussian noise corrupted input vectors rather than new class data, incorporating the fact that local defects result in similar feature vectors instead of globally different textures. Specifically, we add Gaussian noise sampled from N (O, 0.01I) and set π 0.5 in Eq. (6). Since there is no During testing, we batch each position (h, w) of all images and detect anomalies at position (h, w). Because the defects are local, and there is a possibility that there is no defect at some position (ho, wo), we manually add synthetic noisy vectors into the tested batch as the training procedure to ensure the images have low anomaly score at (ho, wo). After getting anomaly scores, we remove the synthetic vector results, leading to scores of size ( (B, 14, 14), and then upscale the scores into the original image size by bilinear interpolation. We acknowledge that using CutPaste [45] to generate more realistic synthetic abnormal samples is another option. We leave the investigation to future work.
ACR-DSVDD and Hyperparameter Search. Our model is five-layer MLP with intermedi- ate batch normalization layers and ReI.U activations. The hidden sizes of the perceptrons are [512, 256, 128, 64, 32]. The center is size 32. The statistics of all batch normalization layers are computed on fly on the training/test batches. We average the gradients of 32 randomly sampled tasks for each parameter update. Each task contains 30 normal feature vectors and 30 noise-corrupted feature vectors. We train the model with Meta Outlier Exposure loss. We set the learning rate 0.0003 and iterate 50 updates for each class. We search the hyperparameters on test subset of bottle class (half of the original test set) and apply the same hyperparameters to all classes afterward.
F.3 Implementation Details on Tabular Data
ACR-NTL has the same model architecture as the baseline NTL, and ACR-DSVDD adds one additional batch normalization layer top of the DSVDD baseline. Our algorithm is applicable to the existing backbone models without complex modifications.
ACR-DSVDD. ACR is applied to the backbone model DSVDD [63]. The neural network of DSVDD is four-layer MLP with intermediate batch normalization layers and ReLU activations. The hidden sizes on Anoshift dataset are [128, 128, 128, 32]. The hidden sizes on Malware dataset are [64, 64, 62.. One batch normalization layer is added on the top of the network on Anoshift experiment. The statistics of all batch normalization layers are computed on fly on the training/test batches. We use Adam with learning rate of 4e on Anoshift dataset and le 4 on Malware dataset.
22

---

ACR-NTL. ACR is applied to the backbone model NTL [56]. The shared encoder of NTL is four-layer MLP with intermediate batch normalization layers and ReLU activations. The hidden sizes of the encoder are [128, 128, 128, 32]. The statistics of all batch normalization layers are computed on fly on the training/test batches. We set the number of neural transformations as 19. Each neural transformation is parametrized by three-layer MLP of hidden size of 128 with ReL.U activations. All networks are optimized jointly with Adam with a learning rate of 4e 4.
G
Meta Outlier Exposure Avoids Trivial Solutions.
The benefit of the outlier exposure loss in meta-training is that the learning algorithm cannot simply learn a model on the average data distribution, i... without learning to adapt. This failure to adapt is common problem in meta-learning. Our solution relies on using each training sample x; in different contexts: depending on the sign of Yi.j. data point x; is considered normal (when drawn from Pj) or anomalous (when drawn from Pj). This ambiguity prevents the model from learning an average model over the meta data set and forces it to adapt to individual distributions instead.
For example, DSVDD with its original loss function may suffer from a trivial solution that maps any input data to the origin in the feature space and achieves the optimal zero loss [63]. This trivial solution also possible in our proposed meta-training procedure. But Meta Outlier Exposure gets rid of this trivial solution because mapping everything to the origin incurs an infinite loss on Ag. Similar reasoning also applies to binary cross entropy loss.
H Connections to Other Areas
Our problem setup and assumptions share similarities with other research areas but differences are also pronounced.
Connection to Batch Normalization- BBadd Test-time Adaptation (TTA). Many works for TTA feature batch-level predictions [10, 49, 53, 67, 76) assumes its test-time data corrupted but from the same semantic classes as the training data, but the zero-shot AD's test data can be drawn from a completely new class.
Connection to Unsupervised Domain Adaptation (UDA). Although our approach uses unlabeled data like the UDA setting [38], UDA assumes the unlabeled data from the shifted domain is available during training and can be used to update the model parameters. But our method doesn't rely on the availability of novel data during training time and doesn't 't require updating the model parameters during test time.
Connection to Zero-shot Classification. Xian et al. [79] explains the nature of zero-shot classifica- tion and writes that "the crux the matter for all zero-shot learning methods is to associate observed and non-observed classes through some form of auxiliary information which encodes visually distin- guishing properties of objects." The auxiliary information demands extra human annotations like picture attributes. In contrast, our method assumes a batch of test data without human annotations. The test distribution information is automatically contained in the batch statistics.
Connection to Meta Meta-learning, Although we assume that there is a meta-train meta-training dataset available like meta-learning, we don't 't require a support set for updating the model during both training time and test time. The presence of support set differentiates our method from meta-learning in many aspects. First, for the most well-known technique (MAML) in meta-learning, training requires second-order derivative information of the support set loss function, which is computationally expensive and slows the optimization. Second, it is unclear how to select the support set size for adaptation. Sometimes, it may require large support set to achieve good adaptations. For example, OC-MAML needs at least a 10-shot support set to perform on par with our method on CIFARIO0-C: Third, the support set requires labeled data. This already adds burdens to practitioners. Fourth, the model parameter updates require additional maintenance and extra cost during testing.
Connection to Contextual AD. Contextual AD considers a changing notion of normality based on context [25, 74). In contextual AD, the training and testing data are from the same data generating process, which involves (hidden or observed) contextual variables controlling the generation. This is different from our setup. We tackle the problem when the training and testing data are from different data generating processes.
23

---

Table AUC (%) with standard deviation for anomaly detection on CIFARI00-C C and Omniglot. As an ablation, rather than utilizing outlier exposure, we trained Zero-shot BN only on normal data of each task.
CIFAR100-C (Gaussian Noise)
Omniglot
1%
5%
10%
20%
5%
10%
20%
One-class loss (data-adapted) ResNet152
72.2+2.2
73.9 1.4
74.2+0.9
73.8+0.3
96.2+1.0
96,420.8
96.2+0.8 99.0s0.1
70.9+2.2
67.610.2
67.0a0.7
64.920.5
99.240.2
99.1+0.1
Table 5: The effects of batch normalization for zero-shot AD. The first two columns show different combinations of batch normalization usage during training and testing. The third column answers the question whether the type of batchnorm usage works for zero-shot AD.
BatchNorm (train)
BatchNorm (test)
Work?
Yes
X
No
X
No
X
X
No
Table 6: The effects of test time batch size on the results of zero-shot AD. We report the test results in AUC when the contamination ratio is set to 5% and 10%. The studies conducted the Gaussian noise version of CIFAR-100C. On the extreme batch sizes, each batch contains one anomaly.
Batch size
6
16
One anomaly
6..442.3
7.8
82.3
84.8&2.0
Batch size
20
40
60
80
100
5%
83.7+1.9
85.3+
1..61122
85.920.8
85.6=0.7
10%
1.5
86.1
85.7+0.7
05.
85.8=0.6
Table 7: The effects of the number of classes used in training on zero-shot AD. We report the test results in AUC when the contamination ratio fixed 10%. The studies conducted the Omniglot dataset.
# TTriining classes
2
10
15
AUROC
59.0=0.6
71.8=0.6
72.5+0.3
72.2=1.0
75.3=0.4
#Training classes
20
40
80
160
320
640
1200
AUROC
79.0+ 1.0
90.5+0.5
95.3=0.2
97.6+0.2
98.1
98.4+0.1
99.1+0.2
I Additional Results
L.1 Ablation Study
Training with Different Losses. We study the benefits of using meta outlier exposure in Eq. (5) and compare to a) using one-class classification loss [x[]] = 1/2 Exe (xx) with (xx) = [x)) cll² where @y is the feature map, b) (data-adapted) ResNet152. The data- adapted ResNet152 first learns the features by performing multi-class classification task with the meta-train training set. Then a batch normalization layer is applied on the top of penultimate layer representations for zero-shot anomaly detection. We train a 100-class classifier for CIFAR100C and Omniglot separately. Note that for Omniglot, we randomly sub-sample 100 classes from its 1400 training classes and train the classifier. From the results in Tab. we can see that both ablations perform competitive with ACR on the simple Omniglot dataset, but perform much worse compared to ACR on the complex CIFAR 100-C dataset. conclusion, using meta outlier exposure in training is favorable.
Training or Testing Without NatchNorm. We investigate whether training or testing without batch normalization works for zero-shot AD or not. To this end, we employ four different combinations
24

---

of batch normalization usage during training and testing and check which combination works and which doesn't. We trained the models with the same meta-training procedure as what we used in Sec. 4.1.1 and tested on CIFAR column, "Yes" indicates the AUROC metric is significantly larger than 0.5, and therefore learns a meaningful zero-shot AD model; "No" indicates the AUROC performance is around 0.5, which means the predicted anomaly scores are just random guesses and the model cannot be used for zero-shot AD. Tab. 5 shows that only when the batch normalization is used both in training and testing, the zero-shot AD works. Otherwise, the meta-training procedure couldn' ' result in meaningful zero-shot AD representations.
and Omniglot. We present the results in Tab. 5. In the third
Moreover, for the DSVDD model, we can theoretically show that training without batch normalization will not work with meta outlier exposure: the optimal loss function has nothing to do with zero-shot AD. Rather, the optimal loss is only related the mixture weight и during training. Without loss of generality, suppose we have two training distributions P1, P2. We learn Deep SVDD model parameterized by 0 and e by the meta outlier exposure method, 1(0cc)
(fo(z) - c)2 -E--PT (L- c)2+
1/1
-Ex- PT (1 (L=)))(f((z)-Q0 c)² 2
Y/2
(fo(z)
c)2
= Emi-Piee-Ps ((6(())) -c)
1-T T7
+ - -))²+
1-
(17)
(fo(12) c)2
(fo(x)) c)
1-x
Bup [(o((x)) -))2+ E1
(fo(z))
2 ((
where 0.5 < π < implies the majority assumption and the equality holds when the model parameters (0 and e) are tuned such that (fo(z;) - c)² =V(1 /(1-x)/m for any zz. All data points will be put at the hypersphere's 's surface centered around e with a radius /(1- - ((1->)) x)/T in the feature space when the model is trivially optimized. However, the optimal loss has nothing to do with distinguishing different distribution's 's input data x in the feature space, which is unlikely to produce useful representations for zero-shot AD.
On the other hand, if we apply batch normalization in the model fa. we will not have the optimal loss function irrelevant to distributions. To see this, note that batch normalization will shift the input toward the origin. Thus ω1 in training task PT and 22 in training task PT should have similar representations they both take the majority in each task. Similarly, 22 in task Pf and 21 in task P are minorities, thus mapped far away from the origin in the feature space. Therefore, the symmetry breaks in Eq. (17), and the above trivial optimal loss disappears.
Ablation Study on Batch Sizes. To test the batch size effects, we add an ablation study on the Gaussian noise version of CIFAR- where we fix the anomaly ratio as 5% or 10% and try different batch sizes. The results are summarized in Tab. results. The performance is similar when the batch size is larger than or equal to 40. Even with a batch size being 20, our results are still better than the best-performing baseline.
shows that larger batch sizes lead to more stable
We also test extreme batch sizes being 3, 6, 11, and 16 where each batch contains one anomaly.
Ablation Study on Number of Training Classes To analyze the effect of the number of training distributions on the zero-shot AD performance, we conducted experiments on Omniglot where we varied the number of available meta-training classes from 1, 2, 5, 10, 15, 20, 40, 80 to 640, 1200. We separately trained ACR-DS DSVDD on each setup and tested the resulting models on the test set that has 10% ground-truth anomaly ratio. We repeated 5 runs of the experiment with random initialization and reported the AUROC results in Tab. 7. It shows that using 320 available classes for this dataset is sufficient achieve decent pero-shot/ AD performance. The results also demonstrate that even though we have only one class in the meta-training set, thanks to the batch norm adaptation, we can still get beaterothan-random zero-shot AD performance.
Ablation Study on Other Normalization Techniques. As follows, we report new experiments involving LayerNorm, InstanceNorm, and GroupNorm for zero-shot AD.
25

---

We stress that, while these methods may have overall benefits in terms of enhancing performance, they do not work in isolation in our zero-shot AD setup. A crucial difference between these methods and batch normalization is that they treat each observation individually, rather than computing normalization statistics across a batch of observations. However, sharing information across the batch (and this way implicitly learning about distribution-level information) was crucial for our method to work.
Our experiments (AUROC results in the table below) with DSVDD on the Omniglot dataset sup- port this reasoning. Using these normalization layers in isolations yields to random outcomes (AUROC=50):
LayerNorm
InstanceNorm
GroupNorm
50.0+0.9
50.6+0.7
50.2+0.5
We also added version the experiment where we combined these methods with batch normaliza- tion in the final layer. The results dramatically improve in this case:
BatchNorm (BN)
LayerNorm + BN InstanceNorm + BN GroupNorm + BN
99.1
98.8+0.1
98.8+0.2
98.2+0.2
Experimental details: We use DSVDD as the anomaly detector and experiment on the Omniglot dataset. Each nonlinear layer of the feature extractor for DSVDD is followed by the respective normalization layer. We apply the same training protocol as Tab. 9 in the paper. For GroupNorm, we separate the channels into two groups wherever we apply group normalization.
Effects of BatchNorm (BN) Layer Position. We conducted additional experiments on two visual anomaly detection tasks anomaly segmentation on the MVTec-AD dataset and object-level AD on CIFARIOOC. We used the same DSVDD model architectures as used in Tables 1 and 2 as the backbone model, except that we switched off BN all but one layer. For anomaly segmentation, there are five possible BN layer positions; and there are four positions for the object-level AD model. We switched off the BN layers in all but one position and then re-trained and tested the model with the same protocol used in main paper (For CIFAR 100C, we tested the model with the test data anomaly ratio 10%). We iterate this procedure across all available BN layer positions. We repeat every experiment with different random seeds five times and report the mean AUROC and standard deviation. The results are summarized in the tables below, where a smaller value of the BN position corresponds to earlier layers (close the input), and larger value corresponds to later layers close to the output. The final column copied from our results in Tables and where BN layers are on all available positions. For both MVTec-AD and CIFAR 1OOC, we average the performance across all test classes.
Results on the two tasks have opposite trends regarding the effects of BN layer positions. Specifically, for anomaly segmentation on MVTec-AD, carlier BN layers are more effective, while for AD on CIFARI00C, later BN layers are more effective. This observation can be explained by the fact that anomaly segmentation is more sensitive to low-level features, while object-level AD is more sensitive to global feature representations. In addition, compared to the results in Tables and (copied the last column in the table below), our results suggest that using BN layers at multiple positions does help re-calibrate the data batches of different distributions from low-level features (early layers) to high-level features (late layers) and shows performance improvement over a single BN layer.
MVTec-AD
BN Position
1
2
3
4
5
(1,2,3,4,5)
Pixel-level
80.8
69.6+ 1.4
73.9+0.9
63.6+1.6
60.9+0.8
92.5+0.2
Image-level
7.9
59.2+ 1.6
63.6+1.3
6..5
65.4+1.3
85.8+0.6
CIFARI0OC
BN Position
2
3
(1,233,4)
AUROC
61.4+0.5
61. 61.0+0.9
68.2+0.9
68.9 1.1
85.9+0.4
26

---

majonit
takes majority
Class 2
Ches
0.00
Figure 3: 2D visualization (after PCA) of the adaptively centered representations for two test tasks in the Omniglot dataset. The same learned DSVDD model adapts with our proposed method and maps samples from the majority class (class 1 (left) and class (right)) to the same center in the embedding space in both tasks.
Robustness of the mixing hy typrparraeeter in Eq. (6). We conduct the following experiments with varying æ. The experiment has the same setup as Table on CIFAR100C with a testing anomaly ratio of 0.1. The results show that all tested r's results are over 84% AUC.
CIFARIOOC
π
0.99
0.95
0.9
0.8
0.6
AUROC
85.8
85.4
84.1
85.9+0.4
84.4=0.6
I,2 Visualization of ACR.
We provide a visualization of the learned representations from DSVDD on the Omniglot dataset as qualitative evidence in Fig. 3. We observe that even though the normal and abnormal data classes flip two plots, the model learns center the samples from the majority class and map the samples from the minority class away to the center in the embedding space. In conclusion, ACR is easy-to-use zero-shot AD method and achieves superior zero-shot AD results on different types of images. The performance of ACR is also robust against the test anomaly ratios.
1.3
Additional Results CIFAR100-C
We test all methods on all corruption types CIFARI00-C. The results are presented Tab. 8.
I.4 Additional Results on Non-natural Images
We further evaluate the methods on two other non-natural Latasets-MNIS" and Omniglot of hand-written characters. MNIST uses the same split and evaluation protocol as OrganA. On Omniglot, we take the first 1200 classes to form the meta-training set and use the remaining unseen 423 classes for testing.
Datasets.
Results. We present the results in Tab. 9. It shows that our approach significantly outperforms all the other basclines by a large margin on both datasets.
1,5 Class-wise Results on MVTec-AD
We present the class-wise results in Tab. 10 for finer comparisons with other methods on MVTec AD benchmark.
Besides, we also implement the synthetic anomalies using images from different distributions during training. The result worse than the model using Gaussian corrupt noise as example anomalies but still better than existing works in anomaly segmentation. We summarize the results in Tab. 11, which suggests that using images from different distributions example anomalies during training helpful for anomaly segmentation MVTec-AD.
I.6 Additional Results on Malware
Dataset. Malware [34] is dataset of malicious and benign computer programs, collected from 11/2010 to 07/2014. Malware attacks are designed adversarially, thus leading to shifts in both normal
27

---

and abnormal data. We adopt the data reader from Li et al. [43]. We follow the preprocessing of [34] and convert the real-valued probabilities p of being malware binary labels (labeled one p 0.6 and zero if < 0.4). The samples with probabilities between 0.4 and 0.6 are discarded. The model is trained on normal samples collected from 01/2011 to 12/2013, validated on normal and abnormal samples from 11/2010 to 12/2010, and tested on normal and abnormal samples from 01/2014 to 07/2014 (the anomaly ratios vary between 1% and 20%).
Results. We report the results on Malware in Tab. 12. ACR-NTL achieves the best results under all anomaly ratios. All baselines except ICL perform worse than random guessing, meaning that the malware successfully fools most basclines, which testifies to the adversarial-upgran explanation in the main paper.
28

---

Table 8: AUC (%) with standard deviation for anomaly detection CIFARI00-C [29].
Nete Type
Nobal
14
35
106
205
ACT- BETEICE
$63622
756.31
7
OCAMANE (1-B00
558-36
S61823
ACT BCT
871:24
063435
56505
387.20
Alluds
OCAMANE
538.47
23611
AOR DEVIOO
$15:11
015135
812+10
010633
TE3
722
Rodeer 10:1
663221
CLIPAD
115:30
127634
A33655
REVID
855.30
36081
755-21
755624
125
OC SAAMA (1 10
523427
125611
533411
CLINAD
666216
857231
ACR naVian
85:11
8:511
550-13
10061
BoNaat554
135w13
Rodiee
609615
CLP-AD
407635
REVOD
86521
53623
OCMMANE
555623
Stalt
CLINAD
411214
45911
035011
ACS HOF
653.10
9111
Rediial 42.8
639631
OUAMANE (t-beo
375.15
3001
ACT BSVID
912404
90411
<51R1
365-31
CLI-AD
91.634
125434
923e03
ACEL BEYOO
86411
191444
516:33
CLIPAD
1M663
943033
DEVAO
476684
ANTE ICT
B6663
25533
355555
511000
05635
CLINAD
917683
123634
903w12
ACE BSVOD
852-18
05311
154433
OC
$19:36
02911
534455
CLIPAD
913653
16:633
103014
Sod
ACTACE
059016
$53624
85335
S13011
OCASANE (1-800
525633
S2411
95763
074631
S7A611
905134
66311
OL-MANI
206-11
365613
551-13
Whuth
#16:1
W03355
(05013
OCSMANE (1-800
358623
DEVIO
841221
ACTE P
Nodhe 35.0
OLCMANI
CUI-AD
947645
ACR DAVIO
Mi11
700p1
87A-10
41
CLIN-D
913618
20635
12505
nEVDD
883633
014414
90634
OC 00:000
325.15
CLINAD
Mill1
204055
NOR N
29621
20015
42.0
514615
645055
OCAMAME
154111
ACT DEV3D
915614
JP'S orggenice
8.633
823.24
000635
613405
CTJP A/
935615
383033
13363
29

---

Table 9: AUC (%) with standard deviation for anomaly detection on non-natural images: Omniglot, MNIST, and OrganA. ACR with both backbone models outperforms all baselines on all datasets. In comparison. CLIP-AD performs much worse on non-natural images.
MNIST
Omniglot
1%
5%
10%
5%
10%
20%
ADIB [14]
50.4=2.0
1.7
49.4+2.0
50.8=1.7
49.5=0.6
49.7=0.4
ResNet152-I [27]
87.2+1.3
84.2+0.2
80.9+0.2
96.4=0.4
95.5+0.3
94.3+0.2
ResNer152-II [27]
8..0=1.9
7..4
74.9+0.3
88.1
86.7+0.5
8..4+0.6
OC-MAML [22]
83.7+3.5
86.0#2.3
86.4+2.8
98.6=0.3
98.4+0.2
98.5+0.1
CLIP-AD [51]
53.9=1.4
53.7=0.9
53.9=0.8
N/A
N/A
N/A
ACR-DSVDD
91.9=
90.4=0.2
88.8+0.2
99.1=0.2
99.1=0.2
99.2+ 0.0
ACR-BCE
88.7=0.6
87.8=0.4
86.5+0.3
98.5=0.2
98.9=0.1
99.1+0.1
Table 10: ACR-DSVDD's pixel-level (segmentation) and image-level (classification) AUCROCS (%) MVTec-AD.
Pixel-level
Image-level
Bottle
95.
99.5+0.2
Cable
87.1=1.1
72.2+1.9
Capsule
95.2=0.9
78.1
Carpet
97.8=0.4
99.820.2
Grid
9..4+1.4
83.813.0
Hazelnut
92.3=0.7
79.2=0.9
Leather
98.6=0.1
100.0=0.0
Metal-nut
79.5=2.5
75.6=5.4
Pill
93.0=1.3
72.2+3.2
Screw
86.7=0.9
48.5+3.1
Tile
90.3=0.9
98.4+0.3
Toothbrush
98.2=0.1
97.0=0.9
Transistor
97.2=0.1
93.1=1.1
Wood
89.2=1.1
98.2=0.8
Zipper
95.8=0.9
90.7+0.9
Average
92.5=0.2
85.8
Table 11: ACR-DSVDD's pixel-level (segmentation) and image-level (classification) AUCROCs (%)
on MVTec-AD. The model uses images from other classes as synthetic anomalies during training.
Pixel-level
Image-level
Bottle
94.5
98.6
Cable
88.1
64.5
Capsule
90.1
70.8
Carpet
97.5
99.5
Grid
74.8
97.9
Hazelnut
84.3
61.9
Leather
97.5
99.1
Metal-nut
67.5
54.2
Pill
89.0
66.8
Screw
76.6
53.2
Tile
90.0
97.6
Toothbrush
93.5
80.8
Transistor
95.2
91.4
Wood
88.1
97.2
Zipper
78.6
79.7
Average
87.0
78.8
30

---

Table 12: AUC (%) with standard deviation for anomaly detection on Malware [34]. ACR-NTL achieves the best results on various anomaly ratios.
1%
5%
10%
20%
OC-SVM
19.5
10.5+1.4
20.3+0.9
20.3+0.8
Forest
22.8=2.9
22.9+1.2
23.3+0.6
23.4+0.8
LOF
22.3+4.9
23.2+1.8
23.3+1.3
23.2+0.4
KNN
21.646.3
22.5+1.6
22.7+0.9
22.6+0.9
DSVDD
25.4+3.3
27.4+1.7
28.9+0.9
28.3+0.8
AE
48.8+2.4
49.1+1.2
49.4+0.6
49.3+0.5
LUNAR
23.1+4.5
23.8+1.2
24.1=0.7
24.240.6
ICL
83.5+1.9
81.0+100
82.9+0.8
83.1+0.9
NTL
25.9+4.8
25.4+1.3
24.5+1.3
25.0+0.8
ACR-DSVDD
73.1
69.543.3
69.4+3.3
66.414.0
ACR-NTL
85.0
84.510.8
85.1+1.2
84.010.8
31


Reviews:
[{"review_id": "TQlGKzaGZ9", "reviewer": "Reviewer_Jqv8", "summary": "This paper introduces Adaptive Centered Representations (ACR) to tackle the zero-shot anomaly detection task. To this end, the ACR enables off-the-shelf deep anomaly detectors to generalize to unseen anomaly detection via training data distribution adaption concerning batch normalization and meta-learning. The numerical experiments of image and tabular data conducted on CIFAR100-C, OrganA, MNIST, Omniglot, MVTec AD, Anoshift, and Malware show the effectiveness of the proposed method.", "questions": "The main concern of this paper is threshold selection while implementing anomaly detection. Besides, some discussions about normalizations and implementations are suggested.  ", "limitations": "The authors discussed their limitations in the manuscript.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "•\tThis work introduces a simple yet effective training scheme, which integrates batch normalization and meta-learning, to enable zero-shot anomaly detection. The codes are included for reimplementation.\n\n•\tExperiments on several datasets show the superiority of the proposed method.\n", "weaknesses": "-\tThe proposed method relies on batch-level normalization. How the other normalizations, such as LayerNorm, InstanceNorm, and GroupNorm, affect the proposed training scheme is expected to be discussed.\n\n-\tThe authors propose to apply batch normalizations in multiple layers for different anomaly scorers. Yet, the discussion and analysis of which layers and how their affections are not included.\n\n-\tThe assumption 2 (A2) is about batch-level prediction, i.e., the anomaly scores are estimated based on batches. While carrying out prediction, is it able to automatically select the threshold for recognizing abnormal data? Or it needs to define a threshold per data class manually?\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> Effects of other normalizations, such as LayerNorm, InstanceNorm, and GroupNorm\n\nWe thank the reviewer for mentioning other normalization techniques. As follows, we report new experiments involving LayerNorm, InstanceNorm, and GroupNorm for zero-shot AD. We will add these discussions to the revised paper.\n\nWe stress that, while these methods may have overall benefits in terms of enhancing performance, they do not work in isolation in our zero-shot AD setup. A crucial difference between these methods and batch normalization is that they treat each observation *individually*, rather than computing normalization statistics across a batch of observations. However, sharing information across the batch (and this way implicitly learning about distribution-level information) was crucial for our method to work. \n\nOur experiments (AUROC results in the table below) with DSVDD on the Omniglot dataset support this reasoning. Using these normalization layers in isolations yields to random outcomes (AUROC=50): \n\n|LayerNorm|InstanceNorm|GroupNorm|\n|---|---|---|\n|50.0$\\pm$0.9|50.6$\\pm$0.7|50.2$\\pm$0.5|\nRebuttal Table 1\n\nWe also added a version of the experiment where we combined these methods with batch normalization in the final layer. The results dramatically improve in this case: \n\n|BatchNorm (BN) | LayerNorm + BN | InstanceNorm + BN | GroupNorm + BN|\n|---|---|---|---|\n|99.1$\\pm$0.2|98.8$\\pm$0.1|98.8$\\pm$0.2|98.2$\\pm$0.2|\nRebuttal Table 2\n\nExperimental details:\nWe use DSVDD as the anomaly detector and experiment on the Omniglot dataset. Each nonlinear layer of the feature extractor for DSVDD is followed by the respective normalization layer. We apply the same training protocol as Table 9 in the paper. For GroupNorm, we separate the channels into 2 groups wherever we apply group normalizations.\n\n\n> The authors propose to apply batch normalizations in multiple layers for different anomaly scorers. Yet, the discussion and analysis of which layers and how their affections are not included.\n\nWe indeed work with different AD backbone models and generally apply batch normalization in all layers, which is standard and does not add much to the computational complexity. We agree that it is interesting to analyze whether batchnorm helps more in earlier or later layers. Our new experiments done above suggest that batchnorm in the final layer seems to work fairly well. We leave more systematic studies for the final version of the paper.\n\n> While carrying out prediction (for different data batches), is it able to automatically select the threshold for recognizing abnormal data? Or it needs to define a threshold per data class manually?\n\nAs commonly done in the anomaly detection literature, we use AUROC as our evaluation metric [62, 60, 68, 49, 35] to assess the ranking between normal and abnormal data. That is, we use the anomaly score to rank the data and leave thresholding to the user. Threshold values depend on user requirements such as minimizing false positives, minimizing false negatives, adding sample rejection options, or setting the thresholds as quantiles. That said, an interesting avenue for future research is whether our meta-learning framework allows us to learn to predict suitable thresholds based on the meta-training set. \n"}, {"review_id": "4jW8BqzcyO", "reviewer": "Reviewer_CaFY", "summary": "I think the paper has an interesting motivation but fails to deliver a convincing contribution. The method is incremental and lacks novelty and analysis.", "questions": "Please see the weakness 3.", "limitations": "The paper lacks novelty and theoretical analysis. The method is essentially a combination of existing techniques: batch normalization, meta-learning, and deep anomaly detection. The paper does not provide new insights or theoretical guarantees on why or how the proposed method works.", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "The paper proposes a method for zero-shot batch-level anomaly detection using batch normalization and meta-training. The main idea is to train a deep anomaly detector on a set of interrelated data distributions and use batch normalization to adapt to unseen distributions at test time. The paper claims to achieve state-of-the-art results on image and tabular data. The paper is well-written and easy to follow", "weaknesses": "- The paper relies on three assumptions that may not hold in some real-world scenarios, such as the availability of a meta-training set of interrelated distributions, the batch-level anomaly detection setting, and the majority of normal data in each test batch. The paper does not justify the assumptions and design choices. The paper does not explain how realistic or generalizable these assumptions are, or how to relax or mitigate them in practice.\n\n- The paper lacks novelty and theoretical analysis. The method is essentially a combination of existing techniques: batch normalization, meta-learning, and deep anomaly detection. The paper does not provide new insights or theoretical guarantees on why or how the proposed method works.\n\n- The paper does not provide qualitative results to illustrate how its method works or what kind of anomalies it can detect. Such as some examples of test images or tabular data with their corresponding anomaly scores or segmentation masks generated by its method and compare them with those of the baselines. This would help to better understand the strengths and limitations of its method and provide some insights into its behavior and adaptability.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> The method is incremental and lacks novelty.\n\nWe respectfully disagree. Before our work, zero-shot anomaly detection relied on large, pre-trained foundation models *specifically on image or language data*. We enrich the toolbox and provide a surprisingly simple solution. That is, one can use any existing deep anomaly detector (as long as their network structure contains batch normalization layers) plus proper training and evaluation schemes to get the zero-shot detection ability. In contrast to using foundation models, our approach may result in small, lightweight models that apply to data beyond text and natural images.  We discover the effect of batch normalization in zero-shot AD and propose to use meta-training to learn to use batch normalization to adapt to zero-shot AD. These points are also highlighted in our contribution list in Section 1 and the method explanation in Section 2.3.\n\n> lacks theoretical analysis\n\nOur paper does contain several theoretical results: we 1) derive generalization guarantees to unseen data distributions in Supplement A, 2) analyze why DSVDD fails when batch normalization is not used (Eq. 17), and 3) provide theoretical justifications for Assumption 3. They are mentioned in the main paper and we will refer to them more clearly.\n\n> Justification of the three assumptions.\n\nThanks for pointing this out; we'll revisit our paper for improving clarity in this regard. Our Assumptions 2 and 3 are already justified in the paragraphs directly following their definition; Assumption 2 is the same assumption used in batch-level prediction work [10, 47, 51, 64, 73]. Assumption 1 is the same assumption typically chosen in the meta-learning literature [Finn et al., ICML 2017; Nichol et al., 2018]. In practice, the meta-training set can be generated using available covariates (e.g. for our tabular data experiment, we used the timestamps, in medical data, one could use data collected from different hospitals or different patients to obtain separate sets for meta-training, and in MVTec-AD, we used the other training classes except for the target class to form the training set.) We both empirically and theoretically justified Assumptions 3. \n\n> The paper does not provide qualitative results to illustrate how its method works or what kind of anomalies it can detect.\n\nThanks for pointing this out–in fact, some of these qualitative results are already shown in the supplement. They are mentioned in the main paper and we will point this out more clearly. Figure 2 (supplement) shows why batch normalization is an adaptive zero-shot AD module and Figure 3 (supplement) visualizes how our method captures the normality (i.e., projects the normal samples towards the neighborhood of the “center” and pushes abnormal samples away) in practice on the Omniglot dataset. \n"}, {"review_id": "KECVLuZyrE", "reviewer": "Reviewer_hQFt", "summary": "This paper proposes Adaptive Centered Representations (ACR) for zero-shot batch-level AD. The simple recipe, batch normalization plus meta-training, seems highly effective and versatile. The experiments are well conducted on multiple datasets.", "questions": "Please refer to the weaknesses.", "limitations": "N/A", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1.\tThis paper is well-written, and very easy to follow.\n2.\tThe contributions are very clear.\n3.\tThe experiments are well conducted on multiple datasets.", "weaknesses": "The main issue with this paper is the confusion between the problems of anomaly detection and one-class classification, which leads to several problems, including uncertainty and irrationality in experimental settings, unfair comparisons with competing methods, and misleading statements in the related works section.\n\n1.\tThe key problem: The key problem lies in the authors' confusion between one-class classification and anomaly detection. In the context of industrial anomaly detection using datasets like MVTec, the training set consists of normal objects without any anomalies, while the testing phase targets specific fine-grained anomalies. This discrepancy makes the task extremely challenging. However, in one-class classification, particularly during meta-training, anomalies can be defined as any coarse-grained objects as they are with the same definition of the anomalies for the test, and the training set can include multiple objects, providing some form of supervision during meta-training.\n2.\tMisleading in the related works: There is confusion in the related works section, specifically in lines 194-206, where the authors misinterpret the settings of RegAD [31] compared to other papers [21, 8, 39]. While all these papers involve few-shot learning during meta-testing (although [31] does not explicitly position itself as meta-learning in its original paper), the crucial difference lies in the meta-training phase. In [21, 8, 39], data from multiple classes are used for supervised learning during the meta-training process for the one-class classification meta-task. However, in [31], the data from multiple objects cannot be used for supervised learning since the real anomalies are fine-grained defects, which do not exist in the entire meta-training dataset. As a result, the difficulty of the two tasks in the meta-training phase is fundamentally different, and the authors should clearly clarify these distinctions to avoid misleading readers.\n3.\tUncertainty and irrationality of experimental settings: There are uncertainties and irrationalities in the experimental settings, particularly in the training process for defect detection on the MVTec dataset. It is unclear how anomalous data is included in the training set of the 14 classes, as the authors suggest that most of the data in one batch are normal. It would be helpful to clarify whether some anomalous data is included in the test set of these 14 classes for training purposes. Additionally, during testing, it is unclear how the test data batches are organized. If some classes in MVTec have more anomalous data than normal data, it becomes crucial to address how this imbalance is handled, as it deviates from the assumed scenario.\n4.\tUnfair comparisons with competing methods: Unfair comparisons are made with competing methods. The traditional definition of zero-shot learning implies that the results can be determined for each independent sample independently. In Table 2, methods such as WinCLIP [35] can classify individual images as normal or anomalous independently. However, in this paper, decisions can only be made after processing a large batch of data, which deviates from the true zero-shot setting as demonstrated in [35].\n\nOverall, this is a good paper with clear contributions; however, it can mislead readers and create confusion regarding some important experimental settings.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> relation between one-class classification and anomaly detection. \n\nWe wonder whether our understanding of the question aligns with your intention, so let us give our understanding of the anomaly detection (AD) problem. AD is a broad field that can be formulated in many ways (Table 3 in Ruff et al. 2021). We consider two setups: image-level anomaly detection (the object-level one-vs-rest setup) and anomaly segmentation (featuring the MVTec-AD dataset). In both cases, we aim to distinguish observed data (considered normal) from distributionally-different data (anomalies) encountered in the future. One-class classification (OCC) is a widely-used method to solve this problem. In contrast to binary classification, OCC tries to learn a tight (e.g., circular) decision boundary around the observed normal class (See Fig. 1 in Li et al., 2023), thus distinguishing it from everything else encountered in the future. This is different from binary classification, which usually involves an unbounded (e.g., linear) decision boundary. We don’t expect the potential problems laid out in this question since our approach promotes robustness to future anomalies, as we ensure that anomalies and normal data encountered during testing (e.g., MNIST digits 5-9) are held out during meta-training (which in the MNIST example only uses digits 0-4). \n\n[Ruff, Lukas, et al. \"A unifying review of deep and shallow anomaly detection.\" Proceedings of the IEEE 109.5 (2021): 756-795.]\n\n[Li et al., Deep Anomaly Detection under Labeling Budget Constraints, ICML 2023]\n\n> anomalies can be defined as any coarse-grained objects as they are with the same definition of the anomalies for the test, and the training set can include multiple objects, providing some form of supervision during meta-training.\n\nWe are not sure we understand the question–are you suggesting that our approach may not be robust to novel types of anomalies encountered during testing? Even though we use a meta-training set, our train-test split of the classes ensures that the objects or classes in the meta-training set are not present at test time, so there is no “information leakage” between the data distributions encountered between meta-training and testing. The availability of a meta-training set provides the possibility of supervising the anomaly detector with synthetic abnormal data from other classes. Second, our ablation study showed that using the supervision signal from synthetic abnormal data at training time leads to better zero-shot AD performance than not using it (compare “one-class loss” in Table 4 against the results in Table 1).\n\n> Additional information on RegAD [31] in the related works\n\nThanks for your information on the additional differences between RegAD [31] and other related works. We will clarify by adding the following sentence in the related work: “RegAD[31] does not exploit the presence of a meta-set to learn a stronger anomaly detector through synthetic outlier exposure. We found that using images from different distributions as example anomalies during training is helpful for anomaly segmentation on MVTec.”\n\n> Meta-training and testing details on MVTec\n\nThe implementation details can be found in Section 4.2, E.2, and our codebase. In summary,\n* When we aim to detect anomalies in one targeted class, we construct the meta-training set by removing the target class from the original MVTec training set.\n* We use class-level images to form supervision signals during meta-training. We don’t use any realistic defects from the official test set at training time. \n* We ensure that we have a clean split between distributions (object classes) available for meta-training and meta-testing. This includes training and test sets.\n* Since our goal is anomaly segmentation, we assign an anomaly score to each data patch, using a sliding window approach.\n* Note that the assumption that normal data are more present than anomalies still holds for anomaly segmentation since anomalous segments are usually localized in tiny regions in the images. (We also noticed that at the image level, there could be more anomalous images than normal ones but this isn’t the case at the patch level.) \n\n> Unfair comparisons with competing methods\n\nWe disagree that our usage of batch-level prediction is unfair to other baselines that score data individually. Zero-shot learning relies on auxiliary information to inform a model of what the current task is [Xian et al., 2018]. Zero-shot AD approaches operate differently when it comes to identifying the “new normal” distribution. WinCLIP carefully selects the prompts handmade by human experts as auxiliary information for normality. In contrast, we use batch-level information as auxiliary information to achieve zero-shot anomaly detection, for which we don’t need any human experts at all. The benefits of our method are three-fold. 1) Batches are cheaper than human expert labeling and prompt engineering, so there is no additional cost. 2) Our method also works for small batches in our ablation studies. 3) We can extend our method to score individual data by presetting the sample mean and variance in BatchNorm layers with a collection of data. These moments are then fixed when predicting anomaly scores for new individual data.\n\n[Xian, Yongqin, et al. “Zero-shot learning—a comprehensive evaluation of the good, the bad and the ugly. TPAMI 2018]\n"}, {"review_id": "FM5SBBFZAT", "reviewer": "Reviewer_GN5r", "summary": "This paper proposes a method called Adaptive Centered Representations (ACR) for zero-shot batch-level anomaly detection. The method utilizes off-the-shelf deep anomaly detectors trained on a meta-training set, combined with batch normalization layers, to achieve automatic zero-shot generalization for unseen anomaly detection tasks. The main contributions are introducing an effective method for zero-shot anomaly detection, demonstrating its applicability to tabular data, and achieving competitive results in anomaly segmentation on specialized image datasets.\n\n\n", "questions": "1. In order to improve the clarity and understandability of this paper, it is suggested to add a schematic diagram outlining the overall method. The visualization diagram of the whole paper is too little, which makes it difficult to grasp the overall method principle.\n2. When discussing meta-learning, I want to explain some basic concepts, rather than explaining Batch Normalization Layers, and I want to explain the pseudo-code of the supplementary material in the article, which ensures that the reader has a full understanding of the background and significance of these models.", "limitations": "Limitations are clearly described and I don't expect negative societal impact of this work.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "In this paper, the effectiveness of the proposed method is verified by a large number of experiments. There is a rigorous mathematical proof process, and the effectiveness of the method is verified by experiments on multiple data sets\nThe results are powerful. In most cases, the proposed methods perform very well.", "weaknesses": "1.The article claims that Batch Normalization Layers can improve the model's adaptive ability in new tasks. In article 4.1.1, Anomaly Detection experiment of CIFAR100-C with Gaussian noise and medical image dataset OrganA was conducted. Why ACR-DSVDD with the addition of Batch Normalization Layers performs better overall than ACR-BCE is not further explained here.\n\n2.As for the anomaly ratio π in this paper, I think it is also a major hyperparameter indicator. The paper only claims that the model is robust when it is equal to 0.01, 0.05, 0.1, 0.2, and 0.4. Ablation experiments with this parameter are expected to prove its robustness. In the Implementation Details of section 4.1.1 it is claimed that π=0.8, but the supplementary material says that π=0.2", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> Why ACR-DSVDD with the addition of Batch Normalization Layers performs better overall than ACR-BCE is not further explained here.\n\nDSVDD has a specific inductive bias for anomaly detection to learn a compact normal data boundary, while a binary classifier doesn’t have such an inductive bias. Li et al. 2023 visualize this inductive bias through the normal data boundary in Fig 1, showing that a binary classifier using binary cross entropy loss (BCE) has a more conservative normal data boundary than DSVDD. \n\n[Li et al. Deep Anomaly Detection under Labeling Budget Constraints. ICML 2023.]\n\n> Robustness of the mixing hyperparameter $\\pi$ in Eq. 6.\n\nThanks for mentioning this ablation study. We conduct the following experiments with varying $\\pi$. The experiment has the same setup as Table 1 on CIFAR100C with a testing anomaly ratio of 0.1. The results show that all tested $\\pi$’s results are over 84% AUC.\n\n|0.99|0.95|0.9|0.8|0.6|\n|---|---|---|---|---|\n|85.8$\\pm$0.5|85.4$\\pm$0.5|84.1$\\pm$0.4|85.9$\\pm$0.4|84.4$\\pm$0.6|\nRebuttal Table 3\n\n> Inconsistency values for the mixing hyperparameter $\\pi$ in the main paper and the supplement.\n\nThank you for noticing this inconsistency and we will correct the typos in the supplement. We report 1-$\\pi$ in the supplement but report $\\pi$ in the main paper. This will be fixed.\n"}, {"review_id": "GrHHX4aaf5", "reviewer": "Reviewer_2urY", "summary": "The paper presents an approach to address the challenging task of zero-shot anomaly detection. The authors base their methodology on three crucial assumptions: the presence of a training set that shares relevant structures with the target task, performing batch-level analysis during inference, and having the majority of the batch comprised of normal class data. Leveraging these assumptions, the authors propose a method that effectively utilizes batch statistics to assign anomaly scores.\nThe proposed method, named Adaptive Centered Representations (ACR), combines batch normalization with meta-training. Through re-centering and scaling the normal samples within a batch, ACR successfully brings them closer to the center while pushing anomalies further away. Experiments were conducted in the study across different types of data, including both image and tabular data.\n", "questions": "1. The paper primarily addresses the one-to-many anomaly detection scenario, but it would be interesting to evaluate its performance in the more realistic many-vs-one scenario [1]. In this scenario, where the data is multi-modal and achieving compactness is more challenging. How the proposed method works on this setting?\n\n2. The assumption of batch-level inference may not be applicable in many practical scenarios, such as the mentioned hospitals and clinics where doctors often assess patients individually rather than in large batches. The performance drop observed in Table 6 with a batch size of 20 raises questions about the method's effectiveness with even smaller batch sizes, such as 5 or 10. What is the performance of the proposed method with lower batch sizes?\n\n\nReferences:\n[1] Ahmed, F.; and Courville, A. Detecting semantic anomalies. AAAI 2020.", "limitations": "Yes.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "The authors tackle the task of zero-shot anomaly detection, which holds significant practical importance in various domains. The clear and well-written notation and assumption section greatly contributes to understanding the authors' approach and provides a solid foundation for the rest of the paper. Experimental evaluations conducted demonstrate the effectiveness of the proposed approach.\n\n", "weaknesses": "The paper has some weaknesses that should be addressed. Mainly, the reliance on some critical assumptions, such as the availability of a meta-training set and batch-level anomaly detection, may limit the practical applicability of the proposed method. \n\n1. The assumption of a meta-training set assumes strong prior knowledge about the types of anomalies, which may not be feasible or practical in many real-world scenarios.\n\n2. Table 7 shows that strong performance required over 100 different classes during training. This raises concerns about the scalability and feasibility of the method when faced with a smaller number of training classes. This is indicated by the lower performance observed when classes were reduced to 20. Further exploration of the method's performance with a smaller number of training sets would provide valuable insights into its limitations and generalization capabilities.\n\n3. The experimental section of the paper could benefit from improved clarity and organization. It was challenging to follow the exact details of the conducted experiments, requiring multiple passes through the main paper and the Appendix. Enhancing the clarity and structure of the experimental section would greatly improve reproducibility and understanding of the reported results.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> The assumption of a meta-training set is strong\n\nThis assumption is common in the meta-training literature, both for general methods, such as MAML as well as meta-training AD work [8, 21, 31, 39]. There are multiple ways to mitigate this assumption. If one does not have a meta-training set at hand, one can train their model on a different but related dataset, .e.g, train on Omniglot but test on MNIST (see results below under this setting. We still get decent AUC results on MNIST).\n\nAnomaly ratio|1%|5%|10%|20%|\n|:---|---|---|---|---|\n|AUROC|84.4$\\pm$2.4|85.2$\\pm$2.5|84.3$\\pm$2.5|82.2$\\pm$2.4|\nRebuttal Table 4: MNIST\n\nIn our main paper, we provide several examples of how to construct a meta-training set for real applications: for our tabular data benchmark, we use the timestamps to construct a meta-training set; in our MVTec-AD experiment, we simply use multiple industrial object classes to construct the meta-training set; in medical domains, we have data from different hospitals or patients or organs to construct the meta-training set. \n\n> Experiments with the extreme number of classes during training (e.g., 1)\n\nThank you for mentioning this extreme setting. We will add the ablation study to Table 7. The results show that even though we have only one class in the meta-training set, thanks to the batch norm adaptation, we can still get better-than-random zero-shot AD performance. \n\n|#Training classes|1|2|5|10|15|\n|:---|---|---|---|---|---|\n|AUROC|59.0$\\pm$0.6|71.8$\\pm$0.6|72.5$\\pm$0.3|72.2$\\pm$1.0|75.3$\\pm$0.4|\nRebuttal Table 5\n\n> Experimental section lacks clarity and organization. \n\nWe did multiple experiments on various data types and anomaly detection tasks. Our experimental section is structured by data types. For each data type, we first provide implementation details (train/val/test split and meta-training/testing construction) and then the results. In addition, pseudo-code and our codebase are provided in the supplement for reproducibility. We are grateful about more specific advice that we could incorporate.\n\n> How the proposed method works in other settings?\n\nMany-vs-one is the reversed version of the one-vs-rest setting. We study a more realistic setup: the N-vs-rest setting on Omniglot. We train the model under the one-vs-rest setup but test under multiple normal modes. The table below shows that as the number of normal classes (N) increases, the performance decreases because the sample mean and variance are not representative of all normal classes. We experiment with the anomaly ratio being 0.1 at test time.\n\n|N=1|N=2|N=5|N=10|N=20|\n|---|---|---|---|---|\n|99.0|95.6|81.5|65.1|28.9|\nRebuttal Table 6\n\n> Validity of batch-level anomaly detection.\n\nBatch-level predictions are widely used in real life. For example, people examine Covid19 test samples at a batch level out of economic and time-efficiency considerations (https://www.fda.gov/medical-devices/coronavirus-covid-19-and-medical-devices/pooled-sample-testing-and-screening-testing-covid-19). Our work makes similar assumptions as batch-level predictions in the literature [10, 47, 51, 64, 73]. Our method can be easily extended to score individual data by presetting the sample mean and variance in BatchNorm layers with a collection of data. These moments are then fixed when predicting new individual data.\n\n\n> Performance in lower batch sizes. \n\nThe table below shows the performance in the lower batch size regime. We will add the results below to Table 6. Each batch contains 1 anomaly. \n\n|Batch size|3|6|11|16|\n|:---|---|---|---|---|\n|AUROC|66.4 $\\pm$ 2.3|77.9 $\\pm$ 2.8|82.3 $\\pm$ 2.7|84.8 $\\pm$ 2.0|\nRebuttal Table 7\n"}]

Author rebuttal:
We thank the reviewers for all the valuable suggestions on additional ablations. Based on their inputs, we have run additional experiments to study different normalization layers (Rebuttal Tables 1 and 2), studying different values for the hyperparameter $\pi$ (Rebuttal Table 3), meta-OOD performance (Rebuttal Table 4), additional varying number of training classes (Rebuttal Table 5), N-vs-rest setting (Rebuttal Table 6), and tiny batch sizes (Rebuttal Table 7). The most interesting findings will be included in the main paper to complement the extensive empirical study: we have studied three different tasks (image-level anomaly detection, anomaly segmentation, and anomaly detection in tabular data)  where each task contains multiple datasets and we compared at least six different methods for each task.

Meta-review: This paper tackles a relatively unexplored meta-learning setting for anomaly detection. The reviewers appreciated the simple approach and the strong performance. Furthermore, the AD community has recently been exploring new modes of supervision for anomaly detection given the impossibility of fully-unsupervised AD. Meta-learning is a promising direction and the simplicity of this approach may encourage further work in this direction. The rebuttal convinced all reviewers to update their reviews to accept or borderline-accept. The AC concurs and recommends acceptance as well.
Decision: Accept (poster)