SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Aodong Li*
Chen Qiu*
TU Kaiserslautern
Padhraic Smyth
Maja Rudolph†
Stephan Mandt†
Anomaly detection (AD) plays a crucial role in many safety-critical application domains. The challenge of adapting an anomaly detector to drift in the normal data distribution, especially when no training data is available for the "new normal", has led to the development of zero-shot AD techniques. In this paper, we propose a simple yet effective method called Adaptive Centered Representations (ACR) for zero-shot batch-level AD. Our approach trains off-the-shelf deep anomaly detectors (such as deep SVDD) to adapt to a set of inter-related training data distributions in combination with batch normalization, enabling automatic zero-shot generalization for unseen AD tasks. This simple recipe, batch normalization plus meta-training, is a highly effective and versatile tool. Our theoretical results guarantee the zero-shot generalization for unseen AD tasks; our empirical results demonstrate the first zero-shot AD results for tabular data and outperform existing methods in zero-shot anomaly detection and segmentation on image data from specialized domains. Code is at https://github.com/aodongli/zero-shot-ad-via-batch-norm
Anomaly detection (AD)—the task of identifying data instances deviating from the norm [65]—plays a significant role in numerous application domains, such as fake review identification, bot detection in social networks, tumor recognition, and industrial fault detection. AD is particularly crucial in safety-critical applications where failing to recognize anomalies, for example, in a chemical plant or a self-driving car, can risk lives.
Consider a medical setting where an anomaly detector encounters a batch of medical images from different patients. The medical images have been recorded with a new imaging technology different from the training data, or the patients are from a demographic the anomaly detector has not been trained on. Our goal is to develop an anomaly detector that can still process such data using batches, assigning low scores to normal images and high scores to anomalies (i.e., images that differ systematically) without retraining. To achieve this zero-shot adaptation, we exploit the fact that anomalies are rare. Given a new batch of test data, a zero-shot AD method [18, 36, 51, 71] has to detect which features are typical of the majority of normal samples and which features are atypical.
We propose Adaptive Centered Representations (ACR), a lightweight zero-shot AD method that combines two simple ideas: batch normalization and meta-training. Assuming an overall majority of "normal" samples, a randomly-sampled batch will typically have more normal samples than anomalies. The effect of batch normalization is then to draw these normal samples closer to the center (in its recentering and scaling operation), while anomalies will end up further away from the center. Notably, this scaling and centering is robust to a distribution shift in the input, allowing a
*Equal contribution † Joint supervision. Correspondence to: aodong11@uci.edu
Figure 1: a) Demonstrations of concrete examples of a meta-training set and a testing distribution. It is not necessary for the meta-training set to include the exact types of samples encountered during testing. For instance, when detecting lions within geese, the training data does not need to include lions or geese. b) Illustration of zero-shot batch-level AD with ACR using a one-class classifier [63]. The approach encounters three tasks  $(P_{1:3}^{\pi}$ , Eq. (6)) during training (black arrows) and learns to map each task's majority of samples (i.e., the normal samples) to a shared learned center in embedding space. At test time (blue arrow), the learned model maps the normal (majority) samples to the same center and the distance from the center serves as AD score.
self-supervised anomaly detector to generalize to distributions never encountered during training. We propose a meta-training scheme to unlock the power of batch normalization layers for zero-shot AD. During training, the anomaly detector will see many different anomaly detection tasks, mixed from different choices for normal and abnormal examples. Through this variability in the training tasks, the anomaly detector will learn to rely as much as possible on the batch normalization operations in its architecture.
Advantages of ACR include that it is theoretically grounded, simple, domain-independent, and compatible with various backbone models commonly used in deep AD [56, 63]. Contrary to recent approaches based on foundation models [36], applicable only to images, ACR can be employed on data from any domain, such as time series, tabular data, or graphs.
We begin by presenting our assumptions and method in Sec. 2. Next, with the main idea in mind, we describe the related work in Sec. 3. We demonstrate the effectiveness of our method with experiments in Sec. 4. Finally, we conclude our work and state the limitations and societal impacts.
- An effective new method. Our results for the first time show that training off-the-shelf deep anomaly detectors on a meta-training set, using batch normalization layers, gives automatic zero-shot generalization for AD. For which we derive a generalization bound on anomaly scores.
- Zero-shot AD on tabular data. We provide the first empirical study of zero-shot AD on tabular data, where our adaptation approach retains high accuracy.
- Competitive results for images. Our results demonstrate not only a substantial improvement in zero-shot AD performance for non-natural images, including medical imaging but also establish a new state-of-the-art in anomaly segmentation on the MVtec AD benchmark [5].
We begin with the problem statement in Sec. 2.1 and then state the assumptions in Sec. 2.2. Finally we present our proposed solution in Sec. 2.3. The training procedure is outlined in Alg. 1 in Supp. C.
2.1 Problem Statement and Method Overview
We consider the problem of learning an anomaly detector that is required to immediately adapt (without any further training) when deployed in a new environment. The main idea is to use batch normalization as a mechanism for adaptive batch-level AD. For any batch of data containing mostly "normal" samples, each batch normalization shifts its inputs to the origin, thereby (1) enabling the discrimination between normal data and outliers/anomalies, and (2) bringing data from different distributions into a common frame of reference. (Notably, we propose applying batch norm in multiple layers for different anomaly scorers.) For the algorithm to generalize to unseen distributions,
we train our model on multiple data sets of "normal" data simultaneously, making sure each training batch contains a majority of related data points (from the same distribution) at a time.
Fig. 1a illustrates this idea, where all distributions are exemplified based on the example of homogeneous groups of animals (only dogs, only robins, etc.) The goal is to detect a lion among geese, where neither geese nor lions have been encountered before. Fig. 1b illustrates the scheme based on the popular example of deep support vector data description (DSVDD) [63], where samples are mapped to a pre-specified point in an embedding space and scored based on their distance to this point. All training distributions are mapped to the same point, as enabled through batch normalization.
2.2 Notation and Assumptions
To formalize the notion of a meta-training set, we consider a distribution of interrelated data distributions (previously referred to as groups) as commonly studied in meta-learning and zero-shot learning [3, 21, 22, 32]. This inter-relatedness can be expressed by assuming that  $K$  training distributions  $P_{1},\ldots ,P_{K}$  and a test distribution  $P_{*}$  are sampled from a meta-distribution  $\mathcal{Q}$ :
$$
P _ {1}, \dots , P _ {K}, P _ {*} \stackrel {{\text {i . i . d .}}} {{\sim}} \mathcal {Q}. \tag {1}
$$
We assume that the distributions in  $\mathcal{Q}$  share some common structure, such that training a model on one distribution has the potential to aid in deploying the model on another distribution. For example, the data  $\mathbf{x}$  could be radiology images from patients, and each  $P_{j}$  or  $P_{\star}$  could be a distribution of images from a specific hospital. These distributions share similarities but differ systematically because of differences in radiology equipment, calibration, and patient demographics<sup>2</sup>. Each of the distributions  $P \in \mathcal{Q}$  defines a different anomaly detection task. For each task, we have to obtain an anomaly scoring function  $S_{\theta}$  that assigns low scores to normal samples  $\mathbf{x} \sim P$  and high scores to anomalies.
We now consider a batch  $\mathcal{B} \subset \mathcal{D}$  of size  $B$ , taken from an underlying data set  $\mathcal{D} \sim P$  of size  $N$ . The batch can be characterized by indexing data points from  $\mathcal{D}$ :
$$
\mathcal {B} \equiv \left(i _ {1}, \dots , i _ {B}\right) \sim \operatorname {U n i f} \left(\{1, \dots , N \}\right). \tag {2}
$$
We denote the anomaly scores on a batch level by defining a vector-valued anomaly score
$$
\mathbf {S} _ {\theta} \left(\mathbf {x} _ {\mathcal {B}}\right) = \left(S _ {\theta} ^ {i _ {1}} \left(\mathbf {x} _ {\mathcal {B}}\right), \dots , S _ {\theta} ^ {i _ {\mathcal {B}}} \left(\mathbf {x} _ {\mathcal {B}}\right)\right), \tag {3}
$$
indicating the anomaly score for every datum in a batch. By thresholding the anomaly scores  $S_{\theta}^{i}(\mathbf{x}_{\mathcal{B}})$ , we obtain binary predictions of whether data point  $\mathbf{x}_i$  is anomalous in the context of batch  $\mathbf{x}_{\mathcal{B}}$ .
By conditioning on a batch of samples, our approach obtains distributional information beyond a single sample. For example, an image of a cat may be normal in the context of a batch of cat images, but it may be anomalous in the context of a batch of otherwise dog images. This is different from current deep anomaly detection schemes that evaluate anomaly scores without referring to a context.
Before presenting a learning scheme of how to combine batch-level information in conjunction with established anomaly detection approaches, we discuss the assumptions that our approach makes. (The empirical or theoretical justifications as well as possibilities of removing or mitigating the assumptions can be found in Supp. A.)
A1 Availability of a meta-training set. As discussed above, we assume the availability of a set of interrelated distributions. The meta-set is used to learn a model that can adapt without re-training.
A2 Batch-level anomaly detection. As mentioned above, we assume we perform batch-level predictions at test time, allowing us to detect anomalies based on reference data in the batch.
A3 Majority of normal data. We assume that normal data points take the majority in every i.i.d. sampled test batch.
Due to the absence of anomaly labels (or text descriptions) at test-time, we cannot infer the correct anomaly labels without assumptions A2 and A3. Together, they instruct us that given a batch of test examples, the majority of the samples in the batch constitute normal samples.
2The divergence between distributions can be much larger than shown in this example. See our experiments.
2.3 Adaptively Centered Representations
Batch Normalization as Adaptation Modules. An important component of our method is batch normalization, which shifts and re-scales any data batch  $\mathbf{x}_{\mathcal{B}}$  to have a sample mean zero and variance one. Batch normalization also provides a naive parameter-free zero-shot batch-level anomaly detector:
$$
S _ {\mathrm {n a i v e}} ^ {i} \left(\mathbf {x} _ {\mathcal {B}}\right) = \left\| \left(\mathbf {x} _ {i} - \bar {\mu} _ {\mathbf {x} _ {\mathcal {B}}}\right) / \bar {\sigma} _ {\mathbf {x} _ {\mathcal {B}}} \right\| _ {2} ^ {2}, \tag {4}
$$
where  $\bar{\mu}$  and  $\bar{\sigma}^2$  are the coordinate-wise sample mean and sample variance.  $\bar{\mu}$  is dominated by the majority of the batch, which by assumption A3, is the normal data. If the  $\mathbf{x}_i$  lie in an informative feature space, anomalies will have a higher-than-usual distance to the mean, making the approach a simple, adaptive AD method, illustrated in Fig. 2 in Supp. D.
While the example provides a proof of concept, in practice, the normal samples typically do not concentrate around their mean in the raw data space. Next, we integrate this idea into neural networks and develop an approach that learns adaptively centered representations for zero-shot AD.
Deep Models with Batch Normalization Layers as Scalable Zero-shot Anomaly Detectors. In deep neural networks, the adaptation ability is obtained for free with batch normalization layers [35]. Batch normalization has become a standard and necessary component to facilitate optimization convergence in training neural networks. In common neural network architectures [27, 33, 60], batch normalization layers are used after each non-linear transformation layer, making a zero-shot adaptation with respect to its input batch. The entire neural network, stacking up many non-linear transformation and normalization layers, has powerful potential in scalable zero-shot adaptation and learning adaptation-needed feature representations for complex data forms.
Training Objective. As discussed above, we can instantiate  $S_{\theta}$  as a deep neural network with batch normalization layers and optimize the neural network weights  $\theta$ . We first provide our objective function and then the rationality. Our approach is compatible with a wide range of deep anomaly detection objectives; therefore we consider a generic loss function  $L[\mathbf{S}_{\theta}(\mathbf{x}_{\mathcal{B}})]$  that is a function of the anomaly score. For example, in many cases, the loss function to be minimized is the anomaly score itself (averaged over the batch).
The availability of a meta-data set (A1) gives rise to the following minimization problem:
$$
\theta^ {*} = \underset {\theta} {\arg \min } \frac {1}{K} \sum_ {j = 1} ^ {K} \mathbb {E} _ {\mathbf {x} _ {\mathcal {B}} \sim P _ {j}} L \left[ \mathbf {S} _ {\theta} \left(\mathbf {x} _ {\mathcal {B}}\right) \right]. \tag {5}
$$
Typical choices for  $L[\mathbf{S}_{\theta}(\mathbf{x}_{\mathcal{B}})]$  include DSVDD [64] and neural transformation learning (NTL) [56]. Details and modifications of this objective will follow.
Why does it work? Batch normalization helps re-calibrate the data batches of different distributions into similar forms: normal data will center around the origin. Such calibration happens from granular features (lower layers) to high-level features (higher layers), resulting in powerful feature learning and adaptation ability<sup>3</sup>. We visualize the calibration in Fig. 3 in Supp. I.2. Therefore, optimizing Eq. (5) are able to learn a (locally) optimal  $\mathbf{S}_{\theta^*}$  that is adaptive to all  $K$  different training distributions. Such learned adaptation ability will be guaranteed to generalize to unseen related distributions  $P_*$ . See Sec. 2.4 below and Supp. B for more details.
Meta Outlier Exposure. While Eq. (5) can be a viable objective, we can significantly improve over it while avoiding trivial solutions<sup>4</sup>. The approach builds on treating samples from other distributions as anomalies during training. The idea is that the synthetic anomalies can be used to guide learning a tighter decision boundary around the normal data [30]. Drawing on the notation from Eq. 1, we thus simulate a mixture distribution by contaminating each  $P_{j}$  by admixing a fraction  $(1 - \pi) \ll 1$  of data from other available training distributions. The resulting corrupted distribution  $P_{j}^{\pi}$  is thereby
$$
P _ {j} ^ {\pi} := \pi P _ {j} + (1 - \pi) \bar {P} _ {j}, \quad \bar {P} _ {j} := \frac {1}{K - 1} \sum_ {i \neq j} P _ {i} \tag {6}
$$
This notation captures the case where the training distribution is free of anomalies  $(\pi = 1)$ .
3Without batch normalization, optimizing Eq. (5) can be meaningless for some objectives. See Supp. I.1. 4We explain how this objective avoids trivial solutions in Supp. G and show the benefits in Tab. 4 of Supp. I.1.
Next, we discuss constructing an additional loss for the admixed anomalies, whose identity is known at training time. As discussed in [30, 58], many anomaly scores  $\mathbf{S}_{\theta}(\mathbf{x}_{\mathcal{B}})$  allow for easily constructing a score  $\mathbf{A}_{\theta}(\mathbf{x}_{\mathcal{B}})$  that behaves inversely. That means, we expect  $\mathbf{A}_{\theta}(\mathbf{x}_{\mathcal{B}})$  to be large when evaluated on normal samples, and small for anomalies. Importantly, both scores share the same parameters. In the context of DSVDD, we define  $\mathbf{S}_{\theta}(\mathbf{x}_{\mathcal{B}}) = 1 / \mathbf{A}_{\theta}(\mathbf{x}_{\mathcal{B}})$ , but other definitions are possible for alternative losses [58, 63, 64]. Using the inverse score, we can construct a supervised AD loss on the meta training set as follows.
We define a binary indicator variable  $y_{j}^{i}$ , indicating whether data point  $i$  is normal or anomalous in the context of distribution  $P_{j}$  (i.e.,  $y_{j}^{i} = 0$  iff  $\mathbf{x}_{\mathcal{B}}^{i} \in P_{j}$ ). We later refer to it as anomaly label. A natural choice for the loss in Eq. (5) is therefore
$$
L \left[ \mathbf {S} _ {\theta} \left(\mathbf {x} _ {\mathcal {B}}\right) \right] = \frac {1}{B} \sum_ {i \in \mathcal {B}} \left\{\left(1 - y ^ {i}\right) \mathbf {S} _ {\theta} ^ {i} \left(\mathbf {x} _ {\mathcal {B}}\right) + y ^ {i} \mathbf {A} _ {\theta} ^ {i} \left(\mathbf {x} _ {\mathcal {B}}\right) \right\}. \tag {7}
$$
The loss function resembles the outlier exposure loss [30], but as opposed to using synthetically generated samples (typically only available for images), we use samples from the complement  $\bar{P}_j$  at training time to synthesize outliers. The training pseudo-code is in Alg. 1 of Supp. C.
In addition to DSVDD, we also study backbone models such as binary classifiers and NTL [56]. For NTL, we adopt the  $\mathbf{S}_{\theta}$  and  $\mathbf{A}_{\theta}$  used by Qiu et al. [58]. For binary classifiers, we set  $\mathbf{S}_{\theta}(\mathbf{x}) = -\log \left(1 - \sigma (f_{\theta}(\mathbf{x}))\right)$  and  $\mathbf{A}_{\theta}(\mathbf{x}) = -\log \sigma (f_{\theta}(\mathbf{x}))$
Batch-level Prediction. After training, we deploy the model in an unseen production environment to detect anomalies in a zero-shot adaptive fashion. Similar to the training set, the distribution will be a mixture of new normal samples  $P_{*}$  and an admixture of anomalies from a distribution never encountered before. For the method to work, we still assume that the majority of samples be normal (Assumption A3). Anomaly scores are assigned based on batches, as during training. For prediction, the anomaly scores are thresholded at a user-specified value.
Time complexity for prediction depends on the network complexity and is constant  $O(1)$  relative to batch size, because the predictions can be trivially parallelized via modern deep learning libraries.
2.4 Theoretical Results
Having described our method, we now establish a theoretical basis for ACR by deriving a bounded generalization error on an unseen test distribution  $P_{*}$ . We define the generalization error in terms of training and testing losses, i.e., we are interested in whether the expected loss generalizes from the meta-training distributions  $P_{1},\dots ,P_{K}$  to an unseen distribution  $P_{*}$ .
To prepare the notations, we split  $S_{\theta}(\mathbf{x})$  into two parts: a feature extractor  $\mathbf{z} = f_{\theta}(\mathbf{x})$  that spans from the input layer to the last batch norm layer and that performs batch normalization, and an anomaly score  $S(\mathbf{z})$  that covers all the remaining layers. We use  $P_{j}^{z}$  to denote the data distribution  $P_{j}$  transformed by the feature extractor  $f_{\theta}$ . We assume that  $P_{j}^{z}$  satisfies  $\mathbb{E}_{\mathbf{z}\sim P_j^z}[\mathbf{z}] = 0$  and  $\mathrm{Var}_{\mathbf{z}\sim P_j^z}[\mathbf{z}] = 1$  for  $j = 1,\dots ,K,*$  because  $f_{\theta}$  ends up with a batch norm layer.
Theorem 2.1. Assume the mini-batches are large enough such that, for batches from each given distribution  $P_{j}$ , the mini-batch means and variances are approximately constant across batches. Furthermore, assume the loss function  $L[S(\mathbf{z})]$  is bounded by  $C$  for any  $\mathbf{z}$ . Let  $\| \cdot \|_{TV}$  denote the total variation. Then, the generalization error is upper bounded by
$$
\left| \mathbb {E} _ {\mathbf {x} _ {\mathcal {B}} \sim P _ {*}} \left[ \frac {1}{B} \sum_ {i = 1} ^ {B} L [ S _ {\theta} ^ {i} (\mathbf {x} _ {\mathcal {B}}) ] \right] - \frac {1}{K} \sum_ {j = 1} ^ {K} \mathbb {E} _ {\mathbf {x} _ {\mathcal {B}} \sim P _ {j}} \left[ \frac {1}{B} \sum_ {i = 1} ^ {B} L [ S _ {\theta} ^ {i} (\mathbf {x} _ {\mathcal {B}}) ] \right] \right| \leq C \left\| P _ {*} ^ {z} - \frac {1}{K} \sum_ {j = 1} ^ {K} P _ {j} ^ {z} \right\| _ {T V}.
$$
The proof is shown in Supp. B. Note that Thm. 2.1 still holds if  $P_{j}$  or  $P_{*}$  are contaminated distributions  $P_{j}^{\pi}$  or  $P_{*}^{\bar{\pi}}$ .
Remark. Thm. 2.1 suggests that the generalization error of the expected loss function is bounded by the total variation distance between  $P_{*}^{z}$  and  $\frac{1}{K}\sum_{j = 1}^{K}P_{j}^{z}$ . While we leave a formal bound of the TV distance to future studies, the following intuition holds: since  $f_{\theta}$  contains batch norm layers, the empirical distributions  $\frac{1}{K}\sum_{j = 1}^{K}P_{j}^{z}$  and  $P_{*}^{z}$  will share the same (zero) mean and (unit) variance. If both distributions are dominated by their first two moments, we can expect the total variation distance to be small, providing an explanation for the approach's favorable generalization performance.
Deep AD. Many recent advances in AD are built on deep learning methods [65] and early strategies used autoencoder [9, 55, 85] or density-based [13, 66] models. Another pioneering stream of research combined one-class classification [70] with deep learning [57, 63]. Many other approaches to deep AD are self-supervised, employing a self-supervised loss function to train the detector and score anomalies [4, 23, 31, 44, 56, 68, 72, 75].
All of these approaches assume that the data distribution will not change too much at test time. However, in many practical scenarios, there will be significant shifts in the abnormal distribution and even the normal distribution. For example, Dragoi et al. [17] observed that existing AD methods fail in detecting anomalies when distribution shifts occur in network intrusion detection. Another line of work in this context requires test-time modeling for the entire test set, e.g., COPOD [47], ECOD [48], and robust autoencoder [85], preventing real-time deployment.