SlowGuess/NeurIPS_2023_OCR huggingface dataset:
SANFlow: Semantic-Aware Normalizing Flow for Anomaly Detection and Localization
Daehyun Kim $^{1}$  Sungyong Baik $^{2}$  Tae Hyun Kim $^{3*}$
Dept. of Artificial Intelligence<sup>1</sup>, Dept. of Data Science<sup>2</sup>, Dept. of Computer Science<sup>3</sup>
{daehyun, dsybaik, taehyunkim}@hanyang.ac.kr
Visual anomaly detection, the task of detecting abnormal characteristics in images, is challenging due to the rarity and unpredictability of anomalies. In order to reliably model the distribution of normality and detect anomalies, a few works have attempted to exploit the density estimation ability of normalizing flow (NF). However, previous NF-based methods forcibly transform the distribution of all features into a single distribution (e.g., unit normal distribution), even when the features can have locally distinct semantic information and thus follow different distributions. We claim that forcibly learning to transform such diverse distributions to a single distribution with a single network will cause the learning difficulty, thereby limiting the capacity of a network to discriminate between normal and abnormal data. As such, we propose to transform the distribution of features at each location of a given input image to different distributions. Specifically, we train NF to map the feature distributions of normal data to different distributions at each location in the given image. Furthermore, to enhance the discriminability, we also train NF to map the distribution of abnormal data to a distribution significantly different from that of normal data. The experimental results highlight the efficacy of the proposed framework in improving the density modeling and thus anomaly detection performance.
Abnormal events that deviate significantly from expected and typical characteristics are often unwanted and are referred to as anomalies. The objective of anomaly detection is to identify such abnormal events. As such, anomaly detection is applicable across domains (e.g., video-surveillance, product defect detection, medical image analysis, etc.), where abnormal events can indicate or lead to critical issues. However, abnormal events occur rarely and can appear in various forms that cannot be known in advance, making it infeasible to collect a large amount of abnormal data in many real-world scenarios. The difficulty in collecting abnormal data has prevented anomaly detection from exploiting the recent breakthrough in supervised learning, leaving the field still challenging.
Acknowledging the difficulty of collecting abnormal data, many works have formulated anomaly detection as unsupervised learning, one-class learning, or few-shot learning [4, 5, 45, 49, 8, 47], in which the goal is to accurately model a distribution of normal data. Then, any data that deviates from the learned distribution of normal data is considered as an anomaly. The approaches to unsupervised anomaly detection differ by how they recognize the differences between normal and abnormal data. Reconstruction-based methods learn to reconstruct normal images and thus distinguish input images as abnormal if the reconstruction error is high [1, 42, 12, 35]. Meanwhile, representation-based approaches aim to learn a feature space, where normal images are brought close to each other
*Correspondence to: Tae Hyun Kim <taehyunkim@hanyang.ac.kr>.
Anomaly input
Estimated base distributions
Base distribution of normal data in complex region
Base distribution of normal data in simple region
Base distribution of abnormal region
Figure 1: Motivation Overview: Different semantic features can be observed in different regions. Abnormal regions (red box) exhibit significantly distinct features compared to normal features (orange and blue boxes). Moreover, normal regions also display different semantic features based on the complexity of the region. Upon our observation, we propose Semantic-Aware Normalizing Flow (SANFlow) that enables more accurate density estimation by adaptively embedding semantic feature distribution into corresponding but different base distributions.
and abnormal images are placed far away from normal images [11, 8, 36]. Another line of works aims to simulate abnormalities through data augmentation [54, 26]. On the other hand, a recently emerging trend is to employ normalizing flow (NF) for a more reliable estimation of normal data distribution [51, 20, 37].
NF aims to achieve better density estimation by learning a sequence of reversible functions to map a complex distribution of input data into a simple distribution (e.g., normal distribution). The goal of NF aligns with that of anomaly detection in that a reliable density estimation of normal samples leads to the accurate detection of anomalies. However, previous NF-based methods have solely relied on the capability of NF to model complex distributions as simple ones. Although features at different locations, scales and images can follow different distributions, previous works have attempted to map such multi-modal distribution to simply a single normal distribution.
Considering that features at different locations and images may follow different distributions, we propose to adaptively embed the features at each location of each image into different distributions accordingly. We make two contributions in this work: First, we propose a novel NF-based framework that transforms the feature distribution of normal data at each location into Gaussian distributions with zero mean but different variances. The variance at each location is estimated by an external network that is conditioned on image features and semantics. Second, to enhance the detection of anomalies, we train NF to transform abnormal features into a distinct Gaussian distribution with a mean that is distant from the distributions of normal features. By embedding the features to locally different distributions, we facilitate the training of NF. We demonstrate that the estimated variance is lower for simple regions such as the background, while it increases for more complex regions. A description of the proposed approach is shown in Figure 1 and evaluation examples by category are shown in Figure 2.
The proposed framework demonstrates outstanding performance in both anomaly detection and localization. The strong empirical results underline the effectiveness of the proposed method in learning a more reliable density estimation, suggesting the importance of learning different target distributions for abnormal features and different normal features.
Unsupervised anomaly detection tackles a challenging scenario, in which training images only consist of normal images that are free of anomalies. Under such challenging scenarios, standard supervised learning algorithms fail to work since training data is available for only one class (normal data), therefore often referred to as one-class learning. Methodologies that have emerged to overcome this issue largely differ by how they recognize the differences between normal data and abnormal data [9].
Reconstruction based. Reconstruction based approaches are hinged on the motivation that reconstruction models can reconstruct images well, if images are drawn from the same distribution as training images. Thus, these approaches distinguish images with high reconstruction error as abnormal. Several methods perform reconstruction by using auto-encoder architecture [41, 57, 6, 55, 32, 48, 45, 5] or the generator part of GAN models [1, 10, 31, 40].
Figure 2: Anomaly localization results by our proposed framework on MVTec benchmark dataset. From top to bottom: anomaly localization prediction, GT mask, anomaly score map and estimated variance map of input image. Variance maps are derived from mode values of  $\alpha$  and  $\beta$  of each pixel and bright areas represent high variance and dark areas represent low variance values for corresponding distribution.
Data augmentation based. A few methods have proposed to generate abnormal images through data augmentation that is based on either cut-out [13], random erasing [43], noise generation [54], or geometric transformations [17, 16, 21]. In particular, CutPaste [26] introduces a new data augmentation to generate more fine-grained and subtle abnormal local patches in normal images. Furthermore, other research areas use synthesized abnormal images to improve performance such as semantic segmentation [23, 19, 7] or image classification [25].
Representation based. Another recent line of research has focused on designing systems that can leverage the strong representation power of pre-trained convolutional neural networks. [11, 36, 8, 49, 46, 56, 34]. Then, anomaly detection is performed by measuring the distances between test image features and normal image features. The methods differ by how the features of normal image are maintained or modeled and how the distance is measured. SPADE [8] employs k-nearest neighbor (k-NN) for normal image features retrieval and feature correspondence for measuring the differences. To mitigate the complexity issue with nearest neighbor search, PatchCore [36] maintains a subset of features via coreset subsampling. Meanwhile, PaDiM [11] models the normal features as a multivariate Gaussian distribution and uses Mahalanobis distance [30] as a distance measure, instead of the k-NN algorithm, to improve the inference complexity.
Normalizing flow based. Recently, a few works have shifted the attention to employing normalizing flow (NF) to better estimate the distribution of the features of normal patches. Through a sequence of learnable and invertible transformations, NF embeds the feature distribution into a simple normal distribution. DifferNet [37] is one of the first to employ NF to transform the distribution of CNN features to a normal distribution, where anomaly detection is performed at image level by measuring the likelihood of samples. Then, CFLOW-AD [20] extends NF-based methodology to pixel-level anomaly detection. FastFlow [51] further improves the performance by utilizing normalizing flow in two-dimensional space, unlike previous one-dimensional NF-based anomaly detection algorithms. In addition, CS-Flow [38] proposed a new kind of NF model that jointly processes multiple feature maps across different scales. However, they forcefully embed different feature distributions of semantic patches to a single normal distribution, despite recent findings in classification that it is beneficial to transform images of different semantics into different distributions [22, 3]. By contrast, we propose to train NF to transform the distributions at different locations to simple, but different distributions to better exploit the density estimation capability of NF by assuming locally varying base distributions.
In this section, we describe the overall pipeline of our framework, dubbed Semantic-Aware Normalizing Flow (SANFlow), as depicted in Figure 3. Given an image  $\pmb{x}$ , a pre-trained feature extractor  $f$  is used to obtain features  $\pmb{v}_i \sim V_i$  at the  $i$ -th position. Then, we employ normalizing flow (NF)  $g^{-1}: V_i \to Z_i$  to embed the feature distribution  $V_i$  (a.k.a. target distribution) to simple and well-known distribution  $Z_i$  (a.k.a. base distribution). To further guide NF to map feature distributions  $V_i$  into different base distributions  $Z_i$  upon semantics, we introduce an external network  $h$  which allows us to predict statistics of  $z_i$  for each feature  $v_i$ . Moreover, similar to [26], we employ data
augmentation to synthesize local anomalies within the input image in order to train NF to embed the distribution of anomaly features into a distribution  $Z_{i}^{a}$  that is distinct from the distributions of normal features  $Z_{i}^{n}$ .
Then, we modify the standard objective function of NF to account for different base distributions (Section 3.5). Finally, at test time, we use the trained NF to estimate the probability of given features to be normal, the negation of which is used as an anomaly score function to detect anomalies in a given image (Section 3.6).
3.1 Background on normalizing flow
Normalizing flow (NF) [33] aims to learn a function  $g$  that transforms the latent variable  $z$ , which follows a simple and tractable base distribution  $Z$ , into the variable  $v$ , which follows a complex target distribution  $V$ . To do so, NF formulates  $g$  as the composition of invertible functions  $\{g^l\}_{l=1}^L$ , i.e.,  $g = g^L \circ g^{L-1} \circ \dots \circ g^1$ , where  $L$  is the number of invertible functions. Such transformation of a complex distribution into a simple one allows for efficient and exact density estimation, serving as a powerful tool in density estimation. By applying the change of variables theorem in a sequence, the log-likelihood of the target distribution  $p_V(v)$  can be expressed in terms of the log-likelihood of the base distribution  $p_Z(z)$ :
$$
\log p _ {V} (\boldsymbol {v}) = \log p _ {Z} (\boldsymbol {z}) - \sum_ {l = 1} ^ {L} \log \left| \det  \frac {d g ^ {l}}{d \boldsymbol {z} ^ {l - 1}} \right|, \tag {1}
$$
where  $z^l$  represents the resulting random variable after applying up to the  $l$ -th function  $g^l$  on  $z$ . Note that  $z^0$  corresponds to  $z$  while  $z^L$  corresponds to  $v$ . For efficient computation of the log-likelihood, functions  $\{g^l\}_{l=1}^L$  need to be invertible and have easy-to-compute Jacobian determinant. To meet such requirements, one of popular NF instances, RealNVP [14], formulated each function  $g^l$  as an affine coupling block. In an affine coupling block, the input is first divided into two equal-dimension parts. Then, each part undergoes affine transformation, whose parameters are generated by a network conditioned on the other part. The transformed parts, in turn, are put back together via concatenation.
Then, the log-likelihood in Equation 1 is maximized (or the negative log-likelihood is minimized) to train the parameters of transformation function. A base distribution  $Z$  is commonly chosen as a normal distribution  $\mathcal{N}(\mathbf{0},\mathbf{I})$  in previous NF-based anomaly detection algorithms [20, 51, 37], resulting in a loss function (i.e., negative log-likelihood) as follows:
$$
\mathcal {L} _ {\mathrm {N L L}} = \frac {| | \boldsymbol {z} | | _ {2} ^ {2}}{2} + \sum_ {l = 1} ^ {L} \log \left| \det  \frac {d g ^ {l}}{d z ^ {l - 1}} \right|. \tag {2}
$$
3.2 Synthetic anomaly generation
In unsupervised anomaly detection scenarios, abnormal training images are typically not available. Therefore, we synthesize abnormal images to facilitate training of NF. Abnormal images usually differ from normal images only in local regions, which are semantically or structurally similar to surrounding normal regions. To generate such abnormal data, CutPaste [26] introduces data augmentation technique that disturbs local regions of normal images with semantically similar patches. The extracted patches, in turn, undergo transformations (e.g., random flip, rotate, blur, and color jittering) and replace the region of other normal images of the same category at random location. In this work, we use the CutPaste method to synthesize abnormal images. We modify this method slightly to generate more realistic abnormal images by blurring the borders of extracted patches and applying diverse color jittering values to these patches. Also, we make abnormal patches by taking small rectangular patches like CutPaste or newly used circular patches. Size of patches are random and from a normal image to retain diversity of abnormal regions and use the same ratio of all types of patches and normal data for training. Then, during training, our NF model handles normal and abnormal regions of each input image  $\pmb{x}$  differently by assuming locally different base distributions. To achieve this, we introduce a binary mask  $M$  as shown in Figure 3 for each synthetic anomaly image  $\pmb{x}$  to denote whether each pixel at pixel location  $(w,h)$  belongs to the normal region  $(M_{w,h} = 1)$  or the abnormal region  $(M_{w,h} = 0)$ .
Figure 3: The overview of our proposed SANFlow framework. Firstly, with synthetic augmentation process, make binary mask  $M$  and anomaly image  $x$  from normal image. Then, a pre-trained feature extractor  $f$  is applied to extract multi-scale feature maps ( $K = 3$ , number of scales in this work) for  $x$ . For each  $i$ -th position at feature map of  $k$ -th level, each feature vector  $v_{i}^{k}$  is concatenated with a position embedding vector  $p_{i}^{k}$  before being fed into NF, which is independently trained for each scale. To be specific, NF consists of 8 coupling blocks (i.e.,  $L = 8$ ), which maps given feature distributions to base distributions. Each base distribution is different in terms of statistics (variance in this work) which are dependent on semantic features of the corresponding location in the given image. In particular, we condition our statistics prediction network  $h$  on the input image to generate parameters of inverse Gamma distribution such as  $\alpha$  and  $\beta$ , to estimate the variances of base distributions, which are assumed to follow inverse Gamma distributions.
3.3 Feature extractor
Recently, features extracted by convolutional neural networks (CNN) pre-trained on ImageNet [39] have been shown to be helpful in anomaly detection, by providing useful semantic information [11, 5, 8, 20, 48]. In particular, several methods have employed a multi-scale feature pyramid [27] to handle anomalies of diverse sizes [20, 48, 8]. This is because features of different scales capture information about regions of corresponding sizes (i.e., effective receptive field) in the image [29]. Motivated by previous findings, we also employ a pre-trained CNN [11, 20, 36] to obtain a  $K$ -level feature pyramid. The feature map at the  $k$ -th scale is denoted by  $\mathbf{A}^k \in \mathbb{R}^{C_k \times H_k \times W_k}$  where  $C_k$  is the number of channels and  $H_k$  and  $W_k$  are the height and width of the feature maps, respectively.
3.4 Semantic-aware normalizing flow
3.4.1 Scale- and spatial-aware normalizing flow
We train NF to map our feature vectors computed by CNN to latents which follow simple base distributions. However, features can follow complex distributions at each scale and spatial location of feature map. Thus, using a single NF model to map such complex features to a single base distribution can be difficult. Aiming to bridge the gap, CFLOW-AD [20] applied two modifications to a standard NF baseline model RealNVP [14]: independent NF models for each scale and conditioning NF on position embedding vector [15, 44], similar to [2]. Inspired by CFLOW-AD, we also employ  $K$  independent NF models to handle features across different scales as we have  $K$ -level pyramid features and position embedding vector conditioning. Moreover, we train the NF model to map features to latents that follow a spatially varying underlying distribution. This allows NF to consider semantic information, such as feature complexity, for corresponding locations in the image.
To be specific, we create a set of vectors by taking a feature vector  $\pmb{v}_i^k$  at the  $i$ -th position (i.e., 2D coordinate) of the  $k$ -th scale feature map  $\pmb{A}^k$ , as done in the previous work [20]. Note that, we omit the scale index  $k$  from this point on to avoid cluttered notation. To retain the spatial information, each feature vector  $\pmb{v}_i$  is concatenated with the corresponding position embedding vector  $\pmb{p}_i \in \mathbb{R}^D$ , whose values are composed of sinusoidal harmonics and unique to each  $i$ -th position [15, 44]. The resulting vector  $\pmb{c}_i = (\pmb{v}_i, \pmb{p}_i)$  is now used to condition each transformation function  $g^l$  in NF, making it process position-aware [2]. Then, the log-likelihood of feature distribution becomes:
$$
\log p _ {V _ {i}} \left(\boldsymbol {v} _ {i}\right) = \log p _ {Z _ {i}} \left(\boldsymbol {z} _ {i}\right) - \sum_ {l = 1} ^ {L} \log \left| \det  \frac {d g ^ {l}}{d \boldsymbol {z} _ {i} ^ {l - 1}} \right|, \tag {3}
$$
where  $Z_{i}$  is a simple, yet locally varying base distribution corresponding to a feature vector  $\pmb{v}_{i}$ .
However, just making NF scale- and spatial-aware is insufficient to handle variations that exist within the same object category. This is because objects are not perfectly aligned across different images of the same category, resulting in images with different semantic information even at the identical spatial location and scale. Thus, in order to enhance the density estimation of our NF model, we propose to embed feature distributions into different base distributions, leveraging the semantic information encoded in the features. Further details on this approach are provided in the following section.
3.4.2 Semantic-aware base distribution
In this study, we instantiate semantic-dependent base distributions as Gaussian distributions with statistics that are estimated based on semantic information of features. Inspired by an algorithm that estimates the statistics of non-i.i.d. noise [52], we employ a lightweight statistics prediction network  $h$  to estimate the statistics for the given feature  $v_{i}$ . We condition the network, parameterized by  $\theta_{h}$ , on the semantic features to be able to estimate appropriate statistics of corresponding base distributions. As we will discuss in our experiments, we empirically observe that estimating both mean  $(\mu_{i})$  and variance  $(\sigma_{i}^{2})$  of the base distribution is difficult and that estimating variance  $(\sigma_{i}^{2})$  alone is beneficial (Section 4.3). Therefore, we fix  $\mu_{i}$  to be 0 for normal regions and 1 for abnormal regions. In doing so, we map abnormal features to a base distribution with minimal overlap with that of normal features during training, thereby aiding NF in mapping normal and abnormal regions into distinct distributions. When samples are non-i.i.d., the distribution of sample variance can be assumed to follow the inverse Gamma distribution [52]. Since image pixels and semantic features are non-i.i.d., we formulate the variances of the corresponding base distributions to follow the inverse Gamma distribution as  $\mathrm{IG}(\cdot |\alpha ,\beta)$  with parameters  $\alpha$  and  $\beta$ . We set  $\alpha$  to be  $\left(\frac{p^2}{2} -1\right)$  and  $\beta$  to be  $\frac{p^2\xi}{2}$ , where  $p^2$  denotes the area of the corresponding receptive field of size  $p\times p$  and  $\xi$  is the mode of the inverse Gamma distribution [52, 50]. In this work,  $\xi$  is a hyperparameter, the value of which is empirically found to be 0.1. Upon the assumption, we train the network  $h$  to estimate  $\alpha_{i}$  and  $\beta_{i}$  for each  $v_{i}$ , resulting in the following regularization loss as:
$$
\begin{array}{l} \mathcal {L} _ {K L} = D _ {K L} \left(I G \left(\alpha_ {i}, \beta_ {i}\right) \mid I G (\alpha , \beta)\right) \\ = \sum_ {i = 1} ^ {H _ {k} \times W _ {k}} \left\{\left(\alpha_ {i} - \alpha\right) \psi \left(\alpha_ {i}\right) + \left(\log \Gamma (\alpha) - \log \Gamma \left(\alpha_ {i}\right)\right) + \alpha \left(\log \beta_ {i} - \log \beta\right) + \alpha_ {i} \left(\frac {\beta}{\beta_ {i}} - 1\right) \right\}, \tag {4} \\ \end{array}
$$
where  $D_{KL}(IG(\alpha_i,\beta_i)\mid IG(\alpha ,\beta))$  computes a KL divergence between the estimated distribution by the network  $h$  and the distribution of variance  $\sigma_i^2$ , which we assume to follow  $IG(\alpha ,\beta)$  as mentioned above;  $\psi$  is a digamma function and  $\Gamma$  is a gamma function. Notably,  $\mathcal{L}_{KL}$  with  $\alpha$  and  $\beta$  acts as regularization to guide the statistics prediction network for stable training. The derivation details and detailed explanations can be found in the supplementary material.
3.5 Loss function
In this section, we delineate the overall objective function which our model is trained to minimize. First, we need to modify the log-likelihood of  $\mathbf{z}_i$  corresponding to feature  $\mathbf{v}_i$  in Equation 3 to account for handling both normal and abnormal features differently: