Title: SANFlow: Semantic-Aware Normalizing Flow for Anomaly Detection
Venue: NeurIPS 2023
Accepted: yes
Keywords: Anomaly Detection

Visual Anomaly Detection

Computer Vision

Normalizing Flow

Anomaly Localization
Abstract: Visual anomaly detection, the task of detecting abnormal characteristics in images, is challenging due to the rarity and unpredictability of anomalies. In order to reliably model the distribution of normality and detect anomalies, a few works have attempted to exploit the density estimation ability of normalizing flow (NF). However, previous NF-based methods have relied solely on the capability of NF and forcibly transformed the distribution of all features to a single distribution (e.g., unit normal distribution), when features can have different semantic information and thus follow different distributions. We claim that forcibly learning to transform such diverse distributions to a single distribution with a single network will cause the learning difficulty, limiting the capacity of a network to discriminate normal and abnormal data. As such, we propose to transform the distribution of features at each location of a given image to different distributions. In particular, we train NF to map normal data distribution to distributions with the same mean but different variances at each location of the given image. To enhance the discriminability, we also train NF to map abnormal data distribution to a distribution with a mean that is different from that of normal data, where abnormal data is synthesized with data augmentation. The experimental results outline the effectiveness of the proposed framework in improving the density modeling and thus anomaly detection performance.

SANFlow: Semantic-Aware Normalizing Flow for Anomaly Detection and Localization

Abstract

Visual anomaly detection, the task of detecting abnormal characteristics in images, is challenging due to the rarity and unpredictability of anomalies. In order to reliably model the distribution of normality and detect anomalies, a few works have attempted to exploit the density estimation ability of normalizing flow (NF). However, previous NF-based methods forcibly transform the distribution of all features into a single distribution (e.g., unit normal distribution), even when the features can have locally distinct semantic information and thus follow different distributions. We claim that forcibly learning to transform such diverse distributions to a single distribution with a single network will cause the learning difficulty, thereby limiting the capacity of a network to discriminate between normal and abnormal data. As such, we propose to transform the distribution of features at each location of a given input image to different distributions. Specifically, we train NF to map the feature distributions of normal data to different distributions at each location in the given image. Furthermore, to enhance the discriminability, we also train NF to map the distribution of abnormal data to a distribution significantly different from that of normal data. The experimental results highlight the efficacy of the proposed framework in improving the density modeling and thus anomaly detection performance.

1 Introduction

Abnormal events that deviate significantly from expected and typical characteristics are often unwanted and are referred to as anomalies. The objective of anomaly detection is to identify such abnormal events. As such, anomaly detection is applicable across domains (e.g., video-surveillance, product defect detection, medical image analysis, etc.), where abnormal events can indicate or lead to critical issues. However, abnormal events occur rarely and can appear in various forms that cannot be known in advance, making it infeasible to collect a large amount of abnormal data in many real-world scenarios. The difficulty in collecting abnormal data has prevented anomaly detection from exploiting the recent breakthrough in supervised learning, leaving the field still challenging.

Acknowledging the difficulty of collecting abnormal data, many works have formulated anomaly detection as unsupervised learning, one-class learning, or few-shot learning [4, 5, 45, 49, 8, 47], in which the goal is to accurately model a distribution of normal data. Then, any data that deviates from the learned distribution of normal data is considered as an anomaly. The approaches to unsupervised anomaly detection differ by how they recognize the differences between normal and abnormal data. Reconstruction-based methods learn to reconstruct normal images and thus distinguish input images as abnormal if the reconstruction error is high [1, 42, 12, 35]. Meanwhile, representation-based approaches aim to learn a feature space, where normal images are brought close to each other

Anomaly input

Estimated base distributions

Base distribution of normal data in complex region

Base distribution of normal data in simple region

Base distribution of abnormal region

and abnormal images are placed far away from normal images [11, 8, 36]. Another line of works aims to simulate abnormalities through data augmentation [54, 26]. On the other hand, a recently emerging trend is to employ normalizing flow (NF) for a more reliable estimation of normal data distribution [51, 20, 37].

NF aims to achieve better density estimation by learning a sequence of reversible functions to map a complex distribution of input data into a simple distribution (e.g., normal distribution). The goal of NF aligns with that of anomaly detection in that a reliable density estimation of normal samples leads to the accurate detection of anomalies. However, previous NF-based methods have solely relied on the capability of NF to model complex distributions as simple ones. Although features at different locations, scales and images can follow different distributions, previous works have attempted to map such multi-modal distribution to simply a single normal distribution.

Considering that features at different locations and images may follow different distributions, we propose to adaptively embed the features at each location of each image into different distributions accordingly. We make two contributions in this work: First, we propose a novel NF-based framework that transforms the feature distribution of normal data at each location into Gaussian distributions with zero mean but different variances. The variance at each location is estimated by an external network that is conditioned on image features and semantics. Second, to enhance the detection of anomalies, we train NF to transform abnormal features into a distinct Gaussian distribution with a mean that is distant from the distributions of normal features. By embedding the features to locally different distributions, we facilitate the training of NF. We demonstrate that the estimated variance is lower for simple regions such as the background, while it increases for more complex regions. A description of the proposed approach is shown in Figure 1 and evaluation examples by category are shown in Figure 2.

The proposed framework demonstrates outstanding performance in both anomaly detection and localization. The strong empirical results underline the effectiveness of the proposed method in learning a more reliable density estimation, suggesting the importance of learning different target distributions for abnormal features and different normal features.

2 Related Work

Unsupervised anomaly detection tackles a challenging scenario, in which training images only consist of normal images that are free of anomalies. Under such challenging scenarios, standard supervised learning algorithms fail to work since training data is available for only one class (normal data), therefore often referred to as one-class learning. Methodologies that have emerged to overcome this issue largely differ by how they recognize the differences between normal data and abnormal data [9].

Reconstruction based. Reconstruction based approaches are hinged on the motivation that reconstruction models can reconstruct images well, if images are drawn from the same distribution as training images. Thus, these approaches distinguish images with high reconstruction error as abnormal. Several methods perform reconstruction by using auto-encoder architecture [41, 57, 6, 55, 32, 48, 45, 5] or the generator part of GAN models [1, 10, 31, 40].

GT Mask

Prediction

bottle

cable

capsule

carpet

grid

hazelnut

leather

metal nut

pill

screw

tile

toothbrush

transistor

wood

zipper

Score Map

500

Data augmentation based. A few methods have proposed to generate abnormal images through data augmentation that is based on either cut-out [13], random erasing [43], noise generation [54], or geometric transformations [17, 16, 21]. In particular, CutPaste [26] introduces a new data augmentation to generate more fine-grained and subtle abnormal local patches in normal images. Furthermore, other research areas use synthesized abnormal images to improve performance such as semantic segmentation [23, 19, 7] or image classification [25].

Representation based. Another recent line of research has focused on designing systems that can leverage the strong representation power of pre-trained convolutional neural networks. [11, 36, 8, 49, 46, 56, 34]. Then, anomaly detection is performed by measuring the distances between test image features and normal image features. The methods differ by how the features of normal image are maintained or modeled and how the distance is measured. SPADE [8] employs k-nearest neighbor (k-NN) for normal image features retrieval and feature correspondence for measuring the differences. To mitigate the complexity issue with nearest neighbor search, PatchCore [36] maintains a subset of features via coreset subsampling. Meanwhile, PaDiM [11] models the normal features as a multivariate Gaussian distribution and uses Mahalanobis distance [30] as a distance measure, instead of the k-NN algorithm, to improve the inference complexity.

Normalizing flow based. Recently, a few works have shifted the attention to employing normalizing flow (NF) to better estimate the distribution of the features of normal patches. Through a sequence of learnable and invertible transformations, NF embeds the feature distribution into a simple normal distribution. DifferNet [37] is one of the first to employ NF to transform the distribution of CNN features to a normal distribution, where anomaly detection is performed at image level by measuring the likelihood of samples. Then, CFLOW-AD [20] extends NF-based methodology to pixel-level anomaly detection. FastFlow [51] further improves the performance by utilizing normalizing flow in two-dimensional space, unlike previous one-dimensional NF-based anomaly detection algorithms. In addition, CS-Flow [38] proposed a new kind of NF model that jointly processes multiple feature maps across different scales. However, they forcefully embed different feature distributions of semantic patches to a single normal distribution, despite recent findings in classification that it is beneficial to transform images of different semantics into different distributions [22, 3]. By contrast, we propose to train NF to transform the distributions at different locations to simple, but different distributions to better exploit the density estimation capability of NF by assuming locally varying base distributions.

3 Proposed method

In this section, we describe the overall pipeline of our framework, dubbed Semantic-Aware Normalizing Flow (SANFlow), as depicted in Figure 3. Given an image x , a pre-trained feature extractor f is used to obtain features v i ∼ V i at the i -th position. Then, we employ normalizing flow (NF) g -1 : V i → Z i to embed the feature distribution V i (a.k.a. target distribution) to simple and wellknown distribution Z i (a.k.a. base distribution). To further guide NF to map feature distributions V i into different base distributions Z i upon semantics, we introduce an external network h which allows us to predict statistics of z i for each feature v i . Moreover, similar to [26], we employ data

augmentation to synthesize local anomalies within the input image in order to train NF to embed the distribution of anomaly features into a distribution Z a i that is distinct from the distributions of normal features Z n i .

Then, we modify the standard objective function of NF to account for different base distributions (Section 3.5). Finally, at test time, we use the trained NF to estimate the probability of given features to be normal, the negation of which is used as an anomaly score function to detect anomalies in a given image (Section 3.6).

3.1 Background on normalizing flow

Normalizing flow (NF) [33] aims to learn a function g that transforms the latent variable z , which follows a simple and tractable base distribution Z , into the variable v , which follows a complex target distribution V . To do so, NF formulates g as the composition of invertible functions { g l } L l =1 , i.e., g = g L ◦ g L -1 ◦ · · · ◦ g 1 , where L is the number of invertible functions. Such transformation of a complex distribution into a simple one allows for efficient and exact density estimation, serving as a powerful tool in density estimation. By applying the change of variables theorem in a sequence, the log-likelihood of the target distribution p V ( v ) can be expressed in terms of the log-likelihood of the base distribution p Z ( z ) :

where z l represents the resulting random variable after applying up to the l -th function g l on z . Note that z 0 corresponds to z while z L corresponds to v . For efficient computation of the log-likelihood, functions { g l } L l =1 need to be invertible and have easy-to-compute Jacobian determinant. To meet such requirements, one of popular NF instances, RealNVP [14], formulated each function g l as an affine coupling block. In an affine coupling block, the input is first divided into two equal-dimension parts. Then, each part undergoes affine transformation, whose parameters are generated by a network conditioned on the other part. The transformed parts, in turn, are put back together via concatenation.

Then, the log-likelihood in Equation 1 is maximized (or the negative log-likelihood is minimized) to train the parameters of transformation function. A base distribution Z is commonly chosen as a normal distribution N ( 0 , I ) in previous NF-based anomaly detection algorithms [20, 51, 37], resulting in a loss function (i.e., negative log-likelihood) as follows:

3.2 Synthetic anomaly generation

In unsupervised anomaly detection scenarios, abnormal training images are typically not available. Therefore, we synthesize abnormal images to facilitate training of NF. Abnormal images usually differ from normal images only in local regions, which are semantically or structurally similar to surrounding normal regions. To generate such abnormal data, CutPaste [26] introduces data augmentation technique that disturbs local regions of normal images with semantically similar patches. The extracted patches, in turn, undergo transformations (e.g., random flip, rotate, blur, and color jittering) and replace the region of other normal images of the same category at random location. In this work, we use the CutPaste method to synthesize abnormal images. We modify this method slightly to generate more realistic abnormal images by blurring the borders of extracted patches and applying diverse color jittering values to these patches. Also, we make abnormal patches by taking small rectangular patches like CutPaste or newly used circular patches. Size of patches are random and from a normal image to retain diversity of abnormal regions and use the same ratio of all types of patches and normal data for training. Then, during training, our NF model handles normal and abnormal regions of each input image x differently by assuming locally different base distributions. To achieve this, we introduce a binary mask M as shown in Figure 3 for each synthetic anomaly image x to denote whether each pixel at pixel location ( w,h ) belongs to the normal region ( M w,h = 1 ) or the abnormal region ( M w,h = 0 ).

Feature

Extractor

Normalizing Flow

Linear

ReLU

Concatenate

Permute

Coupling Block

Coupling Block

Linear

...

Coupling Block

Statistics Prediction

Conv + ReLU + BN

Conv + ReLU

Conv + ReLU + BN

Conv + ReLU + BN

Conv + ReLU

Position Embedding Vector

Feature Vector

Binary Mask

-map

-map

Normal Image

Feature Map

Synthetic

Anomaly

Augmentation

3.3 Feature extractor

Recently, features extracted by convolutional neural networks (CNN) pre-trained on ImageNet [39] have been shown to be helpful in anomaly detection, by providing useful semantic information [11, 5, 8, 20, 48]. In particular, several methods have employed a multi-scale feature pyramid [27] to handle anomalies of diverse sizes [20, 48, 8]. This is because features of different scales capture information about regions of corresponding sizes (i.e., effective receptive field) in the image [29]. Motivated by previous findings, we also employ a pre-trained CNN [11, 20, 36] to obtain a K -level feature pyramid. The feature map at the k -th scale is denoted by A k ∈ R C k × H k × W k where C k is the number of channels and H k and W k are the height and width of the feature maps, respectively.

3.4 Semantic-aware normalizing flow

3.4.1 Scale- and spatial-aware normalizing flow

We train NF to map our feature vectors computed by CNN to latents which follow simple base distributions. However, features can follow complex distributions at each scale and spatial location of feature map. Thus, using a single NF model to map such complex features to a single base distribution can be difficult. Aiming to bridge the gap, CFLOW-AD [20] applied two modifications to a standard NF baseline model RealNVP [14]: independent NF models for each scale and conditioning NF on position embedding vector [15, 44], similar to [2]. Inspired by CFLOW-AD, we also employ K independent NF models to handle features across different scales as we have K -level pyramid features and position embedding vector conditioning. Moreover, we train the NF model to map features to latents that follow a spatially varying underlying distribution. This allows NF to consider semantic information, such as feature complexity, for corresponding locations in the image.

To be specific, we create a set of vectors by taking a feature vector v k i at the i -th position (i.e., 2D coordinate) of the k -th scale feature map A k , as done in the previous work [20]. Note that, we omit the scale index k from this point on to avoid cluttered notation. To retain the spatial information, each feature vector v i is concatenated with the corresponding position embedding vector p i ∈ R D , whose values are composed of sinusoidal harmonics and unique to each i -th position [15, 44]. The resulting vector c i = ( v i , p i ) is now used to condition each transformation function g l in NF, making it process position-aware [2]. Then, the log-likelihood of feature distribution becomes:

where Z i is a simple, yet locally varying base distribution corresponding to a feature vector v i .

However, just making NF scale- and spatial-aware is insufficient to handle variations that exist within the same object category. This is because objects are not perfectly aligned across different images of the same category, resulting in images with different semantic information even at the identical spatial location and scale. Thus, in order to enhance the density estimation of our NF model, we propose to embed feature distributions into different base distributions, leveraging the semantic information encoded in the features. Further details on this approach are provided in the following section.

3.4.2 Semantic-aware base distribution

In this study, we instantiate semantic-dependent base distributions as Gaussian distributions with statistics that are estimated based on semantic information of features. Inspired by an algorithm that estimates the statistics of non-i.i.d. noise [52], we employ a lightweight statistics prediction network h to estimate the statistics for the given feature v i . We condition the network, parameterized by θ h , on the semantic features to be able to estimate appropriate statistics of corresponding base distributions. As we will discuss in our experiments, we empirically observe that estimating both mean ( µ i ) and variance ( σ 2 i ) of the base distribution is difficult and that estimating variance ( σ 2 i ) alone is beneficial (Section 4.3). Therefore, we fix µ i to be 0 for normal regions and 1 for abnormal regions. In doing so, we map abnormal features to a base distribution with minimal overlap with that of normal features during training, thereby aiding NF in mapping normal and abnormal regions into distinct distributions. When samples are non-i.i.d., the distribution of sample variance can be assumed to follow the inverse Gamma distribution [52]. Since image pixels and semantic features are non-i.i.d., we formulate the variances of the corresponding base distributions to follow the inverse Gamma distribution as IG ( ·| α, β ) with parameters α and β . We set α to be ( p 2 2 -1) and β to be p 2 ξ 2 where p 2 denotes the area of the corresponding receptive field of size p × p and ξ is the mode of the inverse Gamma distribution [52, 50]. In this work, ξ is a hyperparameter, the value of which is empirically found to be 0 . 1 . Upon the assumption, we train the network h to estimate α i and β i for each v i , resulting in the following regularization loss as:

,

where D KL ( IG ( α i , β i ) | IG ( α, β )) computes a KL divergence between the estimated distribution by the network h and the distribution of variance σ 2 i , which we assume to follow IG ( α, β ) as mentioned above; ψ is a digamma function and Γ is a gamma function. Notably, L KL with α and β acts as regularization to guide the statistics prediction network for stable training. The derivation details and detailed explanations can be found in the supplementary material.

3.5 Loss function

In this section, we delineate the overall objective function which our model is trained to minimize. First, we need to modify the log-likelihood of z i corresponding to feature v i in Equation 3 to account for handling both normal and abnormal features differently:

where the binary indicator m i is set to be 1 when the corresponding location in the binary mask M is 1, and 0 , otherwise. Note that, Z n i represents the corresponding base distribution when v i is a normal feature, while Z a i represents the base distribution when v i is an abnormal feature. In practice, to compute the likelihood at the k -th scale, the binary mask M is resized to match the size of the feature map A k , which has height H k and width W k . The resizing is done using nearest-neighbor interpolation to preserve the binary nature of the mask. In turn, each log-likelihood term can be formulated as follows (detailed derivations can be found in the supplementary material):

and,

Moreover, to further aid NF in distinguishing normal and abnormal features, we can also train NF to perform binary classification using the loss function as follows:

where BCE denotes the conventional binary cross-entropy loss, while s ( z i ) measures the probability of z i being classified as normal (i.e., s ( z i ) = p Z n i ( z i ) p Z n i ( z i )+ p Z a i ( z i ) ). Note that the log-likelihood of z i being abnormal (Equation 7) can be used only during training such that our framework learns to map abnormal features to base distribution Z a that has small overlap with the base distributions of normal features Z n . This allows NF to transform normal and abnormal features to two distinct base distributions, thereby enhancing the discriminability.

Together with L KL from Equation 4, the overall objective function L overall to minimize is given by,

where λ 1 and λ 2 are hyperparameters to adjust the associated weights of L BCE and L KL.

3.6 Anomaly score functions

During inference, given a test image x ∈ R H × W × 3 , we first compute the log-likelihood of each feature v i at each k -th scale via Equation 3, where the log-likelihood of z i is computed with Equation 6. Then, the likelihood map P k ∈ R H k × W k for the k -th scale is obtained by taking exponential of the log-likelihood 6. We obtain the final likelihood map P ∈ R H × W by summing the likelihood map P k from all scales. These likelihood maps are first upsampled to the image resolution R H × W using bilinear interpolation, following the approach described in [20]. Finally, the anomaly score map is computed as the negative of P , resulting in -P .

4 Experiments

In this section, we evaluate our framework to validate its capability in both pixel-level anomaly localization and image-level anomaly detection. We will release our code and data upon acceptance, and more details and results can be found in the supplementary material.

4.1 Experimental settings

Dataset. The experiments are conducted on two commonly used datasets for unsupervised anomaly detection: STC (ShanghaiTech Campus) dataset [28] and MVTec dataset [4]. STC is a video surveillance dataset, which provides static videos of 13 different scenes of 856 × 480 resolution. It contains 274 , 515 frames for training and 42 , 883 frames for evaluation. The training set consists of only normal sequences, while the evaluation set consists of 300 , 308 regular frames and 42 , 883 irregular frames. MVTec is a dataset that consists of images of industrial products categorized into 5 texture categories and 10 object categories. To evaluate the unsupervised anomaly detection performance, the training set includes only defect-free (e.g., normal) images: 3 , 629 normal images are available for training while 1 , 725 normal and abnormal images are used as test set. Among the test images, 1 , 258 images contain defects (e.g., abnormal images). For data augmentation, abnormal patches undergo random vertical, horizontal flip and rotation during training, as described in 3.2. We evaluate and compare algorithms in terms of area under the receiver operating characteristic curve (AUROC) and area under the per-region-overlap curve (AUPRO), as used in [4, 20]. AUPRO scores can be found in the supplementary.

Implementation details. Following other works [4, 20, 36], our models are trained and evaluated separately for each category. Similar to CFLOW-AD [20], we use K = 3 scales for feature pyramid; normalizing flow consists of L = 8 transformation blocks; the dimension of position embedding vector D is 512 ; Adam optimizer with a learning rate of 5e-4 and 80 train epochs for training. The loss weight hyperparameters λ 1 and λ 2 are set to be 1 . 0 and 0 . 2 , respectively. The statistics prediction network h consists of five convolutional layers with a skip connection as illustrated in Figure 3. At last, we use NVIDIA RTX8000 for training and the model shows near real-time performance by running at 13fps.

4.2 Experimental results

We provide the anomaly localization and detection performance on the STC dataset in Table 1 while the results on MVTec dataset are reported in Table 2 and Table 3. All results presented in this study are obtained using the same backbone architecture, WRN-50 [53], while additional results on MVTec are reported for PatchCore [36] and our method with a larger backbone, WRN-101 [53]. The experimental results demonstrate that the performance of our proposed method is comparable with other state-of-the-art methods. In Table 1, compare to other anomaly detection models that are not video targeted, our method shows outstanding performances in both detection and localization.

tile

Input image

(a)

(b)

Input image

(a)

(b)

cable

Plus, Table 2 and Table 3 reports that our proposed method performs better than existing NF-based anomaly detection algorithms (CFLOW-AD [20] and DifferNet [37]) and other anomaly detection algorithms such as AE SSIM [4], γ -VAE+grad [12], PatchSVDD [49], PaDiM [11] and PatchCore-1, 10 [36]. In particular, the results demonstrate the effectiveness of our proposed semantic-aware base distribution mapping approach in enhancing the density estimation performance of the proposed NF model.

4.3 Ablation study

We conduct ablation experiments to evaluate the efficacy of each component of our framework. Specifically, we perform ablation studies on the loss function (Table 4), synthetic anomaly generation (Table 5), and statistics estimated by our statistics prediction network h (Table 6). All ablation experiments are reported with respect to AUROC, using the WRN-50 backbone, and all average AUROC values over all categories are reported for image-wise anomaly detection.

In Table 4, we analyze the efficacy of our proposed loss functions. To do so, we first compare against a baseline NF model trained using the standard negative log-likelihood loss of NF in Equation 2 (Model (1a) ). Compared to Model (1a) , SANFlow brings substantial improvement, thereby validating the effectiveness of all three proposed loss functions ( L NLL in Equation 3 using Equation 5, L KL, and L BCE). We validate the efficacy of each loss function separately by applying only L NLL using Equation 3 and 5 (Model (1b) ) or disabling either L KL (Model (1c) ) or L BCE (Model (1d) ). While each variant leads to performance improvement, using all three proposed loss functions leads to the largest improvement, suggesting that all three proposed loss functions coordinate and play critical roles in our overall framework.

In Table 5, we evaluate the impact of synthetic anomaly augmentation (generation) on the final performance of our framework. Training without anomaly augmentation and thus without any anomaly data results in performance degradation (Model (2a) ). Also, Model (1d) (Table 4) which has only difference at doing anomaly augmentation in training, performs better than Model (2a) . Thus, the ablation result suggests that not only embedding semantic features to different base distributions but training NF with synthetic anomalies is also an important process. For better understanding, Fig-

ure 4 visualizes anomaly score maps based on estimated normal distribution Z n (a) and abnormal distribution Z a (b) for test images. As described in Section 3.6, Figure 4 (a) can be found with Equation 3 and 6 while Figure 4 (b) can be found with Equation 3 and 7. We note that both maps agree on the abnormal region (highlighted with bright colors), validating that training NF to map normal and abnormal features to distant base distributions helps the model discriminate abnormal regions from normal regions.

Table 6 reports ablation results after fixing either mean ( µ i ), variance ( σ 2 i ), or both statistics of base distributions. If µ i is fixed, it is set to be 0 for normal features and 1 for abnormal features. If σ 2 i is fixed, it is empirically set to be 0 . 1 for both normal and abnormal features. The performance is observed to decrease noticeably when µ i is estimated alone (Model (3b) ) or together with σ 2 i (Model (3c) ). On the other hand, estimating only σ 2 i while fixing µ i brings significant performance improvement. The performance degradation from estimating µ i may be due to learning complexity. Furthermore, to enhance the persuasiveness of the mean margin, we conducted additional experiments on the MVTec dataset, comparing performance at margin values of 0.5, 1.0, and 1.5 in WRN-50 backbone. For margin values 0.5, 1.0, and 1.5, the proposing method shows 98.6, 98.7, and 98.1 AUROC for image-wise and 98.4, 98.5, and 98.3 AUROC for pixel-wise. As the margin of 1.0 yields the highest results in both image- and pixel- wise, we could confirm that the mean margin of 1.0 is a persuasive choice.

4.4 Experiments on other datasets

Although the MVTec dataset is a benchmark that is widely used for anomaly detection and localization, there is difficulty in identifying the advantage of the proposing method with it due to performance saturation. Therefore, we provide a performance comparison in the VisA dataset [58]. Our proposed method with a WRN-50 backbone achieves an image-wise AUROC of 93.4, 98.6, and a PRO score of 89.4, while CFLOW-AD [20] and PaDiM [11] achieve 91.5, 59.8, and 89.1, 85.9 of an image-wise AUROC and a PRO score, respectively. The results show that our approach demonstrates competitive performance against other methods, including other flow-based approaches, demonstrating the effectiveness of our framework in greatly improving the density estimation capability of normalizing flow for anomaly detection and localization. Plus, while our approach excels at detecting anomalies within images, its effectiveness in tasks involving image-wise semantic outlier detection, like CutPaste [26] and CFLOW-AD, is limited. Nonetheless, we conducted an additional experiment with CIFAR-10 [24], where our proposed method achieved an AUROC of 80.8, outperforming performances of CutPaste (69.4) and CFLOW-AD (79.32).

5 Conclusion

In this work, we propose to improve the density estimation of normalizing flow (NF) for anomaly detection, by training NF to dynamically embed given feature distribution to different base distributions. In particular, base distributions are Gaussian distribution with statistics estimated by our statistics prediction network. As a result, our NF framework learns to map not only diverse normal features but also abnormal features to corresponding yet different base distributions, enhancing the density estimation capability.

Limitation While SANFlow has the capability of mapping different semantic features to different base distributions, it requires anomaly augmentation to map anomaly features to a base distribution distant from base distributions of normal features for better anomaly detection. However, augmentations require domain knowledge and do not cover new anomalies that may occur. But, we would like to note that our ablation study reveals that our framework brings improvements even without anomaly augmentation. Nonetheless, reducing the dependence on anomaly augmentations in the proposed framework is an interesting future research direction.

Broader Impacts Finding anomalies in product during fabrication is important to ensure the quality of products. Anomaly detection algorithms can automate the process, which can offload the burden from human workers. This will allow human workers to focus on other important aspects of fabrication, improving the overall process of fabrication and thus the quality of products. Furthermore, anomaly detection can be used to frequently check anomalies of objects that may be difficult for humans but crucial to human safety (e.g., tall buildings, bridges, nuclear power plants, etc.).

Acknowledgements This work was supported by Institute of Information & communications Technology Planning & Evaluation (IITP) grant funded by the Korea government(MSIT) (No. RS2023-00220628, Artificial intelligence for prediction of structure-based protein interaction reflecting physicochemical principles, 50%) and Samsung Electronics Co., Ltd, and Samsung Research Funding Center of Samsung Electronics (No. SRFCIT1901- 06, 50%).

References

Samet Akcay, Amir Atapour-Abarghouei, and Toby P Breckon. Ganomaly: Semi-supervised anomaly detection via adversarial training. In Proceedings of the Asian Conference on Computer Vision (ACCV) . Springer, 2018.

Lynton Ardizzone, Carsten Luth, Jakob Kruse, Carsten Rother, and Ullrich Köthe. Guided image generation with conditional invertible neural networks. arXiv preprint arXiv:1907.02392 , 2019.

Lynton Ardizzone, Radek Mackowiak, Carsten Rother, and Ullrich Köthe. Training normalizing flows with the information bottleneck for competitive generative classification. In Advances in Neural Information Processing Systems (NeurIPS) , 2020.

Paul Bergmann, Michael Fauser, David Sattlegger, and Carsten Steger. Mvtec ad-a comprehensive real-world dataset for unsupervised anomaly detection. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , 2019.

Paul Bergmann, Michael Fauser, David Sattlegger, and Carsten Steger. Uninformed students: Student-teacher anomaly detection with discriminative latent embeddings. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , 2020.

Paul Bergmann, Sindy Löwe, Michael Fauser, David Sattlegger, and Carsten Steger. Improving unsupervised defect segmentation by applying structural similarity to autoencoders. arXiv preprint arXiv:1807.02011 , 2018.

Victor Besnier, Andrei Bursuc, David Picard, and Alexandre Briot. Triggering failures: Outof-distribution detection by learning from local adversarial attacks in semantic segmentation. In Proceedings of the IEEE International Conference on Computer Vision (ICCV) , pages 15701-15710, 2021.

Niv Cohen and Yedid Hoshen. Sub-image anomaly detection with deep pyramid correspondences. arXiv preprint arXiv:2005.02357 , 2020.

Yajie Cui, Zhaoxiang Liu, and Shiguo Lian. A survey on unsupervised industrial anomaly detection algorithms. arXiv preprint arXiv:2204.11161 , 2022.

Lucas Deecke, Robert Vandermeulen, Lukas Ruff, Stephan Mandt, and Marius Kloft. Image anomaly detection with generative adversarial networks. In Joint european conference on machine learning and knowledge discovery in databases . Springer, 2018.

Thomas Defard, Aleksandr Setkov, Angelique Loesch, and Romaric Audigier. Padim: A patch distribution modeling framework for anomaly detection and localization. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , pages 475-489, 2021.

David Dehaene, Oriel Frigo, Sébastien Combrexelle, and Pierre Eline. Iterative energy-based projection on a normal data manifold for anomaly localization. In Proceedings of the International Conference on Learning Representations (ICLR) , 2020.

Terrance DeVries and Graham W Taylor. Improved regularization of convolutional neural networks with cutout. arXiv preprint arXiv:1708.04552 , 2017.

Laurent Dinh, Jascha Sohl-Dickstein, and Samy Bengio. Density estimation using real nvp. arXiv preprint arXiv:1605.08803 , 2016.

Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In Proceedings of the International Conference on Learning Representations (ICLR) , 2021.

Spyros Gidaris, Praveer Singh, and Nikos Komodakis. Unsupervised representation learning by predicting image rotations. arXiv preprint arXiv:1803.07728 , 2018.

Izhak Golan and Ran El-Yaniv. Deep anomaly detection using geometric transformations. In Advances in Neural Information Processing Systems (NeurIPS) , volume 31, 2018.

Dong Gong, Lingqiao Liu, Vuong Le, Budhaditya Saha, Moussa Reda Mansour, Svetha Venkatesh, and Anton van den Hengel. Memorizing normality to detect anomaly: Memoryaugmented deep autoencoder for unsupervised anomaly detection. In Proceedings of the IEEE International Conference on Computer Vision (ICCV) , 2019.

Matej Grci´ c., Petra Bevandi´ c., and Siniša Segvi´ c. Dense open-set recognition with synthetic outliers generated by real nvp. In Proceedings of the International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications (VISAPP) , pages 133-143. INSTICC, SciTePress, 2021.

Denis Gudovskiy, Shun Ishizaka, and Kazuki Kozuka. Cflow-ad: Real-time unsupervised anomaly detection with localization via conditional normalizing flows. In Proceedings of the IEEE Winter Conference on Applications of Computer Vision (WACV) , 2022.

Dan Hendrycks, Mantas Mazeika, Saurav Kadavath, and Dawn Song. Using self-supervised learning can improve model robustness and uncertainty. In Advances in Neural Information Processing Systems (NeurIPS) , volume 32, 2019.

Pavel Izmailov, Polina Kirichenko, Marc Finzi, and Andrew Gordon Wilson. Semi-supervised learning with normalizing flows. In International Conference on Machine Learning (ICML) . PMLR, 2020.

Shu Kong and Deva Ramanan. Opengan: Open-set recognition via open data generation. In Proceedings of the IEEE International Conference on Computer Vision (ICCV) , pages 813-822, 2021.

Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.

Kimin Lee, Kibok Lee, Honglak Lee, and Jinwoo Shin. Training confidence-calibrated classifiers for detecting out-of-distribution samples. In Proceedings of the International Conference on Learning Representations (ICLR) , 2018.

Chun-Liang Li, Kihyuk Sohn, Jinsung Yoon, and Tomas Pfister. Cutpaste: Self-supervised learning for anomaly detection and localization. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , 2021.

Tsung-Yi Lin, Piotr Dollar, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie. Feature pyramid networks for object detection. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , 2017.

Wen Liu, Weixin Luo, Dongze Lian, and Shenghua Gao. Future frame prediction for anomaly detection-a new baseline. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , 2018.

Wenjie Luo, Yujia Li, Raquel Urtasun, and Richard Zemel. Understanding the effective receptive field in deep convolutional neural networks. In Advances in Neural Information Processing Systems (NeurIPS) , 2016.

Prasanta Chandra Mahalanobis. On the generalized distance in statistics. Sankhy¯ a: The Indian Journal of Statistics, Series A (2008-) , 80:S1-S7, 2018.

Stanislav Pidhorskyi, Ranya Almohsen, and Gianfranco Doretto. Generative probabilistic novelty detection with adversarial autoencoders. In Advances in Neural Information Processing Systems (NeurIPS) , 2018.

Jonathan Pirnay and Keng Chai. Inpainting transformer for anomaly detection. In International Conference on Image Analysis and Processing , 2022.

Danilo Jimenez Rezende and Shakir Mohamed. Variational inference with normalizing flows. In International Conference on Machine Learning (ICML) , 2015.

Oliver Rippel, Patrick Mertens, and Dorit Merhof. Modeling the distribution of normal data in pre-trained deep features for anomaly detection. In International Conference on Pattern Recognition . IEEE, 2021.

Nicolae-C˘ at˘ alin Ristea, Neelu Madan, Radu Tudor Ionescu, Kamal Nasrollahi, Fahad Shahbaz Khan, Thomas B Moeslund, and Mubarak Shah. Self-supervised predictive convolutional attentive block for anomaly detection. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , 2022.

Karsten Roth, Latha Pemula, Joaquin Zepeda, Bernhard Schölkopf, Thomas Brox, and Peter Gehler. Towards total recall in industrial anomaly detection. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , 2022.

Marco Rudolph, Bastian Wandt, and Bodo Rosenhahn. Same same but differnet: Semisupervised defect detection with normalizing flows. In Proceedings of the IEEE Winter Conference on Applications of Computer Vision (WACV) , 2021.

Marco Rudolph, Tom Wehrbein, Bodo Rosenhahn, and Bastian Wandt. Fully convolutional crossscale-flows for image-based defect detection. In Proceedings of the IEEE Winter Conference on Applications of Computer Vision (WACV) , 2022.

Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. Imagenet large scale visual recognition challenge. International Journal of Computer Vision (IJCV) , 115(3):211-252, 2015.

Mohammad Sabokrou, Mohammad Khalooei, Mahmood Fathy, and Ehsan Adeli. Adversarially learned one-class classifier for novelty detection. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , 2018.

Mayu Sakurada and Takehisa Yairi. Anomaly detection using autoencoders with nonlinear dimensionality reduction. In Proceedings of the MLSDA 2014 2nd workshop on machine learning for sensory data analysis , 2014.

Thomas Schlegl, Philipp Seeböck, Sebastian M Waldstein, Ursula Schmidt-Erfurth, and Georg Langs. Unsupervised anomaly detection with generative adversarial networks to guide marker discovery. In International conference on information processing in medical imaging . Springer, 2017.

Tareq Tayeh, Sulaiman Aburakhia, Ryan Myers, and Abdallah Shami. Distance-based anomaly detection for industrial surfaces using triplet networks. In 2020 11th IEEE Annual Information Technology, Electronics and Mobile Communication Conference (IEMCON) , pages 0372-0377. IEEE, 2020.

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems (NeurIPS) , 2017.

Shashanka Venkataramanan, Kuan-Chuan Peng, Rajat Vikram Singh, and Abhijit Mahalanobis. Attention guided anomaly localization in images. In Proceedings of the European Conference on Computer Vision (ECCV) . Springer, 2020.

Shenzhi Wang, Liwei Wu, Lei Cui, and Yujun Shen. Glancing at the patch: Anomaly localization with global and local feature comparison. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , 2021.

Ze Wang, Yipin Zhou, Rui Wang, Tsung-Yu Lin, Ashish Shah, and Ser Nam Lim. Few-shot fastadaptive anomaly detection. Advances in Neural Information Processing Systems (NeurIPS) , 2022.

Jie Yang, Yong Shi, and Zhiquan Qi. Unsupervised anomaly segmentation via deep feature reconstruction. Neurocomputing , 2021.

Jihun Yi and Sungroh Yoon. Patch svdd: Patch-level svdd for anomaly detection and segmentation. In Proceedings of the Asian Conference on Computer Vision (ACCV) , 2020.

Hongwei Yong, Deyu Meng, Wangmeng Zuo, and Lei Zhang. Robust online matrix factorization for dynamic background subtraction. IEEE Transactions on Pattern Analysis and Machine Intelligence , 40(7):1726-1740, 2018.

Jiawei Yu, Ye Zheng, Xiang Wang, Wei Li, Yushuang Wu, Rui Zhao, and Liwei Wu. Fastflow: Unsupervised anomaly detection and localization via 2d normalizing flows. arXiv preprint arXiv:2111.07677 , 2021.

Zongsheng Yue, Hongwei Yong, Qian Zhao, Lei Zhang, and Deyu Meng. Variational denoising network: Toward blind noise modeling and removal. In Advances in Neural Information Processing Systems (NeurIPS) , 2019.

Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. In Proceedings of the British Machine Vision Conference (BMVC) , 2016.

Vitjan Zavrtanik, Matej Kristan, and Danijel Skoˇ caj. Draem - a discriminatively trained reconstruction embedding for surface anomaly detection. In Proceedings of the IEEE International Conference on Computer Vision (ICCV) , 2021.

Vitjan Zavrtanik, Matej Kristan, and Danijel Skoˇ caj. Reconstruction by inpainting for visual anomaly detection. Pattern Recognition , 2021.

Ye Zheng, Xiang Wang, Rui Deng, Tianpeng Bao, Rui Zhao, and Liwei Wu. Focus your distribution: Coarse-to-fine non-contrastive learning for anomaly detection and localization. In International Conference on Multimedia and Expo (ICME) , 2022.

Bo Zong, Qi Song, Martin Renqiang Min, Wei Cheng, Cristian Lumezanu, Daeki Cho, and Haifeng Chen. Deep autoencoding gaussian mixture model for unsupervised anomaly detection. In Proceedings of the International Conference on Learning Representations (ICLR) , 2018.

Yang Zou, Jongheon Jeong, Latha Pemula, Dongqing Zhang, and Onkar Dabeer. Spot-thedifference self-supervised pre-training for anomaly detection and segmentation. In Proceedings of the European Conference on Computer Vision (ECCV) , pages 392-408. Springer, 2022.

A Loss Function Derivation

In this section, we show the detailed derivations for Equation 4 in Section 3.4.2 and Equation 6 and 7 in Section 3.5.

A.1 KL-Divergence

We present the detailed derivations of our KL divergence loss L KL, where IG ( α i , β i ) and IG ( α, β ) represent the probability density function of the inverse Gamma distribution with their corresponding parameters α i , β i and α, β . Furthermore, s i and s denote random variables that follow the corresponding distributions IG ( α i , β i ) and IG ( α, β ) , respectively. The detailed formulations are as follows:

A.2 Log-likelihood Derivation

To calculate the semantic-aware variance σ i for log-likelihood computation, we approximate it using the mode value ( σ i ) from the inverse Gamma distribution IG ( α i , β i ) . This approximation is employed for computational efficiency, following the approach used in VDNet [52]. Therefore, in Equation S2 and Equation S3, we use h ( σ 2 i | x ) that predicts statistics of semantic-aware variances (i.e., σ i ) from input image x as:

and

B Implementation Details

B.1 Hyperparameters

As described in the main manuscript, our overall framework comprises a pre-trained feature extractor f , a normalizing flow g , and a statistics prediction network h . For the feature extraction process, we utilize a multi-scale feature pyramid approach. Specifically, we extract features from three different scales ( K = 3 ) using a pre-trained feature extractor. Regarding the normalizing flow model, we employ L = 8 transformation blocks for each scale.

Similar to VDNet [52], our statistics prediction network takes an image as input and consists of five convolutional layers and a skip-connection. Each convolutional layer comprises 64 filters wit a size of 3 × 3 except for the last layer. In particular, the last layer of the statistics prediction network produces a two-channel output: one for α and the other for β .

During training phase on the MVTec dataset [4], we fixed the learning rate to 5e-4. Moreover, we empirically determined the hyperparameters λ 1 and λ 2 . Specifically, we selected the values for λ 1 and λ 2 from the set { 0 . 1 , 0 . 2 , 0 . 3 , 0 . 5 , 1 . 0 , 2 . 0 } based on their impact on the validation results, and we use λ 1 = 1 . 0 and λ 2 = 0 . 2 .

As for STC (ShanghaiTech Campus) dataset [28], all training details and hyperparameters are the same, except for a learning rate that is set to be (2 e -4 ) .

B.2 Data Augmentation

Small-sized rectangular shaped mask: 2 ≤ H ≤ 32 , 10 ≤ W ≤ 50

Medium-sized rectangular shaped mask: 16 ≤ H ≤ 64 , 64 ≤ W ≤ 256

Circular shaped mask: 10 ≤ radius ≤ 25

To generate anomaly data, we first extracted patches based on the masks described above. These masks corresponded to small and medium-sized rectangular shapes, as well as circular shapes. Subsequently, the extracted patches underwent distortions to simulate anomalies, where we randomly perform color jittering between 0 and 0.1 (brightness, contrast, saturation, and hue). Additionally, we introduce random horizontal and vertical flips to further augment the dataset. These patches are then applied at the random locations of normal images from the same category. Figures S1 and S2 present categorical examples of synthetic anomaly images, with anomalies generated using rectangular patches in Figure S1 and circular patches in Figure S2.

We constructed a batch of size 8 by including an equal number of normal images and three types of abnormal images, with each type corresponding to a specific mask shape. This ensured a balanced representation of normal and abnormal samples within the batch, allowing for effective training of our model.

bottle

cable

capsule

carpet

grid

hazelnut

leather

metal nut

pill

screw

tile

toothbrush

transistor

wood

zipper

Text

500

bottle

cable

capsule

carpet

grid

hazelnut

leather

metal nut

pill

screw

tile

toothbrush

transistor

wood

zipper

Text

500

C Additional Experimental Results

C.1 Qualitative Results

In Figure S3, we give additional qualitative results of anomaly score maps similar to Figure 4 in the main manuscript. Specifically, we compare against Model (2a) from Table 5, which is a variant of our model that lacks anomaly augmentation. Figure S3 illustrates that even without anomaly augmentation, Model (2a) is still able to localize anomaly regions, highlighting that the major effectiveness of our framework does not come from anomaly augmentations but from the capability to transform the distribution of features at each location of a given input image to different distributions. Regardless, the figure shows that our final model gives more accurate and clear anomaly score maps, owing to anomaly augmentation.

C.2 Quantitative Results

In this section, we provide additional and detailed quantitative experimental results for Table 4, 5, and 6 in the main manuscript. In particular, we provide the performance of each category for Table 4, 5, and 6 in the corresponding tables: Table S2, S3, and S4, respectively. In Table S5 and S6, we provide additional ablation results, which are obtained by applying the proposed semantic-aware normalizing flow (NF) schemes to other NF-based methods: CFLOW-AD [20] and FastFlow [51]. The consistent performance improvement across CFLOW-AD and FastFlow demonstrates the flexibility and effectiveness of our proposed semantic-aware normalizing flow (NF) schemes.

C.3 Anomaly Score Function

Unlike the loss function, we do not use s ( z i ) value in Equation 8 for computing anomaly score. However, s ( z i ) maybe helpful for defining better anomaly score function. To be sure that s ( z i ) is not helpful, we provide visualization examples Figure S4 and S5. Figure S4 represents the results during training, while Figure S5 demonstrates the results during test. As it can be observed in Figure S5,

during inference, there are instances where it tends to segment only localized regions or produce high scores in unrelated areas. This may be due to discrepancies between abnormal augmentation used during training and actual anomalies during test. Since, s ( z i ) is computed based on both negative log-likelihood (NLL) of normal and abnormal features, it is necessary for the model to project normal and abnormal features precisely. However, Figure S5 illustrates that the projection of abnormal features is not as accurate as as normal features, showing unsatisfactory results with s ( z i ) as a anomaly scoring function. Therefore, it does not exhibit suitability comparable to NLL for anomaly scoring. In the future, we have plans to utilize s ( z i ) for performance enhancement.

C.4 Additional Analysis for Table 6

While estimating both mean and variance to give higher performance due to more flexibility gives a high expectation for the performances, we observe that giving too much flexibility and increasing training difficulty, thereby resulting in performance degradation as observed in Table 6 of the main paper and Figure S6. To better support the claim, we allow the model to estimate both mean and variance (i.e., Model (3c) ) and plot the distribution of estimated mean by utilizing histogram in Figure S6. Graphs demonstrates that there are significant degree of overlap between normal (blue) and abnormal (orange) distributions, when considering variance as well.

Γ7 ΓUΓXΓWΓK

ΓSΓLΓOΓO

Γ7 ΓHΓ♭ΓWΓXΓU

ΓH

Γ2ΓEΓMΓHΓFΓW

ΓZΓRΓRΓG

ΓFΓDΓUΓSΓHΓW

ΓJΓUΓLΓG

ΓPΓHΓWΓDΓOΓΛΓQΓXΓW

ΓVΓFΓU

HΓZ

Γ2ΓXΓUΓV

Γ0ΓRΓGΓHΓOΓΛΓαΓλΓDΓβ

Γ⇀ΓU

ΓRΓXΓQΓGΓΛ

GT

s(z)

prediction

500

500

福

GT

s(z)

prediction

A

彩

GT

score map

prediction

normal mean

abnormal mean

bottle

cable

zipper

bottle

cable

zipper

20000

17500

15000

12500

10000

7500

5000

2500

0.750.500.250.00

0.250.50

0.75

1.00

1.25

6000

5000

4000

3000

2000

1000

0.5

00

0.5

1.0

1.5

17500

15000

12500

10000

7500

5000

2500

0.500.250.00

0.75

1.001.25

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe manuscript proposes a new method for dense anomaly detection in industrial applications. The proposed method first extracts dense features with a frozen pretrained model. Then, a normalizing flow learns the distribution of extracted features at every spatial location. While the flow parameters are shared across the spatial locations, the parameters of flow prior are independently computed by an additional network h conditioned on the input image. In particular, the network h outputs variances of flow priors while means remain fixed. During the training, the input samples are augmented with synthetic negatives. The optimization objective is formed of the likelihood maximization for inlier features,  regularization of network h outputs, and discrimination between inliers and synthetic negatives. During the inference, the method detects anomalies according to the aggregated likelihood of features across different resolutions. The developed method achieves competitive results on two considered benchmarks.\n\nSTRENGTHS:\nS1. The manuscript is well-written and easy to follow.\n\nS2. Modelling different normalizing flow prior at every spatial location appears to be novel.\n\nS3. The proposed method achieves performance gains over related works.\n\nWEAKNESSES:\nW1. The difference between CFlow-AD [19] and the proposed SANFlow i) is modelling independent flow priors for each spatial location and ii) mapping anomalous and non-anomalous features to flow priors with different means. These contributions may be rather incremental.\n\nW2. Intuition on why estimating both mean and variance is inferior to estimating variance only (L197) should be comprehensively elaborated beyond L323-328. Especially since predicting parameters of flow prior is the main contribution of the manuscript.\n\nW3. Missing related works which augment datasets with synthetic negatives [a,b,c,d].\n\n[a] Shu Kong, Deva Ramanan: OpenGAN: Open-Set Recognition via Open Data Generation. ICCV 2021.\n\n[b] Matej Grcic, Petra Bevandic, Sinisa Segvic:\nDense Open-set Recognition with Synthetic Outliers Generated by Real NVP.  VISAPP 2021.\n\n[c] Victor Besnier, Andrei Bursuc, David Picard, Alexandre Briot: Triggering Failures: Out-Of-Distribution detection by learning from local adversarial attacks in Semantic Segmentation. ICCV 2021.\n\n[d] Kimin Lee, Honglak Lee, Kibok Lee, Jinwoo Shin:\nTraining Confidence-calibrated Classifiers for Detecting Out-of-Distribution Samples. ICLR 2018\n\nQUESTIONS:\nAdding a colorbar to Fig. 4 might improve clarity.\n\nLIMITATIONS:\nThe manuscript adequately addressed the limitations of the method.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-01"}

comments: SUMMARY:
This paper presents a novel NF-based framework for anomaly detection (AD) and anomaly localization (AL) task. The proposed Semantic-Aware Normalizing Flow (SANFlow) framework map the distributions of normal data to different distributions at each location in the given image instead of mapping a distribution of whole image and makes distributions of abnormal data distant from normal data. The proposed method achieves the state-of-the-art in both AD and AL.

STRENGTHS:
[1. The proposed SANFlow model addresses the problem of learning difficulty in anomaly detection, which is conventional in NF-based methods. This innovation makes full use of semantic information and enhances the accuracy of anomaly detection.
2. This paper provides a detailed description of the proposed SANFlow model. The methodology is well-developed and supported by theoretical analysis.
3. The experimental results on the Mvtec-AD demonstrate the superiority of SANFlow.

WEAKNESSES:
1. This paper lacks the implementation details of the multi-scale FN, making it difficult to reproduce the study. And no code is provided to reproduce the proposed method.
2. This paper needs ablation studies about the margin between normal and abnormal means.
3. More experiments should be conducted. The paper lacks experiments on other standard AD datasets like CIFAR10.

QUESTIONS:
1.	Can the authors provide more details on the implementation of the SANFlow model, including the multi-scale FN?
2.	Can you explain why no code implementation is provided, since open-source is necessary for the community?
3.	Have the authors considered comparing the performance of the proposed model on other datasets, such as CIFAR10?

LIMITATIONS:
Yes.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-05

comments: SUMMARY:
This paper propose a NF-based framework to model the normal feature distribution as Gaussian distributions with zero mean but different variances, and the abnormal features as distinct Gaussian distribution. The proposed model demonstrates relatively good performances in both anomaly detection and localization.

STRENGTHS:
The writing of this paper is clear and concise. It's easy to follow and understand each component of the proposed model. The motivation to model various distributions according to different regoins are reasonable.

WEAKNESSES:
1. The novalty of this paper seems to be increamental. It's based on / combanition of the reference papers [34] [22] [19] [48].
2. The incorporation of semantic information seems to be naive. All the pixels have the same base distributions following the inverse
Gamma distribution as IG(·|α, β).  I wonder if the semantic information can be incorparated into the corresponding base distributions (The same regoin share the same base distribution).
3.  I don't understand why you fix µi to be 0 for normal regions and 1 for abnormal regions. As the normal pixels have various characteristics, maybe corresponding to various µi, at least fot different regoins.
4.  s(zi) maybe helpful for computing anomaly score. I'd like to see the correlations between s(zi)  and  GT location, which can reflect the gap between simulated training  anomalies and real testing anomalies.

QUESTIONS:
Is the performance gain of 0.1% brought by anomaly augmentation? It seems the simulated anomalies are meaningless.

LIMITATIONS:
Yes
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-05

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents an enhanced flow-based anomaly detection method by utilizing density estimation on normalizing flow (NF). The proposed approach involves predicting the inverse Gamma distribution using a statistics prediction network, which facilitates dynamic training of normalized flow distribution. Additionally, the training process benefits from the introduction of anomaly feature distribution and incorporates data augmentation techniques to enhance performance. By learning to map diverse normal features and abnormal features to distinct base distributions, the proposed NF framework enhances its capability for density estimation. The experimental results show that the improved density modeling achieved by the proposed framework, resulting in enhanced anomaly detection performance.\n\nSTRENGTHS:\n+ In contrast to previous flow-based anomaly detection methods that transform the distribution of all features into a single distribution, this work introduces a statistical prediction network to guide the training of different semantic features according to their respective distributions.\n+ By incorporating the data augmentation, both normal and anomaly distribution are predicted and employed to derive to the anomaly score. The qualitive results demonstrate that the localization performance of the proposed method is improved by data augmentation.\n\nWEAKNESSES:\n- The paper does not provide the detection performance of the PRO metric. The flow-based method tends to produce oversized defect regions, particularly for small-sized defects. Therefore, it is crucial to evaluate the location performance using the PRO metric.\n- For image based anomaly detection, several recent AD methods achieves promising the image level and pixel AUROC on MVTEC AD, which make it difficult to identify the advantage while comparison. Therefore, it is necessary to compare the performance on other datasets, e.g. VisA. In VisA, images exhibit complex structures, objects placed in sporadic locations, and different objects, which could differentiate the detection performance of different methods.\n- The paper utilizes a self-defined data augmentation method to synthesize anomalies. However, several existing data augmentation methods, such as Draem, nsa, and cutpaste, have been proposed for anomaly detection. It is important to consider whether the choice of data augmentation method impacts the detection performance.\n\nQUESTIONS:\nPlease see the weaknesses section. The comprehensive results on image-based anomaly detection are expected.\n\nLIMITATIONS:\nAuthors have addressed the limitations of the method.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}