SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} \log p _ {Z _ {i} ^ {a}} \left(\boldsymbol {z} _ {i}\right) = E _ {h \left(\sigma_ {i} ^ {2} \mid \boldsymbol {x}\right)} \left[ \log p \left(\boldsymbol {z} _ {i} \mid 1, \sigma_ {i} ^ {2}\right) \right] \\ = \int h \left(\sigma_ {i} ^ {2} | \boldsymbol {x}\right) \log p \left(\boldsymbol {z} _ {i} \mid 1, \sigma_ {i} ^ {2}\right) d \sigma^ {2} \\ = \int h (\sigma_ {i} ^ {2} | \pmb {x}) \{- \frac {1}{2} \log 2 \pi - \frac {1}{2} \log \sigma_ {i} ^ {2} - \frac {| | \pmb {z} _ {i} - 1 | | _ {2} ^ {2}}{2 \sigma_ {i} ^ {2}} \} d \sigma^ {2} \\ = - \frac {1}{2} \log 2 \pi - \frac {1}{2} \int h \left(\sigma_ {i} ^ {2} | \boldsymbol {x}\right) \log \sigma_ {i} ^ {2} d \sigma_ {i} ^ {2} - \frac {1}{2} | | \boldsymbol {z} _ {i} - 1 | | _ {2} ^ {2} \int h \left(\sigma_ {i} ^ {2} | \boldsymbol {x}\right) \frac {1}{\sigma_ {i} ^ {2}} d \sigma_ {i} ^ {2} \tag {S3} \\ = - \frac {1}{2} \log 2 \pi - \frac {1}{2} E [ \log \sigma_ {i} ^ {2} ] - \frac {1}{2} | | \boldsymbol {z} _ {i} - 1 | | _ {2} ^ {2} E [ \frac {1}{\sigma_ {i} ^ {2}} ] \\ = - \frac {1}{2} \log 2 \pi - \frac {1}{2} \left(\log \beta_ {i} - \psi \left(\alpha_ {i}\right)\right) - \frac {\alpha_ {i}}{2 \beta_ {i}} | | \boldsymbol {z} _ {i} - 1 | | _ {2} ^ {2}. \\ \end{array}
$$
As described in the main manuscript, our overall framework comprises a pre-trained feature extractor  $f$ , a normalizing flow  $g$ , and a statistics prediction network  $h$ . For the feature extraction process, we utilize a multi-scale feature pyramid approach. Specifically, we extract features from three different scales ( $K = 3$ ) using a pre-trained feature extractor. Regarding the normalizing flow model, we employ  $L = 8$  transformation blocks for each scale.
Similar to VDNet [52], our statistics prediction network takes an image as input and consists of five convolutional layers and a skip-connection. Each convolutional layer comprises 64 filters wit a size of  $3 \times 3$  except for the last layer. In particular, the last layer of the statistics prediction network produces a two-channel output: one for  $\alpha$  and the other for  $\beta$ .
During training phase on the MVtec dataset [4], we fixed the learning rate to 5e-4. Moreover, we empirically determined the hyperparameters  $\lambda_{1}$  and  $\lambda_{2}$ . Specifically, we selected the values for  $\lambda_{1}$  and  $\lambda_{2}$  from the set  $\{0.1, 0.2, 0.3, 0.5, 1.0, 2.0\}$  based on their impact on the validation results, and we use  $\lambda_{1} = 1.0$  and  $\lambda_{2} = 0.2$ .
As for STC (ShanghaiTech Campus) dataset [28], all training details and hyperparameters are the same, except for a learning rate that is set to be  $(2e^{-4})$ .
B.2 Data Augmentation
- Small-sized rectangular shaped mask:  $2 \leq H \leq 32$ ,  $10 \leq W \leq 50$
Medium-sized rectangular shaped mask:  $16\leq H\leq 64$ $64\leq W\leq 256$
- Circular shaped mask:  $10 \leq radius \leq 25$
To generate anomaly data, we first extracted patches based on the masks described above. These masks corresponded to small and medium-sized rectangular shapes, as well as circular shapes. Subsequently, the extracted patches underwent distortions to simulate anomalies, where we randomly perform color jittering between 0 and 0.1 (brightness, contrast, saturation, and hue). Additionally, we introduce random horizontal and vertical flips to further augment the dataset. These patches are then applied at the random locations of normal images from the same category. Figures S1 and S2 present categorical examples of synthetic anomaly images, with anomalies generated using rectangular patches in Figure S1 and circular patches in Figure S2.
We constructed a batch of size 8 by including an equal number of normal images and three types of abnormal images, with each type corresponding to a specific mask shape. This ensured a balanced representation of normal and abnormal samples within the batch, allowing for effective training of our model.
Figure S1: Synthetic anomaly images generated with rectangular shaped patches.
Figure S2: Synthetic anomaly images generated with circular shaped patches.
C.1 Qualitative Results
In Figure S3, we give additional qualitative results of anomaly score maps similar to Figure 4 in the main manuscript. Specifically, we compare against Model (2a) from Table 5, which is a variant of our model that lacks anomaly augmentation. Figure S3 illustrates that even without anomaly augmentation, Model (2a) is still able to localize anomaly regions, highlighting that the major effectiveness of our framework does not come from anomaly augmentations but from the capability to transform the distribution of features at each location of a given input image to different distributions. Regardless, the figure shows that our final model gives more accurate and clear anomaly score maps, owing to anomaly augmentation.
C.2 Quantitative Results
In this section, we provide additional and detailed quantitative experimental results for Table 4, 5, and 6 in the main manuscript. In particular, we provide the performance of each category for Table 4, 5, and 6 in the corresponding tables: Table S2, S3, and S4, respectively. In Table S5 and S6, we provide additional ablation results, which are obtained by applying the proposed semantic-aware normalizing flow (NF) schemes to other NF-based methods: CFLOW-AD [20] and FastFlow [51]. The consistent performance improvement across CFLOW-AD and FastFlow demonstrates the flexibility and effectiveness of our proposed semantic-aware normalizing flow (NF) schemes.
C.3 Anomaly Score Function
Unlike the loss function, we do not use  $s(z_{i})$  value in Equation 8 for computing anomaly score. However,  $s(z_{i})$  may be helpful for defining better anomaly score function. To be sure that  $s(z_{i})$  is not helpful, we provide visualization examples Figure S4 and S5. Figure S4 represents the results during training, while Figure S5 demonstrates the results during test. As it can be observed in Figure S5,
during inference, there are instances where it tends to segment only localized regions or produce high scores in unrelated areas. This may be due to discrepancies between abnormal augmentation used during training and actual anomalies during test. Since,  $s(z_{i})$  is computed based on both negative log-likelihood (NLL) of normal and abnormal features, it is necessary for the model to project normal and abnormal features precisely. However, Figure S5 illustrates that the projection of abnormal features is not as accurate as as normal features, showing unsatisfactory results with  $s(z_{i})$  as a anomaly scoring function. Therefore, it does not exhibit suitability comparable to NLL for anomaly scoring. In the future, we have plans to utilize  $s(z_{i})$  for performance enhancement.
C.4 Additional Analysis for Table 6
While estimating both mean and variance to give higher performance due to more flexibility gives a high expectation for the performances, we observe that giving too much flexibility and increasing training difficulty, thereby resulting in performance degradation as observed in Table 6 of the main paper and Figure S6. To better support the claim, we allow the model to estimate both mean and variance (i.e., Model (3c)) and plot the distribution of estimated mean by utilizing histogram in Figure S6. Graphs demonstrate that there are significant degree of overlap between normal (blue) and abnormal (orange) distributions, when considering variance as well.
Figure S3: Visualization of anomaly score maps produced by our proposed framework and Model (2a) on texture and object categories of MVtec benchmark dataset. From top to bottom: GT mask, anomaly score map of proposed model and anomaly score map of Model (2a).
Figure S4: Ground truth, score map, and prediction for training data when using  $s(z)$  as the anomaly score.
Figure S5: Ground truth, score map, and prediction for test data when using  $s(z)$  as the anomaly score.
Table S1: Results by category in WRN-101 backbone.
Category	bottle	cable	capsule	carpet	grid	hazelnut	leather	metal_nut	pill	screw	tile	toothbrush	transistor	wood	zipper	average
Image-Wise	100	99.7	98.9	99.9	100	100	100	100	98.2	96.2	100	100	99.4	99.3	100	99.4
Pixel-Wise	99.1	98.8	98.9	99.4	99.3	99.0	99.8	98.7	99.1	99.2	99.1	99.2	95.1	97.9	99.6	98.8
Pro-Score	96.6	98.7	96.9	98.0	97.5	97.3	99.3	96.3	97.6	95.8	97.4	94.2	80.6	96.3	98.3	96.1
prnneep
normalean
aannnnnne
bottle
Figure S6: Results of Model (3c) (model that estimates both mean and variance), including ground truth, score map, prediction, normal mean estimated map, abnormal mean estimated map and histogram. For histogram plots, each displays estimated mean distribution of normal and abnormal features for each category: bottle, cable, and zipper. Orange part of histogram represents estimated abnormal means and blue part of histogram represents estimated normal means.
Table S2: Results by category in Table 4.
Category	bottle	cable	capsule	carpet	grid	hazelnut	leather	metal_nut	pill	screw	tile	toothbrush	transistor	wood	zipper	average
Image-Wise	(1b)	100	97.7	97.4	99.8	99.6	100	100	99.3	96.0	90.5	100	98.3	98.2	97.4	99.8	98.2
(1c)	100	96.7	98.2	99.7	99.5	100	100	99.5	96.6	89.0	100	97.8	98.8	98.6	99.8	98.3
(1d)	100	98.2	97.5	99.8	99.7	100	100	99.5	96.2	92.8	100	95.3	99.1	98.9	99.9	98.5
SANFlow	100	98.7	97.7	99.8	99.3	100	100	99.8	96.8	94.0	100	96.7	99.3	99.1	99.9	98.7
Pixel-Wise	(1b)	98.6	97.9	98.9	99.4	98.4	99.1	99.5	98.0	99.1	98.8	98.9	98.9	94.4	96.1	98.9	98.3
(1c)	98.8	97.1	99.0	99.4	98.5	99.2	99.5	98.3	99.1	98.9	99.0	99.0	93.6	96.5	98.5	98.3
(1d)	98.6	98.3	99.0	99.3	98.6	99.2	99.6	98.3	99.1	98.9	99.0	99.0	94.3	96.0	99.1	98.4
SANFlow	98.6	98.5	99.1	99.3	98.5	99.2	99.6	98.5	99.2	99.0	98.9	98.9	94.4	96.4	98.9	98.5
Pro-Score	(1b)	92.8	95.8	92.9	96.7	94.6	96.3	98.2	87.8	94.7	94.4	93.9	87.5	88.1	90.9	95.5	93.3
(1c)	94.1	94.8	93.9	96.5	94.2	96.6	98.5	91.7	95.3	94.9	94.4	89.2	87.8	91.2	94.6	93.8
(1d)	93.1	96.5	93.9	96.5	94.6	96.8	98.5	91.9	94.9	94.7	94.7	89.3	88.4	91.4	96.2	94.1
SANFlow	92.7	96.7	94.1	96.4	94.3	96.8	98.3	92.8	95.6	95.3	94.2	87.9	87.6	91.1	95.7	94.0
Table S3: Results by category in Table 5.
Category	bottle	cable	capsule	carpet	grid	hazelnut	leather	metal_nut	pill	screw	tile	toothbrush	transistor	wood	zipper	average
Image-Wise	(2a)	100	97.5	97.2	99.4	99.2	100	100	98.9	97.4	93.2	99.9	98.0	96.6	99.0	99.1	98.4
SANFlow	100	98.7	97.7	99.8	99.3	100	100	99.8	96.8	94.0	100	96.7	99.3	99.1	99.9	98.7
Pixel-Wise	(2a)	98.8	97.7	99.0	99.3	98.1	99.0	99.5	98.2	99.0	98.8	98.0	98.6	95.6	95.9	98.4	98.2
SANFlow	98.6	98.5	99.1	99.3	98.5	99.2	99.6	98.5	99.2	99.0	98.9	98.9	94.4	96.4	98.9	98.5
Pro-Score	(2a)	94.1	92.6	93.5	96.5	93.0	96.1	98.8	92.0	94.3	94.3	92.3	87.6	85.3	93.1	92.9	93.1
SANFlow	92.7	96.7	94.1	96.4	94.3	96.8	98.3	92.8	95.6	95.3	94.2	87.9	87.6	91.1	95.7	94.0
Table S4: Results by category in Table 6.
Category	bottle	cable	capsule	carpet	grid	hazelnut	leather	metal_nut	pill	screw	tile	toothbrush	transistor	wood	zipper	average
Image-Wise	(3a)	100	98.2	97.4	99.3	100	100	100	99.3	95.3	87.1	100	95.8	94.4	99.5	98.0	97.6
(3b)	100	94.9	98.6	99.4	86.9	100	100	99.2	96.0	89.4	100	97.5	95.1	99.7	98.0	97.0
(3c)	100	98.0	96.6	99.7	99.9	100	100	99.8	95.4	89.3	99.9	95.6	98.6	99.0	99.9	98.1
SANFlow	100	98.7	97.7	99.8	99.3	100	100	99.8	96.8	94.0	100	96.7	99.3	99.1	99.9	98.7
Pixel-Wise	(3a)	98.9	97.1	99.0	99.1	97.2	99.2	99.5	98.3	98.7	98.5	98.2	99.0	93.2	96.2	98.3	98.1
(3b)	98.9	97.1	99.0	99.2	96.2	99.1	99.5	98.1	99.0	98.6	97.9	99.0	92.7	95.9	98.0	97.9
(3c)	98.6	98.0	98.9	99.3	98.6	99.1	99.5	98.4	99.0	98.8	99.0	99.0	93.1	96.0	99.0	98.3
SANFlow	98.6	98.5	99.1	99.3	98.5	99.2	99.6	98.5	99.2	99.0	98.9	98.9	94.4	96.4	98.9	98.5
Pro-Score	(3a)	94.8	92.6	94.3	95.4	88.4	96.8	98.5	92.8	95.4	93.9	91.9	90.8	80.9	92.1	91.2	92.7
(3b)	95.2	90.7	93.2	95.4	91.8	96.0	97.8	90.6	95.1	92.6	93.1	87.7	86.2	90.8	93.9	92.7
(3c)	93.3	96.0	93.5	96.5	94.6	97.0	97.9	90.9	94.5	95.1	92.9	90.6	84.6	91.2	96.2	93.7
SANFlow	92.7	96.7	94.1	96.4	94.3	96.8	98.3	92.8	95.6	95.3	94.2	87.9	87.6	91.1	95.7	94.0
Table S5: Comparison for other normalizing flow models with statistics prediction (SP  $\checkmark$ ) and without statistics prediction(SP  $\times$ ). Experiments are conducted with the same backbone network WRN-50 on MVtec and results are reported with respect to AUROC metric for two scenarios: image-wise anomaly detection and pixel-wise anomaly localization. For convenience, we represent statistic prediction as SP and NF based methods as NF in this table. Since there is no official code for FastFlow [51], we conduct experiments with a third party code (https://github.com/openvinotoolkit/anomalib).
NF	SP	bottle	cable	capsule	carpet	grid	hazelnut	leather	metal_nut	pill	screw	tile	toothbrush	transistor	wood	zipper	average
Image-Wise	CFLOW-AD	X	100	97.59	97.68	98.73	99.60	99.98	100	99.26	96.82	91.89	99.88	99.65	95.21	99.12	98.48	98.26
✓	100	97.63	97.17	99.44	99.21	100	100	98.92	97.38	93.20	99.93	98.03	96.62	99.04	99.11	98.37
FastFlow	X	100	96.2	96.3	99.4	100	99.4	99.9	99.5	94.2	83.9	100	83.6	97.9	99.2	95.1	96.3
✓	100	96.6	93.6	99.0	100	97.7	100	99.6	94.1	92.4	100	99.9	94.4	96.5	100	97.6
Pixel-Wise	CFLOW-AD	X	98.76	97.64	98.98	99.23	96.89	98.82	99.61	98.56	98.95	98.10	97.71	98.56	93.28	94.49	98.41	97.87
✓	98.75	97.73	98.99	99.28	98.10	98.99	99.54	98.21	98.97	98.81	98.02	98.62	95.55	95.86	98.44	98.28
FastFlow	X	98.6	97.2	99.0	99.1	99.2	98.0	99.6	98.8	97.6	96.6	96.6	98.0	97.1	94.1	98.5	97.9
✓	98.7	97.4	99.0	99.3	99.1	98.4	99.6	98.5	99.4	98.0	98.9	98.5	95.1	97.6	98.9	98.4
Table S6: Detection and localization performances on STC dataset. CFLOW-AD+SP represents the results of CFLOW-AD with statistic prediction (SP) in Table S5.
(1b)	(1c)	(1d)	(2a)	(3a)	(3b)	(3c)	CFLOW-AD	CFLOW-AD+SP	SANFlow
Image-Wise	73.5	73.1	74.2	73.6	75.9	75.1	76.1	72.63	73.6	76.1
Pixel-Wise	91.5	91.5	94.6	94.5	94.6	94.1	93.7	94.48	94.5	94.8