SlowGuess/NeurIPS_2023_OCR huggingface dataset:
An effective strategy is to utilize autoencoders with varying latent space dimensionalities  $D_{\mathbf{z}}$ . For high-dimensional data, such as images,  $\mathcal{M}$  with different  $D_{\mathbf{z}}$  tends to capture distinct modes of variation in data. As depicted in Fig. 4, a perturbation on  $\mathcal{M}$  with small  $D_{\mathbf{z}}$  corresponds to low-frequency variation in  $\mathcal{X}$ , whereas for  $\mathcal{M}$  with large  $D_{\mathbf{z}}$ , it corresponds to higher-frequency variation. Using multiple  $D_{\mathbf{z}}$ 's in MPDR gives us more diverse  $\mathbf{x}^{-}$  and eventually better anomaly detection performance.
3.5 Energy Function Design
MPDR is a versatile training algorithm for general EBMs, compatible with various types of energy functions. The design of an energy function plays a crucial role in anomaly detection performance, as the inductive bias of an energy governs its behavior in out-of-distribution regions. We primarily explore two designs for energy functions: MPDR-Scalar (MPDR-S), a feed-forward neural network that takes input  $\mathbf{x}$  and produces a scalar output, and MPDR-Reconstruction (MPDR-R), the reconstruction error from an autoencoder,  $E_{\theta}(\mathbf{x}) = ||\mathbf{x} - g_d(g_e(\mathbf{x}))||^2$ , for an encoder  $g_{e}$  and a decoder  $g_{d}$ . The autoencoder  $(g_{e},g_{d})$  is separate from the autoencoder  $(f_{e},f_{d})$  used in MPD. First proposed in [24], a reconstruction-based energy function has an inductive bias towards assigning high energy values to off-manifold regions (Fig. 3). Training such energy functions using conventional techniques like CD [15] or persistent chains [13, 11] is reported to be challenging [24]. However, MPDR effectively trains both scalar and reconstruction energies. Additionally, in Sec. 4.4, we demonstrate that MPDR is also compatible with an energy function based on a masked autoencoder.
4.1 Implementation of MPDR
An autoencoder  $(f_{e},f_{d})$  is trained by minimizing the reconstruction error of the training data and remains fixed during the training of  $E_{\theta}(\mathbf{x})$ . When using an ensemble of manifolds, each autoencoder is trained independently. For anomaly detection, the energy value  $E_{\theta}(\mathbf{x})$  serves as an anomaly score which assigns a high value for anomalous  $\mathbf{x}$ . All optimizations are performed using Adam with a learning rate of 0.0001. Each run is executed on a single Tesla
Table 2: MNIST OOD detection performance measured in AUPR. We test models from hold-out digit 9 experiment (Table 1). The overall performance is high, as detecting these outliers is easier than identifying the hold-out digit.
KMNIST	EMNIST	Omniglot	FashionMNIST	Constant
AE	0.999	0.977	0.947	1.000	0.954
IGEBM	0.990	0.923	0.845	0.996	1.000
NAE	1.000	0.993	0.997	1.000	1.000
MPDR-S	0.999	0.995	0.994	0.999	1.000
MPDR-R	0.999	0.989	0.997	0.999	0.990
V100 GPU. Other details, including network architectures and LMC hyperparameters, can be found in the Appendix.
Spherical Latent Space In all our implementations of autoencoders, we utilize a hyperspherical latent space  $\mathcal{Z} = \mathbb{S}^{D_{\mathbf{z}} - 1}$  [28, 29, 30]. The encoder output is projected onto  $\mathbb{S}^{D_{\mathbf{z}} - 1}$  via division by its norm before being fed into the decoder. Employing  $\mathbb{S}^{D_{\mathbf{z}} - 1}$  standardizes the length scale of
$\mathcal{Z}$ , allowing us to use the same value of  $\sigma$  across various data sets and autoencoder architectures. Meanwhile, the impact of  $\mathbb{S}^{D_{\mathbf{z}} - 1}$  on the reconstruction error is minimal.
Regularization For a scalar energy function, MPDR-S, we add  $L_{reg} = E_{\theta}(\mathbf{x})^{2} + E_{\theta}(\mathbf{x}^{-})^{2}$  to the loss function, as proposed by [11]. For a reconstruction energy function, MPDR-R, we add  $L_{reg} = E_{\theta}(\mathbf{x}^{-})^{2}$ , following [24].
Scaling Perturbation Probability Applying regularization to an energy restricts its scale, causing a mismatch in scales between the two terms in the recovery likelihood (Eq. (7)). To remedy this mismatch, we heuristically introduce a scale factor  $\gamma < 1$  to  $\log p(\mathbf{z}|\tilde{\mathbf{z}})$ , resulting in the modified recovery energy  $\tilde{E}_{\theta}^{(\gamma)}(\mathbf{x}|\tilde{\mathbf{z}}) = E_{\theta}(\mathbf{x}) + \frac{\gamma}{2\sigma^2} ||\tilde{\mathbf{z}} -f_e(\mathbf{x})||^2$ . We use  $\gamma = 0.0001$  for all experiments.
4.2 2D Density Estimation
We show MPDR's ability to estimate multi-modal density using a mixture of eight circularly arranged 2D Gaussians (Fig. 2). We construct an autoencoder with  $S^1$  latent space, which approximately captures the circular arrangement. The encoder and the decoder are MLPs with two layers of 128 hidden neurons. To show the impact of the design of energy functions, we implement both scalar energy and reconstruction energy. Three-hidden-layer MLPs are used for the scalar energy function, and the encoder and the decoder in the reconstruction energy function. Note that the network architecture of the reconstruction energy is not the same as the autoencoder used for MPD. The density estimation results are presented in Fig. 3. We quantify density estimation performance using  $l_1$  error. After numerically normalizing the energy function and true density on the visualized bounded domain, we compute the  $l_1$  error at 10,000 grid points. While both energies capture the overall landscape of the density, the reconstruction energy achieves a smaller error by suppressing probability density at off-manifold regions.
4.3 Image Out-of-Distribution Detection
MNIST Hold-Out Digit Detection Following the protocol of [26, 27], we evaluated the performance of MPDR on MNIST hold-out digit detection benchmark, where one of the ten digits in the MNIST dataset is considered anomalous, and the remaining digits are treated as in-distribution. This is a challenging task due to the diversity of the in-distribution data and a high degree of similarity between target anomalies and inliers. In particular, selecting digits 1, 4, 5, 7, and 9 as anomalous is known to be especially difficult. The results are shown in Table 1.
In MPDR, we use a single autoencoder  $(f_e, f_d)$  with  $D_{\mathbf{z}} = 32$ . The energy function of MPDR-S is initialized from scratch, and the energy function of MPDR-R is initialized from the  $(f_e, f_d)$  used in MPD. Even without a manifold ensemble, MPDR shows significant improvement over existing algorithms, including ones leveraging an autoencoder in EBM training [19, 24]. The performance of MPDR is stable over a range of  $D_{\mathbf{z}}$ , as demonstrated in Appendix B.1.
MNIST OOD Detection To ensure that MPDR is not overfitted to the hold-out digit, we test MPDR in detecting five non-MNIST outlier datasets (Table 2). The results demonstrated that MPDR excels in detecting a wide range of outliers, surpassing the performance of naive algorithms such as autoencoders (AE) and scalar EBMs (IGEBM). Although MPDR achieves high overall detection performance, MPDR-R exhibits slightly weaker performance on EMNIST and Constant datasets. This can be attributed to the limited flexibility of the autoencoder-based energy function employed in MPDR-R.
CIFAR-10 OOD Detection We evaluate MPDR on the CIFAR-10 inliers, a standard benchmark for EBM-based OOD detection. The manifold ensemble includes three convolutional autoencoders, with  $D_{\mathbf{z}} = 32,64,128$ . MPDR-S uses a ResNet energy function used in IGEBM [11]. MPDR-R adopts the ResNet-based autoencoder architecture used in NAE [24].
Table 3 compares MPDR to state-of-the-art EBMs. MPDR-R shows competitive performance across five OOD datasets, while MPDR-S also achieves high AUROC on SVHN and Constant. As both MPDR-R and NAE use the same autoencoder architecture for the energy, the discrepancy in performance can be attributed to the MPDR training algorithm. MPDR-R outperforms NAE on four out of five OOD datasets. Comparison between MPDR-S and DRL demonstrates the effectiveness of non-Gaussian manifold-aware perturbation used in MPDR. CLEL shows strong
Table 3: OOD detection with CIFAR-10 as in-distribution. AUROC values are presented. The largest value in the column is marked as boldface, and the second and the third largest values are underlined. Asterisks denote that the results are adopted from literature.
SVHN	Textures	Constant	CIFAR100	CelebA
PixelCNN++ [31]*	0.32	0.33	0.71	0.63	-
GLOW [8]*	0.24	0.27	-	0.55	0.57
IGEBM [11]*	0.63	0.48	0.39	-	-
NVAE [21]	0.4402	0.4554	0.6478	0.4943	0.6804
VAEBM [18]*	0.83	-	-	0.62	0.77
JEM [32]*	0.67	0.60	-	0.67	0.75
Improved CD [14]	0.7843	0.7275	0.8000	0.5298	0.5399
NAE [24]	0.9352	0.7472	0.9793	0.6316	0.8735
DRL [16]	0.8816	0.4465	0.9884	0.4377	0.6398
CLEL [33]*	0.9848	0.9437	-	0.7161	0.7717
MPDR-S (ours)	0.9860	0.6583	0.9996	0.5576	0.7313
MPDR-R (ours)	0.9807	0.7978	0.9996	0.6354	0.8282
MPDR-R Single AE
Dz=32	0.8271	0.6606	0.9877	0.5835	0.7751
Dz=64	0.9330	0.6631	0.9489	0.6223	0.8272
Dz=128	0.9886	0.6942	0.9651	0.6531	0.8500
Table 4: OOD detection on pretrained ViT-B_16 representation with CIFAR-100 as in-distribution. Performance is measured in AUROC.
CIFAR10	SVHN	CelebA
Supervised
MD [34]	0.8634	0.9638	0.8833
RMD [35]	0.9159	0.9685	0.4971
Unsupervised
AE	0.8580	0.9645	0.8103
NAE [24]	0.8041	0.9082	0.8181
IGEBM [11]	0.8217	0.9584	0.9004
DRL [16]	0.5730	0.6340	0.7293
MPDR-S (ours)	0.8338	0.9911	0.9183
MPDR-R (ours)	0.8626	0.9932	0.8662
MPDR-R Single AE
Dz=128	0.6965	0.9326	0.9526
Dz=256	0.8048	0.9196	0.7772
Dz=1024	0.7443	0.9482	0.9247
overall performance, indicating that learning semantic information is important in this benchmark. Incorporating contrastive learning into MPDR framework is an interesting future direction.
CIFAR-100 OOD Detection on Pretrained Representation In Table 4, we test MPDR on OOD detection with CIFAR-100 inliers. To model a distribution of diverse images like CIFAR-100, we follow [34] and apply generative modeling in the representation space from a large-scale pretrained model. As we assume an unsupervised setting, we use pretrained representations without fine-tuning. Input images are transformed into 768D vectors by ViT-B_16 model [36]. ViT outputs are normalized with its norm and projected onto a hypersphere. We observe that adding a small Gaussian noise of 0.01 to training data improves stability of all algorithms. We use MLP for all energy functions and autoencoders. In MPDR, the manifold ensemble comprises three autoencoders with  $D_{\mathbf{z}} = 128$ , 256, 1024. We also implement supervised baselines (MD [34] and RMD [35]). The spherical projection is not applied for MD and RMD to respect their original implementation.
MPDR demonstrates strong anomaly detection performance in the representation space, with MPDR-S and MPDR-R outperforming IGEBM and AE/NAE, respectively. This success can be attributed to the low-dimensional structure often found in the representation space of in-distribution data, as observed in [37]. MPDR's average performance is nearly on par with supervised methods, MD and RMD, which utilize class information. Note that EBM inputs are no longer images, making previous EBM training techniques based on image transformation [14, 33] inapplicable.
Ablation Study Table 3 and 4 also report the results from single-manifold MPDR-R with varying latent dimensionality  $D_{\mathbf{z}}$  to show MPDR's sensitivity to a choice of an autoencoder manifold. Manifold ensemble effectively hedges the risk of relying on a single autoencoder which may not be optimal for detecting all types of outliers. Furthermore, manifold ensemble often achieves better AUROC score than MPDR with a single autoencoder. Additional ablation studies can be found in the Appendix. In Sec. B.2, we examine the sensitivity of MPDR to noise magnitude  $\sigma$  and the effectiveness of the noise magnitude ensemble. Also in Sec. B.4.4, we investigate the effect to scaling parameter  $\gamma$  and show that the training is unstable when  $\gamma$  is too large.
Multi-Class Anomaly Detection on MVTec-AD Using Pretrained Representation MVTec-AD [38] is also a popular anomaly detection benchmark dataset, containing images of 15 objects in their normal and defective forms. We follow the "unified" experimental setting from [39]. Normal images from 15 classes are used as the training set, where no label information is provided. We use the same feature extraction procedure used in [39]. Each image is transformed to a  $272 \times 14 \times 14$  feature map using EfficientNet-b4. When running MPDR, we treat each spatial dimension of a feature map
Table 5: Acoustic anomaly detection on DCASE 2020 Track 2 Dataset. AUROC and pAUROC (in parentheses) are displayed per cent. pAUROC is defined as AUROC computed over a restricted false positive rate interval  $[0, p]$ , where we set  $p = 0.1$ , following [41].
Toy Car	Toy Conveyor	Fan	Pump	Slider	Valve
AE [41]	75.40 (62.03)	77.38 (63.02)	66.44 (53.40)	71.42 (61.77)	89.65 (74.69)	72.52 (52.02)
MPDR-R	81.54 (68.21)	78.61 (63.99)	71.72 (55.95)	78.27 (68.14)	90.91 (76.58)	75.23 (51.04)
IDNN [44]	76.15 (72.36)	78.87 (62.50)	72.74 (54.30)	73.15 (61.25)	90.83 (74.16)	90.27 (69.46)
MPDR-IDNN	78.53 (73.34)	79.54 (65.35)	73.27 (54.57)	76.58 (66.49)	91.56 (75.19)	91.10 (70.87)
as an input vector to MPDR, transforming the task into a 272D density estimation problem. We normalize a 272D vector with its norm and add a small white noise with a standard deviation of 0.01 during training. We use the maximum energy value among  $14 \times 14$  vectors as an anomaly score of an image. For the localization task, we resize  $14 \times 14$  anomaly score map to  $224 \times 224$  image and compute AUROC for each pixel with respect to the ground true mask. We compare MPDR-R with UniAD [39] and DRAEM [40]. The results are shown in Table 12. Details on MPDR implementation can be found in B.5.
MPDR achieves AUROC scores that are very close to that of UniAD, outperforming DRAEM. The trend is consistent in both detection and localization tasks. The performance gap between MPDR and UniAD can be attributed to the fact that UniAD leverages spatial information of  $14 \times 14$  feature map while our implementation of MPDR processes each pixel in the feature map separately.
4.4 Acoustic Anomaly Detection
We apply MPDR to anomaly detection with acoustic signals, another form of non-image data. We use DCASE 2020 Challenge Task 2 dataset [41], containing recordings from six different machines, with three to four instances per machine type. The task is to detect anomalous sounds from deliberately damaged machines, which are unavailable during training. We applied the standard preprocessing methods in[41] to obtain a 640-dimensional vector built up of consecutive mel-spectrogram features. Many challenge submissions exploit dataset-specific heuristics and ensembles for high performance, e.g., [42, 43]. Rather than competing, we focus on demonstrating MPDR's effectiveness in improving common approaches, such as autoencoders and Interpolation Deep Neural Networks (IDNN) [44]. IDNN is a variant of masked autoencoders which predicts the middle (the third) frame given the remaining frames. Similarly to autoencoders, IDNN predicts an input as anomaly when the prediction error is large. We first train AE and IDNN for 100 epochs and then apply MPDR by treating the reconstruction (or prediction) error as the energy. Manifold ensemble consists of autoencoders with  $D_{\mathbf{z}} = 32,64,128$ . More training details can be found in the Appendix.
Table 5 shows that MPDR improves anomaly detection performance for both AE and IDNN. The known failure mode of AE and IDNN is producing unexpected low prediction (or reconstruction) error for anomalous inputs. By treating the error as the energy and applying generative training, the error in OOD region is increased through training, resulting in improved anomaly detection performance.
4.5 Anomaly Detection on Tabular Data
We test MPDR on ADBench [45], which consists 47 tabular datasets for anomaly detection. We compare MPDR with 13 baselines from ADBench. The baseline results are reproduced using the official ADBench repository. We consider the setting where the training split does not contain anomalies. For each dataset, we run each algorithm on three random splits and computed the AUROC on the corresponding test split. We then ranked the algorithms based on the averaged AUROC. In Table A, we present a summary table with the average rank across the 47 datasets.
We employ MPDR-R with a single manifold. For both the manifold and the energy, the same MLP-based autoencoder architecture is used. The encoder and the decoder are MLPs with two 1024-hidden-neuron layers. If the input space dimensionality is smaller than or equal to 100, the latent space dimensionality is set to have the same dimensionality  $D_{\mathbf{z}} = D_{\mathbf{x}}$ . If  $D_{\mathbf{x}} > 100$ , we set  $D_{\mathbf{z}}$  as  $70\%$  of  $D_{\mathbf{x}}$ . We employ 1 step of Langevin Monte Carlo in the latent space and 5 steps
in the input space. The step sizes are 0.1 for the latent space and 10 for the input space. All the hyperparameters except  $D_{\mathbf{z}}$  are fixed across 47 datasets.
As shown in Table 6, MPDR achieves highly competitive performance on ADBench, demonstrating a higher average rank than the isolation forest, with some overlap of confidence interval. This result indicates that MPDR is a general-purpose anomaly detection algorithm capable of handling tabular data. We empirically observe that MPDR is most effective when AUROC from its autoencoder is low, meaning that the outliers are near the autoencoder manifold. When the autoencoder already achieves good AUROC, the improvement from MPDR training is often marginal.
Leveraging autoencoders in training, MPDR is closely related to other EBM training algorithms that incorporate auxiliary modules. While the use of a single variational autoencoder in EBM training is explored in [18, 19, 46], MPDR employs multiple autoencoders. MPDR is also compatible with nonvariational autoencoders, offering greater flexibility. Normalizing flows [17, 25, 47] provide informative negative samples and a latent space for EBM training, but unlike MPDR, they do not exploit the low-dimensional structure of data. EBMs can also be trained with additional generator modules [48, 49], which plays a similar role to the decoder of an autoencoder. A contrastive representation learning module [33] improves EBM performance but relies on domain-specific data augmentations and is only demonstrated on images. In contrast, MPDR is applicable to a wider range of data, utilizing a more general assumption of low-dimensionality in data.
MPDR presents a novel objective function for EBM by extending recovery likelihood framework [22, 16]. Investigating its connection to other previously studied objective functions, such as  $f$ -divergence [50], pseudo-spherical scoring rule [51], divergence triangle [46], and Stein discrepancy [52], would also be an interesting future direction.
MPDR contributes to the field of anomaly detection [45, 39, 40, 53, 54]. The promising results from MPDR demonstrate that learning the distribution of data is a principled and effective approach for detecting anomalies [24].
Table 6: The rank of AUROC averaged over 47 datasets in ADBench. A smaller value indicates the algorithm achieves higher AUROC score compared to other algorithms on average. The standard errors are also displayed.
Method	Average Rank
MPDR (ours)	4.43 ± 0.50
IForest	5.28 ± 0.39
OCSVM	7.94 ± 0.47
CBLOF	5.98 ± 0.53
COF	9.23 ± 0.58
COPOD	7.10 ± 0.61
ECOD	6.97 ± 0.58
HBOS	6.53 ± 0.51
KNN	6.64 ± 0.56
LOF	9.04 ± 0.62
PCA	6.45 ± 0.64
SOD	7.91 ± 0.52
DeepSVDD	11.43 ± 0.42
DAGMM	10.09 ± 0.52
Contributions We propose MPDR, a novel method of utilizing autoencoders for training EBM. An autoencoder in MPDR provides an informative starting point for MCMC, offers the latent space for the effective traversal of a high-dimensional space, and guides the drift of an MCMC sampler (Eq. 7). MPDR performs competitively on various anomaly detection benchmarks involving diverse types of data, contributing to the enhancement of generative modeling for anomaly detection with high-dimensional data.
Limitations First, the practical performance of MPDR is still sensitive to the specifics of autoencoders used in MPD. Second, some data, such as high-resolution images or texts, may not exhibit a clear low-dimensional structure. In these cases, MPDR may require a separate representation learning stage, as demonstrated in our CIFAR-100 experiment with ViT (Sec. 4.3). Third, MPDR is not optimized for generating samples starting from a simple distribution, such as a Gaussian, while DRL is. We may resort to longer-chain MCMC to generate samples from MPDR.
S. Yoon would like to thank Luigi Favaro and Tilman Plehn for their helpful discussions during the early development of the algorithm. S. Yoon is supported by a KIAS Individual Grant (AP095701) via the Center for AI and Natural Sciences at Korea Institute for Advanced Study. Y.-K. Noh was partly supported by NRF/MSIT (No. 2018R1A5A7059549, 2021M3E5D2A01019545) and IITP/MSIT (IITP-2021-0-02068, 2020-0-01373, RS-2023-00220628). This work was supported in part by IITP-MSIT grant 2021-0-02068 (SNU AI Innovation Hub), IITP-MSIT grant 2022-0-00480 (Training and Inference Methods for Goal-Oriented AI Agents), KIAT grant P0020536 (HRD Program for Industrial Innovation), ATC+ MOTIE Technology Innovation Program grant 20008547, SRRC NRF grant RS-2023-00208052, SNU-AIIS, SNU-IAMD, SNU BK21+ Program in Mechanical Engineering, and SNU Institute for Engineering Research.