SlowGuess/NeurIPS_2023_OCR huggingface dataset:
[1] Paul Bergmann, Michael Fauser, David Sattlegger, and Carsten Steger. Mvtec ad - a comprehensive real-world dataset for unsupervised anomaly detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2019.
[2] Yuma Koizumi, Shoichiro Saito, Hisashi Uematsu, Noboru Harada, and Keisuke Imoto. Toyadmos: A dataset of miniature-machine operating sounds for anomalous sound detection. In 2019 IEEE Workshop on Applications of Signal Processing to Audio and Acoustics (WASPAA), pages 313-317, 2019.
[3] Harsh Purohit, Ryo Tanabe, Kenji Ichige, Takashi Endo, Yuki Nikaido, Kaori Suefusa, and Yohei Kawaguchi. Mimii dataset: Sound dataset for malfunctioning industrial machine investigation and inspection. arXiv preprint arXiv:2206.14225, 2019.
[4] Barry M Dillon, Luigi Favaro, Tilman Plehn, Peter Sorrenson, and Michael Kramer. A normalized autoencoder for lhc triggers. arXiv preprint arXiv:2206.14225, 2022.
[5] Christopher M Bishop. Novelty detection and neural network validation. IEEE Proceedings-Vision, Image and Signal processing, 141(4):217-222, 1994.
[6] Diederik P Kingma and Max Welling. Auto-encoding variational bayes. International Conference on Learning Representations (ICLR), 2014.
[7] Aaron van den Oord, Nal Kalchbrenner, and Koray Kavukcuoglu. Pixel recurrent neural networks. arXiv preprint arXiv:1601.06759, 2016.
[8] Durk P Kingma and Prafulla Dhariwal. Glow: Generative flow with invertible 1x1 convolutions. In Advances in Neural Information Processing Systems, pages 10215-10224, 2018.
[9] Dan Hendrycks, Mantas Mazeika, and Thomas Dietterich. Deep anomaly detection with outlier exposure. In International Conference on Learning Representations, 2019.
[10] Eric Nalisnick, Akihiro Matsukawa, Yee Whye Teh, Dilan Gorur, and Balaji Lakshminarayanan. Do deep generative models know what they don't know? In International Conference on Learning Representations, 2019.
[11] Yilun Du and Igor Mordatch. Implicit generation and modeling with energy based models. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d Alche-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems 32, pages 3608-3618. Curran Associates, Inc., 2019.
[12] Erik Nijkamp, Mitch Hill, Song-Chun Zhu, and Ying Nian Wu. Learning non-convergent non-persistent short-run mcmc toward energy-based model. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 32, pages 5232-5242. Curran Associates, Inc., 2019.
[13] Tijmen Tieleman. Training restricted boltzmann machines using approximations to the likelihood gradient. In Proceedings of the 25th international conference on Machine learning, pages 1064-1071, 2008.
[14] Yilun Du, Shuang Li, B. Joshua Tenenbaum, and Igor Mordatch. Improved contrastive divergence training of energy based models. In Proceedings of the 38th International Conference on Machine Learning (ICML-21), 2021.
[15] Geoffrey E Hinton. Training products of experts by minimizing contrastive divergence. Neural computation, 14(8):1771-1800, 2002.
[52] Will Grathwohl, Kuan-Chieh Wang, Joern-Henrik Jacobsen, David Duvenaud, and Richard Zemel. Learning the stein discrepancy for training and evaluating energy-based models without sampling. In Hal Daumé III and Aarti Singh, editors, Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 3732-3747. PMLR, 13-18 Jul 2020.
[53] Zongyuan Huang, Baohua Zhang, Guoqiang Hu, Longyuan Li, Yanyan Xu, and Yaohui Jin. Enhancing unsupervised anomaly detection with score-guided network. IEEE Transactions on Neural Networks and Learning Systems, 2023.
[54] Dmitrii Gavrilev and Evgeny Burnaev. Anomaly detection in networks via score-based generative models. In ICML 2023 Workshop on Structured Probabilistic Inference & Generative Modeling, 2023.
[55] Tarin Clanuwat, Mikel Bober-Irizar, Asanobu Kitamoto, Alex Lamb, Kazuaki Yamamoto, and David Ha. Deep learning for classical japanese literature. arXiv preprint arXiv:1812.01718, 2018.
[56] Gregory Cohen, Saeed Afshar, Jonathan Tapson, and Andre Van Schaik. Emmist: Extending mnist to handwritten letters. In 2017 international joint conference on neural networks (IJCNN), pages 2921-2926. IEEE, 2017.
[57] Brenden M Lake, Ruslan Salakhutdinov, and Joshua B Tenenbaum. The omniglot challenge: a 3-year progress report. Current Opinion in Behavioral Sciences, 29:97-104, 2019.
[58] Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms. arXiv preprint arXiv:1708.07747, 2017.
[59] A Krizhevsky. Learning multiple layers of features from tiny images. Master's thesis, University of Tront, 2009.
[60] Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y Ng. Reading digits in natural images with unsupervised feature learning. In NIPS Workshop on Deep Learning and Unsupervised Feature Learning, 2011.
[61] M. Cimpoi, S. Maji, I. Kokkinos, S. Mohamed, , and A. Vedaldi. Describing textures in the wild. In Proceedings of the IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2014.
[62] Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaou Tang. Deep learning face attributes in the wild. In Proceedings of International Conference on Computer Vision (ICCV), December 2015.
[63] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pages 1597-1607. PMLR, 2020.
[64] Gareth O Roberts, Richard L Tweedie, et al. Exponential convergence of Langevin distributions and their discrete approximations. Bernoulli, 2(4):341-363, 1996.
A Consistency of MPDR
Let us denote our model for recovery likelihood as  $p(\mathbf{x}|\tilde{\mathbf{z}};\theta)$ . We assume that this model is identifiable (different model parameters correspond to different distribution) and correctly specified (there exists  $\theta^*$  such that  $p(\mathbf{x}|\tilde{\mathbf{z}};\theta^*) = p_{data}(\mathbf{x}|\tilde{\mathbf{z}})$ ). We also assume that  $p(\mathbf{x}|\tilde{\mathbf{z}};\theta)$  is non-zero for all  $\mathbf{x}$ ,  $\tilde{\mathbf{z}}$ , and  $\theta$ . Our objective is to  $\max_{\theta}\frac{1}{N}\sum_{i=1}^{N}\log p(\mathbf{x}_i|\tilde{\mathbf{z}}_i;\theta)$  where  $(\mathbf{x}_i,\tilde{\mathbf{z}}_i)\sim p(\mathbf{x},\tilde{\mathbf{z}}) = p_{data}(\mathbf{x})p(\tilde{\mathbf{z}}|\mathbf{x})$ . In the limit of  $N\to \infty$ , the average converges to the expectation  $\frac{1}{N}\sum_{i=1}^{N}\log p(\mathbf{x}_i|\tilde{\mathbf{z}}_i;\theta)\rightarrow \mathbb{E}_{(\mathbf{x},\tilde{\mathbf{x}})}[\log p(\mathbf{x}|\tilde{\mathbf{z}};\theta)]$ . If we subtract  $\mathbb{E}_{(\mathbf{x},\tilde{\mathbf{x}})}[\log p(\mathbf{x}|\tilde{\mathbf{z}};\theta^*)]$  which is constant with respect to  $\theta$ , then the expression can be written with respect to KL divergence as follows:
$$
\begin{array}{l} \mathbb {E} _ {(\mathbf {x}, \tilde {\mathbf {x}})} [ \log p (\mathbf {x} | \tilde {\mathbf {z}}; \theta) - \log p (\mathbf {x} | \tilde {\mathbf {z}}; \theta^ {*}) ] = \int p (\mathbf {x}, \tilde {\mathbf {z}}) \log \frac {p (\mathbf {x} | \tilde {\mathbf {z}} ; \theta)}{p (\mathbf {x} | \tilde {\mathbf {z}} ; \theta^ {*})} d \mathbf {x} d \tilde {\mathbf {z}} (9) \\ = - \int p (\tilde {\mathbf {z}}) \operatorname {K L} \left(p (\mathbf {x} | \tilde {\mathbf {z}}; \theta^ {*}) \| p (\mathbf {x} | \tilde {\mathbf {z}}; \theta)\right) d \tilde {\mathbf {z}} (10) \\ \end{array}
$$
The maximum of Eq. 10 is 0, as the minimum of KL divergence is 0. The maximum is achieved if and only if  $\theta = \theta^{*}$ . Note that  $p(\tilde{\mathbf{z}})$  is assumed to be constant with respect to  $\theta$ .
Since we did not rely on the specifics of how a perturbation  $p(\tilde{\mathbf{z}}|\mathbf{x})$  is actually performed, this consistency result holds for any choices of the encoder  $f_{e}$  and the noise magnitude  $\sigma$ , as long as the recovery likelihood  $p(\mathbf{x}|\tilde{\mathbf{z}};\theta)$  remains non-zero for all  $\mathbf{x}$ ,  $\tilde{\mathbf{z}}$ , and  $\theta$ .
B Experimental Details and Additional Results
In this section, we provide detailed information on how each experiment is conducted and also provide some additional supporting experimental results. The hyperparameters related to LMC is summarized in Table 7. ConvNet architectures are provided in Table 8. The contents are organized by the training dataset.
Table 7: Hyperparameters for LMC. Latent chain hyperparameters are denoted by  $\mathcal{Z}$  and  $\mathcal{X}$  indicates visible chain hyperparameters. "scale ( $\gamma$ " refers to the multiplicative scale factor on the perturbation probability.
Experiment	Z Steps	Z Step Size	Z Noise	Z scale (γ)	X Steps	X Step Size	X Noise	X scale (γ)
MNIST
MPDR-S	2	0.05	0.02	0.0001	5	10	0.005	0
MPDR-R	5	0.1	0.02	0.0001	5	10	0.005	0
CIFAR-10
MPDR-S	10	0.1	0.01	0.0001	20	10	0.005	0
MPDR-R	10	0.1	0.01	0.0001	20	10	0.005	0
CIFAR-100 + ViT
MPDR-S	0	-	-	-	30	1	0.005	0.0001
MPDR-R	0	-	-	-	30	1	0.005	0.0001
DCASE 2020 (Toy Car, Toy Conveyor, Pump)
MPDR-R	0	-	-	-	5	10	0.005	0.0001
MPDR-IDNN	0	-	-	-	5	10	0.005	0.0001
DCASE 2020 (Fan, Slider, Valve)
MPDR-R	0	-	-	-	5	10	0.005	0.0001
MPDR-IDNN	20	0.1	0.01	0.0001	20	10	0.005	0.0001
MVTec-AD
MPDR-R	0	-	-	-	10	0.1	0.1	0.0001
ADBench
MPDR-R	1	0.1	0.05	0.0001	5	10	0.1	0.0001
B.1 MNIST
All input images are  $28 \times 28$  grayscale, and each pixel value is represented as a floating number between [0, 1]. Models are trained on the training split of MNIST $^2$ , excluding the digit designated to be held-out. The training split contains 60,000 images, and the hold-out procedure reduces the training set to approximately  $90\%$ . We evaluate the models on the test split of MNIST, which contains a total of 10,000 images. For non-MNIST datasets used in evaluation, we only use their test split. All non-MNIST datasets used in the experiment also follow the  $28 \times 28$  grayscale format, similar to MNIST.
- KMNIST (KMNIST-MNIST) [55] contains Japanese handwritten letters, pre-processed into the same format as MNIST. The license of KMNIST is CC BY-SA 4.0.
- EMNIST (EMNIST-Letters) [56] contains grayscale handwritten English alphabet images. Its test split contains 20,800 samples. No license information is available.
- For the Omniglot [57] dataset, the evaluation set is used. The set consists of 13,180 images. No license information is available.
- We use the test split of FashionMNIST [58], which contains 10,000 images. The dataset is made public under the MIT license.
- Constant dataset is a synthetic dataset that contains  $28 \times 28$  images, where all pixels have the same value. The value is randomly drawn from a uniform distribution over  $[0,1]$ . We use 4,000 constant images.
B.1.2 Autoencoder Implementation and Training
The encoder and the decoder used in MPDR,  $f_{e}$  and  $f_{d}$ , have the architecture of MNIST Encoder and MNIST Decoder, provided in Table 8, respectively. We use the spherical latent space  $\mathbb{S}^{D_{\mathbf{z}} - 1}$  where  $D_{\mathbf{z}} = 32$  for the main experiment. The autoencoder is trained to minimize the  $l_{2}$  reconstruction error of the training data for 30 epochs with Adam of learning rate  $1 \times 10^{-4}$ . The batch size is 128 and no data augmentation is applied. The  $l_{2}$  norm of the encoder is regularized with the coefficient of  $1 \times 10^{-4}$ . The same autoencoder is used for both MPDR-S and MPDR-R.
B.1.3 MPDR Implementation and Training
In MPDR-S, the energy function  $E_{\theta}$  has the architecture of MNIST Encoder (Table 8) with  $D_{\mathbf{z}} = 1$ . The network is randomly initialized from PyTorch default setting and the spectral normalization is applied. The energy function is trained with MPDR algorithm for 50 epochs. The learning rate is  $1 \times 10^{-4}$ . The batch size is 128. The perturbation probability scaling factor  $\gamma$  for the visible LMC chain is set to zero. For an image-like data such as MNIST, the gradient of perturbation probability  $\nabla_{\mathbf{x}}(||\tilde{\mathbf{z}} - f_e(\mathbf{x})||^2)$  introduces non-smooth high-frequency patterns resembling adversarial noise, harming the stability of the training. Therefore, we only use non-zero  $\gamma$  in latent chains in MNIST and CIFAR-10 experiments.
For MPDR-R, the energy function is initialized from  $(f_e,f_d)$  and then trained through MPDR algorithm. The learning rate is set to  $1\times 10^{-5}$ . All the other details are identical to the MPDR-S case.
B.2 Sensitivity to  $\sigma$
As an ablation study for noise magnitude ensemble, we perform single-noise-magnitude experiment for MPDR and examine MPDR's sensitivity to the noise magnitude  $\sigma$ . We evaluate the OOD detection performance of MPDR-S on the MNIST hold-out digit 9 setting with varying values of  $\sigma$ . Results are shown in Table 9.
The choice of  $\sigma$  has a significant impact on MPDR's OOD detection performance. In Table 9,  $\sigma = 0.01$  gives poor OOD detection performance, particularly with respect to the hold-out digit. The performance generally improves as  $\sigma$  grows larger, but a large  $\sigma$  is not optimal for detecting EMNIST.
2http://yann.1ecun.com/exdb/mnist/
<sup>3</sup>https://github.com/rois-codh/kmnist
Table 8: Convolutional neural network architectures used in experiments. The parenthesis following the network name indicates the activation function used in the network.
MNIST Encoder (ReLU)	MNIST Decoder (ReLU)
Conv3(1, 32)	ConvT4(Dz, 128)	CIFAR-10 Encoder 1	CIFAR-10 Decoder 1
Conv3(32, 64)	Upsample(2x)	Conv4(3, 32, stride=2)	ConvT8(Dz, 256)
MaxPool(2x)	ConvT3(128, 64)	Conv4(32, 64, stride=2)	ConvT4(256, 128, stride=2, pad=1)
Conv3(64, 64)	ConvT3(64, 64)	Conv4(64, 128, stride=2)	ConvT4(128, 64, stride=2, pad=1)
Conv3(64, 128)	Upsample(2x)	Conv2(128, 256, stride=2)	ConvT1(64, 3)
MaxPool(2x)	ConvT3(64, 32)	FC(256, Dz)	Sigmoid()
Conv4(128, 1024)	ConvT3(32, 1)
FC(1024, Dz)	Sigmoid()
CIFAR-10 Encoder 2
Conv3(3, 128, pad=1)	CIFAR-10 Decoder 2
ResBlock(128, 128, down=True)	ConvT4(Dz, 128)
ResBlock(128, 128)	ResBlock(128, 128, up=True)
ResBlock(128, 256, down=True)	ResBlock(128, 128, up=True)
ResBlock(256, 256)	ResBlock(128, 128, up=True)
ResBlock(256, 256, down=True)	ResBlock(128, 128, up=True)
ResBlock(256, 256)	Conv3(128, 3, pad=1)
GlobalAvgPool()
FC(256, Dz)
Selecting a single optimal  $\sigma$  will be very difficult, and therefore, we employ noise magnitude ensemble which can hedge the risk of choosing a suboptimal value for  $\sigma$ .
B.2.1 Sensitivity to  $D_{\mathbf{z}}$
As an ablation study for manifold ensemble, we investigate the sensitivity of MPDR to the choice of the latent space dimensionality of the autoencoder. We evaluate the OOD detection performance of MPDR-S on the MNIST hold-out digit 9 setting with varying values of  $D_{\mathbf{z}}$ . Table 10 presents the results. Consequently, MPDR runs stably for a large range of  $D_{\mathbf{z}}$ , producing decent OOD performance. One hypothesis is that, for MNIST, it is relatively easy for these autoencoders to capture the manifold structure of MNIST sufficiently well.
Meanwhile, we do observe that the choice of  $D_{\mathbf{z}}$  affects OOD performance in an interesting way. Increasing  $D_{\mathbf{z}}$  enhances AUPR for certain OOD datasets but deteriorates AUPR for others. For example, AUPR of Omniglot is increased with larger  $D_{\mathbf{z}}$ , but AUPR of EMNIST, FashionMNIST, and Constant dataset decreases. No single autoencoder is optimal for detecting all outlier datasets. This observation motivates the use of manifold ensemble, employed in non-MNIST MPDR experiments.
B.2.2 Note on Reproduction
For an autoencoder-based outlier detector, denoted as "AE" in Table 1, we use the same autoencoder used in MPDR with  $D_{\mathbf{z}}$ .
NAE is reproduced based on its public code base<sup>4</sup>.
We tried to reproduce DRL on MNIST but failed to train it stably. The original paper and the official code base also does not provide a guidance on training DRL on MNIST.
B.3 CIFAR-10 OOD Detection
B.3.1 Datasets
All data used in CIFAR-10 experiment are in the  $32 \times 32$  RGB format. Models are only trained on CIFAR-10 training set, and evaluated on the testing set of each dataset.
- CIFAR-10 [59] contains 60,000 training images and 10,000 testing images. Models are trained on the training set. We don't use its class information, as we consider only unsupervised setting. No license information available.
<sup>4</sup>https://github.com/swyoon/normalized-autoencoders
Table 9: Sensitivity to  $\sigma$ . MPDR-S is run with an autoencoder with varying values of noise magnitude  $\sigma$ . AUPR against various outlier datasets are presented. For MNIST 9, we present the standard deviation computed over the last 10 epochs.
σ	MNIST 9	KMNIST	EMNIST	Omniglot	FashionMNIST	Constant
0.01	0.098 ± 0.009	0.667	0.827	0.940	0.944	0.895
0.1	0.330 ± 0.063	0.983	0.995	0.971	0.994	0.998
0.2	0.522 ± 0.053	0.999	0.993	0.997	0.999	1.000
0.3	0.558 ± 0.039	1.000	0.990	0.997	1.000	1.000
Table 10: Sensitivity to  $D_{\mathbf{z}}$ . MPDR-S is run with an autoencoder with varying values of  $D_{\mathbf{z}}$ . AUPR against various outlier datasets are presented. For MNIST 9, we present the standard deviation computed over the last 10 epochs. Noise magnitude ensemble is applied.
Dz	MNIST 9	KMNIST	EMNIST	Omniglot	FashionMNIST	Constant
16	0.611 ± 0.041	0.979	0.996	0.958	0.999	1.000
32	0.525 ± 0.039	0.999	0.994	0.994	0.999	1.000
64	0.512 ± 0.048	0.999	0.993	0.998	0.998	0.999
128	0.505 ± 0.051	0.999	0.991	0.999	0.988	0.946
256	0.590 ± 0.041	0.996	0.983	0.996	0.958	0.866
- SVHN [60] is a set of digit images. Its test set containing 26,032 is used in the experiment. The dataset is non-commercial use only.
- Texture [61] dataset, also called Describable Textures Dataset (DTD), contains 1,880 test images. The images are resized into  $32 \times 32$ . No license information available.
CelebA [62] is a dataset of cropped and aligned human face images. The test set contains 19,962 images. The dataset is for non-commercial research purposes only. We center-crop each image into  $140 \times 140$  and then resize it into  $32 \times 32$ .
- Constant dataset is a synthetic dataset that contains 4,000  $32 \times 32$  RGB monochrome image. All  $32 \times 32$  pixels have the same RGB value which is drawn uniform-randomly from  $[0,1]^3$ .
- CIFAR-100 [59] contains 60,000 training images and 10,000 testing images. No license information available.
B.3.2 Autoencoder Implementation and Training
The autoencoders for CIFAR-10 experiment have an architecture of "CIFAR-10 Encoder 1" and "CIFAR-10 Decoder 1" in Table 8 with  $D_{\mathbf{z}} = 32,64,128$ . Each autoencoder is trained for 40 epoch with learning rate  $1\times 10^{-4}$  and batch size 128. During training the autoencoders, we apply the following data augmentation operations: random horizontal flipping with the probability of 0.5, random resize crop with the scale parameter [0.08, 1] with the probability 0.2, color jittering with probability of 0.2, random grayscale operation with the probability 0.2. The color jittering parameters are the same with the one used in SimCLR [63] (brightness 0.8, contrast 0.8, saturation 0.8, hue 0.4, with respect to torchvision.transorms.ColorJitter implementation). The  $l_{2}$  norm of an encoder is regularized with the coefficient of 0.00001. The same autoencoder is used for both MPDR-S and MPDR-R. To boost the performance, we heuristically introduce sample replay buffer for negative samples and apply data augmentation in the middle of LMC.
B.3.3 MPDR Implementation and Training
The energy function in MPDR-S is "CIFAR-10 Encoder 2" with  $D_{\mathbf{z}} = 1$ . The energy function in MPDR-R is "CIFAR-10 Encoder 2" and "CIFAR-10 Decoder 2" with  $D_{\mathbf{z}} = 1$ . In MPDR-R, the energy function is pre-trained by minimizing the reconstruction error of the training data for 40 epochs. Only random horizontal flip is applied and no other data augmentation is used. Similarly to the MNIST case, we set  $\gamma = 0$  for the visible LMC chain.
<sup>5</sup>https://mmlab.ie.cuhk.edu.hk/projects/CelebA.html
B.3.4 Note on Reproduction
For Improved CD, we train a CIFAR-10 model from scratch using the training script provided by the authors without any modification. We use the model with the best Inception Score to compute AUROC scores.
For DRL, we use the official checkpoint for T6 CIFAR-10 model and compute its energy to perform OOD detection.
Also for NVAE, we use the official CIFAR-10 checkpoint provided in the official repository  $^{8}$ . We use negative log-likelihood as the outlier score. Due to computational budget constraint, we could only set the number of importance weighted sample to be one -num iw_samples=1.
As in MNIST, we use the official CIFAR-10 checkpoint of NAE provided by the authors.
B.4 CIFAR-100 OOD Detection on Pretrained Representation
B.4.1 Datasets
CIFAR-100, CIFAR-10, SVHN, and CelebA datasets are used and are described in the previous section. Each image is resized to  $224 \times 224$  and fed to ViT-B_16 to produce a 768-dimensional vector. MD and RMD operate with this vector. For other methods, the 768D vector is projected onto a hypersphere.
B.4.2 Autoencoder Implementation and Training
Each encoder and decoder is an MLP with two hidden layers where each layer contains 1024 hidden neurons. The leaky ReLU activation function is used in all hidden layers. We use  $D_{\mathbf{z}} = 128,256,1024$ . The autoencoders are trained to minimize the reconstruction error of the training data. During training, the Gaussian noise with the standard deviation of 0.01 is added to each training sample. The  $l_{2}$  norm of the encoder's weights are regularized with the coefficient of  $1\times 10^{-6}$ .
B.4.3 MPDR Implementation and Training
The energy functions are also MLPs. The energy function of MPDR-S has the same architecture as the encoder of the autoencoder with  $D_{\mathbf{z}} = 1$ . The energy function of MPDR-R is an autoencoder with the latent dimensionality of 1024. The energy functions are trained for 30 epochs with the learning rate of  $1 \times 10^{-4}$ .
B.4.4 Sensitivity to  $\gamma$
We examine how  $\gamma$  affects MPDR's performance. As seen in Table 11, MPDR shows the best performance on the small  $\gamma$  regime, roughly from 0.0001 to 0.001. Setting too large  $\gamma$  is detrimental for the performance and often even incurs training instabilities. It is interesting to note that  $\gamma = 0$  also gives a decent result. One possible explanation is that  $\gamma = 0$  reduces the negative sample distribution  $p_{\theta}(\mathbf{x}|\tilde{\mathbf{z}})$  to the model distribution  $p_{\theta}(\mathbf{x})$ , which is still a valid negative sample distribution for training an EBM.
Table 11: Sensitivity of  $\gamma$ , demonstrated in CIFAR-100 experiment. AUROC values are displayed.
γ	CIFAR10	SVHN	CelebA
0	0.8580	0.9931	0.8456
0.0001	0.8626	0.9932	0.8662
0.001	0.8639	0.9918	0.8625
0.01	0.8496	0.9894	0.8576
0.1	0.8186	0.9424	0.8511
<sup>6</sup>https://github.com/yilundu/improved_contrastive_divergence
<sup>7</sup>https://github.com/ruiqigao/recovery_likelihood
<sup>8</sup>https://github.com/NVlabs/NVAE
Table 12: MVTec-AD detection and localization task in the unified setting. AUROC scores (percent) are computed for each class. UniAD and DRAEM results are adopted from [39]. The largest value in a task is marked as boldface.
MPDR (ours)	Detection UniAd[39]	DRAEM [40]	MPDR (ours)	Localization UniAd[39]	DRAEM[40]
Bottle	100.0	99.7	97.5	98.5	98.1	87.6
Cable	95.5	95.2	57.8	95.6	97.3	71.3
Capsule	86.4	86.9	65.3	98.2	98.5	50.5
Hazelnut	99.9	99.8	93.7	98.4	98.1	96.9
Metal Nut	99.9	99.2	72.8	94.5	94.8	62.2
Pill	94.0	93.7	82.2	94.9	95.0	94.4
Screw	85.9	87.5	92.0	98.1	98.3	95.5
Toothbrush	89.6	94.2	90.6	98.7	98.4	97.7
Transistor	98.3	99.8	74.8	95.4	97.9	65.5
Zipper	95.3	95.8	98.8	96.2	96.8	98.3
Carpet	99.9	99.8	98.0	98.8	98.5	98.6
Grid	97.9	98.2	99.3	96.9	96.5	98.7
Leather	100.0	100	98.7	98.5	98.8	97.3
Tile	100.0	99.3	98.7	94.6	91.8	98.0
Wood	97.9	98.6	99.8	93.8	93.2	96.0
Mean	96.0	96.5	88.1	96.7	96.8	87.2