SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Sangwoong Yoon
Korea Institute for Advanced Study
swyoon@kias.re.kr
Young-Uk Jin
yueric.jin@samsung.com
Yung-Kyun Noh*
Hanyang University
nohyung@hanyang.ac.kr
Frank C. Park*
Saige Research
ftp@snu.ac.kr
We present a new method of training energy-based models (EBMs) for anomaly detection that leverages low-dimensional structures within data. The proposed algorithm, Manifold Projection-Diffusion Recovery (MPDR), first perturbs a data point along a low-dimensional manifold that approximates the training dataset. Then, EBM is trained to maximize the probability of recovering the original data. The training involves the generation of negative samples via MCMC, as in conventional EBM training, but from a different distribution concentrated near the manifold. The resulting near-manifold negative samples are highly informative, reflecting relevant modes of variation in data. An energy function of MPDR effectively learns accurate boundaries of the training data distribution and excels at detecting out-of-distribution samples. Experimental results show that MPDR exhibits strong performance across various anomaly detection tasks involving diverse data types, such as images, vectors, and acoustic signals.
Unsupervised detection of anomalous data appears frequently in practical applications, such as industrial surface inspection [1], machine fault detection [2, 3], and particle physics [4]. Modeling the probability distribution of normal data  $p_{data}(\mathbf{x})$  is a principled approach for anomaly detection [5]. Anomalies, often called outliers or out-of-distribution (OOD) samples, lie outside of the data distribution and can thus be characterized by low probability density under the distribution. However, many deep generative models that can evaluate the likelihood of data, including variational autoencoders (VAE; [6]), autoregressive models [7], and flow-based models [8] are known to perform poorly on popular anomaly detection benchmarks such as CIFAR-10 (in) vs SVHN (out), by assigning high likelihood on seemingly trivial outliers [9, 10].
On the other hand, deep energy-based models (EBMs) have demonstrated notable improvement in anomaly detection compared to other deep generative models [11]. While the specific reason for the superior performance of EBMs has not been formally analyzed, one probable factor is the explicit mechanism employed in EBM's maximum likelihood training that reduces the likelihood of negative samples. These negative samples are generated from the model distribution  $p_{\theta}(\mathbf{x})$  using Markov Chain Monte Carlo (MCMC). Since modern EBMs operate in high-dimensional data spaces, it is extremely difficult to cover the entire space with a finite-length Markov chain. The difficulty in
Figure 1: (Left) An illustration of Manifold Projection-Diffusion. A datum  $\mathbf{x}$  is first projected into the latent space  $\mathcal{Z}$  through the encoder  $f_{e}(\mathbf{x})$  and then diffused with a local noise, such as Gaussian. The perturbed sample  $\tilde{\mathbf{x}}$  is obtained by projecting it back to the input space  $\mathcal{X}$  through the decoder  $f_{d}(\mathbf{z})$ . (Right) Comparison between the perturbations in MPDR and DRL [16] on CIFAR-10 examples.
generating negative samples triggered the development of several heuristics such as using truncated chains [12], persistent chains [13], sample replay buffers [11], and data augmentation in the middle of the chain [14].
Instead of requiring a Markov chain to cover the entire space, EBM can be trained with MCMC running within the vicinity of each training datum. For example, Contrastive Divergence (CD; [15]) uses a short Markov chain initialized on training data to generate negative samples close to the training distribution. Diffusion Recovery Likelihood (DRL; [16]) enables MCMC to focus on the training data's neighborhood using a different objective function called recovery likelihood. Recovery likelihood  $p_{\theta}(\mathbf{x}|\tilde{\mathbf{x}})$  is the conditional probability of training datum  $\mathbf{x}$  given the observation of its copy  $\tilde{\mathbf{x}}$  perturbed with a Gaussian noise. Training of DRL requires sampling from the recovery likelihood distribution  $p_{\theta}(\mathbf{x}|\tilde{\mathbf{x}})$ , which is easier than sampling from the model distribution  $p_{\theta}(\mathbf{x})$ , as  $p_{\theta}(\mathbf{x}|\tilde{\mathbf{x}})$  is close to uni-modal and concentrated near  $\mathbf{x}$ . While CD and DRL significantly stabilize the negative sampling process, the resulting EBMs exhibit limited anomaly detection performance due to the insufficient coverage of negative samples in the input space.
In this paper, we present a novel training algorithm for EBM that does not require MCMC covering the entire space while achieving strong anomaly detection performance. The proposed algorithm, Manifold Projection-Diffusion Recovery (MPDR), extends the recovery likelihood framework by replacing Gaussian noise with Manifold Projection-Diffusion (MPD), a novel perturbation operation that reflects low-dimensional structure of data. Leveraging a separately trained autoencoder manifold, MPD firstly projects a training datum onto the manifold and perturbs the datum along the manifold. Compared to Gaussian noise, MPD reflects relevant modes of variation in data, such as change in colors or shapes in an image, as shown in Fig. 1. A MPD-perturbed sample serves as an informative starting point for MCMC that generates a negative sample, teaching EBM to discriminate challenging outliers that has partial similarity to training data.
Given an MPD-perturbed sample, EBM is trained by maximizing the recovery likelihood. We derive a simple expression for evaluating the recovery likelihood for MPD. We also propose an efficient two-stage MCMC strategy for sampling from the recovery likelihood, leveraging the latent space of the autoencoder. Additionally, we show that the model parameter estimation by MPDR is consistent under standard assumptions.
Compared to existing EBM training algorithms using an autoencoder-like auxiliary module [17, 18, 19, 20], MPDR has two advantages. First, MPDR allows the use of multiple autoencoders for training. The manifold ensemble technique stabilizes the training and improves anomaly detection performance by providing diverse negative samples. Second, MPDR achieves good anomaly detection performance with lightweight autoencoders. For example, autoencoders in our CIFAR-10 experiment has about only  $1.5\sim 3$  million parameters, which are significantly fewer than that of NVAE [21] ( $\sim 130$  million) or Glow [8] ( $\sim 44$  million), both of which are previously used in EBM training [17, 18].
- We propose MPDR, a novel method of using autoencoders for training EBM. MPDR is compatible with any energy functions and gives consistent density parameter estimation with any autoencoders.
- We provide a suite of practical strategies for achieving successful anomaly detection with MPDR, including two-stage sampling, energy function design, and assembling multiple autoencoders.
- We demonstrate the effectiveness of MPDR on unsupervised anomaly detection tasks with various data types, including images, representation vectors, and acoustic signals.
Energy-Based Models (EBM) An energy-based generative model, or an unnormalized probabilistic model, represents a probability density function using a scalar energy function  $E_{\theta}:\mathcal{X}\to \mathbb{R}$ , where  $\mathcal{X}$  denotes the domain of data. The energy function  $E_{\theta}$  defines a probability density function  $p_{\theta}$  through the following relationship:
$$
p _ {\theta} (\mathbf {x}) \propto \exp (- E _ {\theta} (\mathbf {x})). \tag {1}
$$
The parameters  $\theta$  can be learned by maximum likelihood estimation given iid samples from the underlying data distribution  $p_{data}(\mathbf{x})$ . The gradient of the log-likelihood for a training sample  $\mathbf{x}$  is well-known [15] and can be written as follows:
$$
\nabla_ {\theta} \log p _ {\theta} (\mathbf {x}) = - \nabla_ {\theta} E _ {\theta} (\mathbf {x}) + \mathbb {E} _ {\mathbf {x} ^ {-} \sim p _ {\theta} (\mathbf {x})} [ \nabla_ {\theta} E _ {\theta} (\mathbf {x} ^ {-}) ], \tag {2}
$$
where  $\mathbf{x}^{-}$  denotes a "negative" sample drawn from the model distribution  $p_{\theta}(\mathbf{x})$ . Typically,  $\mathbf{x}^{-}$  is generated using Langevin Monte Carlo (LMC). In LMC, points are arbitrarily initialized and then iteratively updated in a stochastic manner to simulate independent sampling from  $p_{\theta}(\mathbf{x})$ . For each time step  $t$ , a point  $\mathbf{x}^{(t)}$  is updated by  $\mathbf{x}^{(t + 1)} = \mathbf{x}^{(t)} - \lambda_1\nabla_{\mathbf{x}}E_{\theta}(\mathbf{x}^{(t)}) + \lambda_2\epsilon^{(t)}$ , for  $\epsilon^{(t)} \sim \mathcal{N}(0,\mathbf{I})$ . The step size  $\lambda_{1}$  and the noise scale  $\lambda_{2}$  are often tuned separately in practice. Since LMC needs to be performed in every iteration of training, it is infeasible to run negative sampling until convergence, and a compromise must be made. Popular heuristics include initializing MCMC on training data [15], using short-run LMC [12], and utilizing a replay buffer [13, 11].
Recovery Likelihood [22, 16] Instead of directly maximizing the likelihood (Eq. (2)),  $\theta$  can be learned through the process of denoising data from artificially injected Gaussian noises. Denoising corresponds to maximizing the recovery likelihood  $p(\mathbf{x}|\tilde{\mathbf{x}})$ , the probability of recovering data  $\mathbf{x}$  from its noise-corrupted version  $\tilde{\mathbf{x}} = \mathbf{x} + \sigma \epsilon$ , where  $\epsilon \sim \mathcal{N}(0,\mathbf{I})$ .
$$
p _ {\theta} (\mathbf {x} | \tilde {\mathbf {x}}) \propto p _ {\theta} (\mathbf {x}) p (\tilde {\mathbf {x}} | \mathbf {x}) \propto \exp \left(- E _ {\theta} (\mathbf {x}) - \frac {1}{2 \sigma^ {2}} \| \mathbf {x} - \tilde {\mathbf {x}} \| ^ {2}\right), \tag {3}
$$
where Bayes' rule is applied. The model parameter  $\theta$  is estimated by maximizing the log recovery likelihood, i.e.,  $\max_{\theta} \mathbb{E}_{\mathbf{x}, \tilde{\mathbf{x}}}[\log p_{\theta}(\mathbf{x}|\tilde{\mathbf{x}})]$ , for  $\mathbf{x} \sim p_{data}(\mathbf{x})$ ,  $\tilde{\mathbf{x}} \sim p(\tilde{\mathbf{x}}|\mathbf{x})$ , where  $p(\tilde{\mathbf{x}}|\mathbf{x}) = \mathcal{N}(\mathbf{x}, \sigma^2\mathbf{I})$ . This estimation is shown to be consistent under the usual assumptions (Appendix A.2 in [16]). DRL [16] uses a slightly modified perturbation  $\tilde{\mathbf{x}} = \sqrt{1 - \sigma^2}\mathbf{x} + \sigma\epsilon$  in training EBM, following [23]. This change introduces a minor modification of Eq. (3).
The recovery likelihood  $p_{\theta}(\mathbf{x}|\tilde{\mathbf{x}})$  (Eq. 3) is essentially a new EBM with the energy  $\tilde{E}_{\theta}(\mathbf{x}|\tilde{\mathbf{x}}) = E_{\theta}(\mathbf{x}) + ||\mathbf{x} - \tilde{\mathbf{x}}||^2 / 2\sigma^2$ . Therefore, the gradient  $\nabla_{\theta}\log p_{\theta}(\mathbf{x}|\tilde{\mathbf{x}})$  has the same form with the log-likelihood gradient of EBM (Eq. (2)), except that negative samples are drawn from  $p_{\theta}(\mathbf{x}|\tilde{\mathbf{x}})$  instead of  $p_{\theta}(\mathbf{x})$ :
$$
\nabla_ {\theta} \log p _ {\theta} (\mathbf {x} | \tilde {\mathbf {x}}) = - \nabla_ {\theta} E _ {\theta} (\mathbf {x}) + \mathbb {E} _ {\mathbf {x} ^ {-} \sim p _ {\theta} (\mathbf {x} | \tilde {\mathbf {x}})} [ \nabla_ {\theta} E _ {\theta} (\mathbf {x} ^ {-}) ], \tag {4}
$$
where  $\nabla_{\theta}\tilde{E}_{\theta}(\mathbf{x}|\tilde{\mathbf{x}}) = \nabla_{\theta}E_{\theta}(\mathbf{x})$  as the Gaussian noise is independent of  $\theta$ . Sampling from  $p_{\theta}(\mathbf{x}|\tilde{\mathbf{x}})$  is more stable than sampling from  $p_{\theta}(\mathbf{x})$ , because  $p_{\theta}(\mathbf{x}|\tilde{\mathbf{x}})$  is close to a uni-modal distribution concentrated near  $\mathbf{x}$  [16, 22]. However, it is questionable whether Gaussian noise is the most informative way to perturb data in a high-dimensional space.
3 Manifold Projection-Diffusion Recovery
We introduce the Manifold Projection-Diffusion Recovery (MPDR) algorithm, which trains EBM by recovering from perturbations that are more informative than Gaussian noise. Firstly, we propose Manifold Projection-Diffusion (MPD), a novel perturbation operation leveraging the low-dimensional structure inherent in the data. Then, we derive the recovery likelihood for MPD. We also provide an efficient sampling strategy and practical implementation techniques for MPDR. The implementation of MPDR is publicly available<sup>1</sup>.
<https://github.com/swyoon/manifold-projection-diffusion-recovery-pytorch>
Figure 2: Negative sample generation process in MDPR. Data  $\mathbf{x}$  (blue dots) are first projected and diffused on autoencoder manifold  $\mathcal{M}$  (gray line), resulting in  $\tilde{\mathbf{x}}$ . The latent chain starting from  $\tilde{\mathbf{x}}$  generates  $\mathbf{x}_0^- \in \mathcal{M}$ . The visible chain draws negative samples  $\mathbf{x}^-$  from  $p(\mathbf{x}|\tilde{\mathbf{x}})$  starting from  $\mathbf{x}_0^-$ .
Autoencoders MPDR assumes that a pretrained autoencoder approximating the training data manifold is given. The autoencoder consists of an encoder  $f_{e}:\mathcal{X}\to \mathcal{Z}$  and a decoder  $f_{d}:\mathcal{Z}\rightarrow \mathcal{X}$ , both assumed to be deterministic and differentiable. The latent space is denoted as  $\mathcal{Z}$ . The dimensionalities of  $\mathcal{X}$  and  $\mathcal{Z}$  are denoted as  $D_{\mathbf{x}}$  and  $D_{\mathbf{z}}$ , respectively. We assume  $f_{e}$  and  $f_{d}$  as typical deep neural networks jointly trained to reduce the training data's reconstruction error  $l(\mathbf{x})$ , where  $l(\mathbf{x})$  is typically an  $l_{2}$  error,  $l(\mathbf{x}) = ||\mathbf{x} - f_d(f_e(\mathbf{x}))||^2$ .
3.1 Manifold Projection-Diffusion
We propose a novel perturbation operation, **Manifold Projection-Diffusion** (MPD). Instead of adding Gaussian noise directly to a datum  $\mathbf{x} \xrightarrow{+\sigma\epsilon} \tilde{\mathbf{x}}$  as in the conventional recovery likelihood, MPD first encodes  $\mathbf{x}$  using the autoencoder and then applies a noise in the latent space:
$$
\mathbf {x} \xrightarrow {f _ {e}} \mathbf {z} \xrightarrow {+ \sigma \epsilon} \tilde {\mathbf {z}} \xrightarrow {f _ {d}} \tilde {\mathbf {x}}, \tag {5}
$$
where  $\mathbf{z} = f_{e}(\mathbf{x})$ ,  $\tilde{\mathbf{z}} = \mathbf{z} + \sigma \epsilon$ , and  $\tilde{\mathbf{x}} = f_{d}(\tilde{\mathbf{z}})$ . The noise magnitude  $\sigma$  is a predefined constant and  $\epsilon \sim \mathcal{N}(0, \mathbf{I})$ . The first step projects  $\mathbf{x}$  into  $\mathcal{Z}$ , and the second step diffuses the encoded vector  $\mathbf{z}$ . When decoded through  $f_{d}$ , the output  $\tilde{\mathbf{x}}$  always lies on the decoder manifold  $\mathcal{M} = \{\mathbf{x} | \mathbf{x} = f_{d}(\mathbf{z}), \mathbf{z} \in \mathcal{Z}\}$ , a collection of all possible outputs from the decoder  $f_{d}(\mathbf{z})$ . The process is visualized in the left panel of Fig. 1.
Since  $\mathbf{z}$  serves as a coordinate of  $\mathcal{M}$ , a Gaussian noise in  $\mathcal{Z}$  corresponds to perturbation of data along the manifold  $\mathcal{M}$ , reflecting more relevant modes of variation in data than Gaussian perturbation in  $\mathcal{X}$  (Fig. 1). Note that MPD reduces to the conventional Gaussian perturbation if we set  $\mathcal{Z} = \mathcal{X}$  and set  $f_{e}$  and  $f_{d}$  as identity mappings.
3.2 Manifold Projection-Diffusion Recovery Likelihood
We define the recovery likelihood for MPD as  $p_{\theta}(\mathbf{x}|\tilde{\mathbf{z}})$ , which is evaluated as follows:
$$
\begin{array}{l} p _ {\theta} (\mathbf {x} | \tilde {\mathbf {z}}) \stackrel {\text {(i)}} {\propto} p _ {\theta} (\mathbf {x}) p (\tilde {\mathbf {z}} | \mathbf {x}) = p _ {\theta} (\mathbf {x}) \left(\int p (\tilde {\mathbf {z}} | \mathbf {z}) p (\mathbf {z} | \mathbf {x}) \mathrm {d} \mathbf {z}\right) \stackrel {\text {(i i)}} {=} p _ {\theta} (\mathbf {x}) \left(\int p (\tilde {\mathbf {z}} | \mathbf {z}) \delta_ {f _ {e} (\mathbf {x})} (\mathbf {z}) \mathrm {d} \mathbf {z}\right) (6) \\ \stackrel {\text {(i i i)}} {=} p _ {\theta} (\mathbf {x}) p \left(\tilde {\mathbf {z}} \mid \mathbf {z} = f _ {e} (\mathbf {x})\right) \propto \exp \left(- E _ {\theta} (\mathbf {x}) + \log p \left(\tilde {\mathbf {z}} \mid \mathbf {z}\right)\right) \equiv \exp \left(- \tilde {E} _ {\theta} (\mathbf {x} | \tilde {\mathbf {z}})\right). (7) \\ \end{array}
$$
In (i), we apply Bayes' rule. In (ii), we treat  $p(\mathbf{z}|\mathbf{x})$  as  $\delta_{\mathbf{z}}(\cdot)$ , a Dirac measure on  $\mathcal{Z}$  at  $\mathbf{z} = f_e(\mathbf{x})$ , as our encoder is deterministic. Equality (iii) results from the property of Dirac measure  $\int f(\mathbf{x})\delta_{\mathbf{y}}(\mathbf{z})\mathrm{d}\mathbf{z} = f(\mathbf{y})$ . Using Gaussian perturbation simplifies the energy to  $\tilde{E}_{\theta}(\mathbf{x}|\tilde{\mathbf{z}}) = E_{\theta}(\mathbf{x}) + \frac{1}{2\sigma^2} ||\tilde{\mathbf{z}} -f_e(\mathbf{x})||^2$ .
Now,  $E_{\theta}(\mathbf{x})$  is trained by maximizing  $\log p_{\theta}(\mathbf{x}|\tilde{\mathbf{z}})$ . The gradient of  $\log p_{\theta}(\mathbf{x}|\tilde{\mathbf{z}})$  with respect to  $\theta$  has the same form as the conventional maximum likelihood case (Eq. 2) and the Gaussian recovery likelihood case (Eq. 4) but with a different negative sample distribution  $p_{\theta}(\mathbf{x}|\tilde{\mathbf{z}})$ :
$$
\nabla_ {\theta} \log p _ {\theta} (\mathbf {x} | \tilde {\mathbf {z}}) = - \nabla_ {\theta} E _ {\theta} (\mathbf {x}) + \mathbb {E} _ {\mathbf {x} ^ {-} \sim p _ {\theta} (\mathbf {x} | \tilde {\mathbf {z}})} [ \nabla_ {\theta} E _ {\theta} (\mathbf {x} ^ {-}) ]. \tag {8}
$$
Negative samples  $\mathbf{x}^{-}$  are drawn from  $p_{\theta}(\mathbf{x}|\tilde{\mathbf{z}})$  using MCMC.
Another possible definition for the recovery likelihood is  $p(\mathbf{x}|\tilde{\mathbf{x}})$ , which becomes equivalent to  $p(\mathbf{x}|\tilde{\mathbf{z}})$  when  $f_{d}$  is an injective function, i.e., no two instances of  $\tilde{\mathbf{z}}$  map to the same  $\tilde{\mathbf{x}}$ . However, if  $f_{d}$  is not injective, additional information loss may occur and becomes difficult to compute. As a result,  $p(\mathbf{x}|\tilde{\mathbf{z}})$  serves as a more general and also convenient choice for the recovery likelihood.
Algorithm 1 Manifold Projection-Diffusion Recovery
1: while converged do
2: Sample a mini-batch of positive samples  $\mathbf{x}$
3: Compute  $\tilde{\mathbf{z}} = f_{e}(\mathbf{x}) + \sigma \epsilon$
4: Sample  $\mathbf{z}^{-}$  from energy  $\tilde{H}_{\theta}(\mathbf{z}|\tilde{\mathbf{z}})$  using LMC on  $\mathcal{Z}$  starting from  $\tilde{\mathbf{z}}$
5: Sample  $\mathbf{x}^{-}$  from energy  $\tilde{E}_{\theta}(\mathbf{x}|\tilde{\mathbf{z}})$  using LMC on  $\mathcal{X}$  starting from  $\mathbf{x}_0^- = f_d(\mathbf{z}^-)$
6: Update  $\theta$  with the gradient:  $-\frac{\partial}{\partial\theta} E_{\theta}(\mathbf{x}) + \frac{\partial}{\partial\theta} E_{\theta}(\mathbf{x}^{-})$
7: end while
MPDR-S  $E_{\theta} = \text{Scalar}$  I1 Error: 0.018
Figure 3: 2D density estimation with a scalar energy function (left) and an autoencoder-based energy function (right).
MPDR-R  $E_{\theta} = AE$ $I_{1}$  Error: 0.012
**Consistency** Maximizing  $\log p(\mathbf{x}|\tilde{\mathbf{z}})$  leads to the consistent estimation of  $\theta$ . The consistency is shown by transforming the objective function into KL divergence. The assumptions resemble those for maximum likelihood estimation, including infinite data, identifiability, and a correctly specified model. Additionally, the autoencoder  $(f_e, f_d)$  and the noise magnitude  $\sigma$  should be independent of  $\theta$ , remaining fixed during training. The consistency holds for any choices of  $(f_e, f_d)$  and  $\sigma$ , as long as the recovery likelihood  $p_\theta(\mathbf{x}|\tilde{\mathbf{z}})$  is non-zero for any  $(\mathbf{x}, \tilde{\mathbf{z}})$ . Further details can be found in the Appendix.
3.3 Two-Stage Sampling
A default method for drawing  $\mathbf{x}^{-}$  from  $p_{\theta}(\mathbf{x}|\tilde{\mathbf{z}})$  is to execute LMC on  $\mathcal{X}$ , starting from  $\tilde{\mathbf{x}}$ , as done in DRL [16]. While this visible chain should suffice in principle with infinite chain length, sampling can be improved by leveraging the latent space  $\mathcal{Z}$ , as demonstrated in [24, 18, 20, 25]. For MPDR, we propose a latent chain, a short LMC operating on  $\mathcal{Z}$  that generates a better starting point  $\mathbf{x}_0^-$  for the visible chain. We first define the auxiliary latent energy  $\tilde{H}_{\theta}(\mathbf{z}) = \tilde{E}_{\theta}(f_d(\mathbf{z})|\tilde{\mathbf{z}})$ , the pullback of  $\tilde{E}_{\theta}(\mathbf{x}|\tilde{\mathbf{z}})$  through the decoder  $f_{d}(\mathbf{z})$ . Then, we run a latent LMC that samples from a probability density proportional to  $\exp(-\tilde{H}_{\theta}(\mathbf{z}))$ . The latent chain's outcome,  $\mathbf{z}_0^-$ , is fed to the decoder to produce the visible chain's starting point  $\mathbf{x}_0^- = f_d(\mathbf{z})$ , which is likely to have a smaller energy than the original starting point  $\tilde{E}_{\theta}(\mathbf{x}_0^-|\tilde{\mathbf{z}}) < \tilde{E}(\tilde{\mathbf{x}}|\tilde{\mathbf{z}})$ . Introducing a small number of latent chain steps improves anomaly detection performance in our experiments. A similar latent-space LMC method appears in [24] but requires a sample replay buffer not used in MPDR. Fig.2 illustrates the sampling procedure, and Algorithm1 summarizes the training algorithm.
3.4 Perturbation Ensemble
Using multiple perturbations simultaneously during MPDR training improves performance and stability, while not harming the consistency. Although the consistency of estimation is independent of the specifics of the perturbation, the perturbation design, such as  $(f_e, f_d)$  and  $\sigma$ , have a significant impact on performance in practice. Perturbation ensemble alleviates the risk of adhering to a single suboptimal choice of perturbation.
Noise Magnitude Ensemble We randomly draw the perturbation magnitude  $\sigma$  from a uniform distribution over a predefined interval for each sample in a mini-batch. In our implementation, we draw  $\sigma$  from the interval [0.05, 0.3] throughout all the experiments.
Figure 4: CIFAR-10 negative samples from a manifold ensemble. Negative samples become more visually complex as  $D_{\mathbf{z}}$  grows larger.
Manifold Ensemble We can also utilize multiple autoencoder manifolds  $\mathcal{M}$  in MPD. Given  $K$  autoencoders, a mini
batch is divided into  $K$  equally sized groups. For each group, negative samples are generated separately using the corresponding autoencoder. Only a minimal increase in training time is observed as  $K$  increases, since each autoencoder processes a smaller mini-batch. Memory overhead does exist but is manageable, since MPDR can achieve good performance with relatively smaller autoencoders.
Table 1: MNIST hold-out digit detection. Performance is measured in AUPR. The standard deviation of AUPR is computed over the last 10 epochs. The largest mean value is marked in bold, while the second-largest is underlined. Asterisks denote that the results are adopted from literature.
Hold-Out Digit	1	4	5	7	9
Autoencoder (AE)	0.062 ± 0.000	0.204 ± 0.003	0.259 ± 0.006	0.125 ± 0.003	0.113 ± 0.001
IGEBM	0.101 ± 0.020	0.106 ± 0.019	0.205 ± 0.108	0.100 ± 0.042	0.079 ± 0.015
MEG [26]*	0.281 ± 0.035	0.401 ± 0.061	0.402 ± 0.062	0.290 ± 0.040	0.324 ± 0.034
BiGAN-σ [27]*	0.287 ± 0.023	0.443 ± 0.029	0.514 ± 0.029	0.347 ± 0.017	0.307 ± 0.028
Latent EBM[20]*	0.336 ± 0.008	0.630 ± 0.017	0.619 ± 0.013	0.463 ± 0.009	0.413 ± 0.010
VAE+EBM [19]*	0.297 ± 0.033	0.723 ± 0.042	0.676 ± 0.041	0.490 ± 0.041	0.383 ± 0.025
NAE [24]	0.802 ± 0.079	0.648 ± 0.045	0.716 ± 0.032	0.789 ± 0.041	0.441 ± 0.067
MPDR-S (ours)	0.764 ± 0.045	0.823 ± 0.018	0.741 ± 0.041	0.857 ± 0.022	0.478 ± 0.048
MPDR-R (ours)	0.844 ± 0.030	0.711 ± 0.029	0.757 ± 0.024	0.850 ± 0.014	0.569 ± 0.036