Title: Energy-Based Models for Anomaly Detection: A Manifold Diffusion Recovery Approach
Venue: NeurIPS 2023
Accepted: yes
Keywords: Energy-based Models

Anomaly Detection

Generative Models

Out-of-Distribution Detection

Recovery Likelihood
Abstract: We present a new method of training energy-based models (EBMs) for anomaly detection that leverages low-dimensional structures within data. The proposed algorithm, Manifold Projection-Diffusion Recovery (MPDR), first perturbs a data point along a low-dimensional manifold that approximates the training dataset. Then, EBM is trained to maximize the probability of recovering the original data. The training involves the generation of negative samples via MCMC, as in conventional EBM training, but from a different distribution concentrated near the manifold. The resulting near-manifold negative samples are highly informative, reflecting relevant modes of variation in data. An energy function of MPDR effectively learns accurate boundaries of the training data distribution and excels at detecting out-of-distribution samples. Experimental results show that MPDR exhibits strong performance across various anomaly detection tasks involving diverse data types, such as images, vectors, and acoustic signals.

Energy-Based Models for Anomaly Detection: A Manifold Diffusion Recovery Approach

Abstract

We present a new method of training energy-based models (EBMs) for anomaly detection that leverages low-dimensional structures within data. The proposed algorithm, Manifold Projection-Diffusion Recovery (MPDR), first perturbs a data point along a low-dimensional manifold that approximates the training dataset. Then, EBM is trained to maximize the probability of recovering the original data. The training involves the generation of negative samples via MCMC, as in conventional EBM training, but from a different distribution concentrated near the manifold. The resulting near-manifold negative samples are highly informative, reflecting relevant modes of variation in data. An energy function of MPDR effectively learns accurate boundaries of the training data distribution and excels at detecting out-of-distribution samples. Experimental results show that MPDR exhibits strong performance across various anomaly detection tasks involving diverse data types, such as images, vectors, and acoustic signals.

1 Introduction

Unsupervised detection of anomalous data appears frequently in practical applications, such as industrial surface inspection [1], machine fault detection [2, 3], and particle physics [4]. Modeling the probability distribution of normal data p data ( x ) is a principled approach for anomaly detection [5]. Anomalies, often called outliers or out-of-distribution (OOD) samples, lie outside of the data distribution and can thus be characterized by low probability density under the distribution. However, many deep generative models that can evaluate the likelihood of data, including variational autoencoders (VAE; [6]), autoregressive models [7], and flow-based models [8] are known to perform poorly on popular anomaly detection benchmarks such as CIFAR-10 (in) vs SVHN (out), by assigning high likelihood on seemingly trivial outliers [9, 10].

On the other hand, deep energy-based models (EBMs) have demonstrated notable improvement in anomaly detection compared to other deep generative models [11]. While the specific reason for the superior performance of EBMs has not been formally analyzed, one probable factor is the explicit mechanism employed in EBM's maximum likelihood training that reduces the likelihood of negative samples. These negative samples are generated from the model distribution p θ ( x ) using Markov Chain Monte Carlo (MCMC). Since modern EBMs operate in high-dimensional data spaces, it is extremely difficult to cover the entire space with a finite-length Markov chain. The difficulty in

Young-Uk Jin

Frank C. Park ∗

𝒳

𝒵

𝐱

𝐳

෤ 𝐳

෤ 𝐱

ℳ

𝒇𝒆

𝒇𝒅

+𝝈𝝐

PerturbationonZ(MPD)

Perturbationon&(Gaoetal.,2021)

0.05

0.10

0.15

0.20

b

0.01

0.32

0.52

0.63

generating negative samples triggered the development of several heuristics such as using truncated chains [12], persistent chains [13], sample replay buffers [11], and data augmentation in the middle of the chain [14].

Instead of requiring a Markov chain to cover the entire space, EBM can be trained with MCMC running within the vicinity of each training datum. For example, Contrastive Divergence (CD; [15]) uses a short Markov chain initialized on training data to generate negative samples close to the training distribution. Diffusion Recovery Likelihood (DRL; [16]) enables MCMC to focus on the training data's neighborhood using a different objective function called recovery likelihood. Recovery likelihood p θ ( x | ˜ x ) is the conditional probability of training datum x given the observation of its copy ˜ x perturbed with a Gaussian noise. Training of DRL requires sampling from the recovery likelihood distribution p θ ( x | ˜ x ) , which is easier than sampling from the model distribution p θ ( x ) , as p θ ( x | ˜ x ) is close to uni-modal and concentrated near x . While CD and DRL significantly stabilize the negative sampling process, the resulting EBMs exhibit limited anomaly detection performance due to the insufficient coverage of negative samples in the input space.

In this paper, we present a novel training algorithm for EBM that does not require MCMC covering the entire space while achieving strong anomaly detection performance. The proposed algorithm, Manifold Projection-Diffusion Recovery (MPDR), extends the recovery likelihood framework by replacing Gaussian noise with Manifold Projection-Diffusion (MPD), a novel perturbation operation that reflects low-dimensional structure of data. Leveraging a separately trained autoencoder manifold, MPD firstly projects a training datum onto the manifold and perturbs the datum along the manifold. Compared to Gaussian noise, MPD reflects relevant modes of variation in data, such as change in colors or shapes in an image, as shown in Fig. 1. A MPD-perturbed sample serves as an informative starting point for MCMC that generates a negative sample, teaching EBM to discriminate challenging outliers that has partial similarity to training data.

Given an MPD-perturbed sample, EBM is trained by maximizing the recovery likelihood. We derive a simple expression for evaluating the recovery likelihood for MPD. We also propose an efficient two-stage MCMC strategy for sampling from the recovery likelihood, leveraging the latent space of the autoencoder. Additionally, we show that the model parameter estimation by MPDR is consistent under standard assumptions.

Compared to existing EBM training algorithms using an autoencoder-like auxiliary module [17, 18, 19, 20], MPDR has two advantages. First, MPDR allows the use of multiple autoencoders for training. The manifold ensemble technique stabilizes the training and improves anomaly detection performance by providing diverse negative samples. Second, MPDR achieves good anomaly detection performance with lightweight autoencoders. For example, autoencoders in our CIFAR-10 experiment has about only 1.5 ∼ 3 million parameters, which are significantly fewer than that of NV AE [21] ( ∼ 130 million) or Glow [8] ( ∼ 44 million), both of which are previously used in EBM training [17, 18].

Our contributions can be summarized as follows:

We propose MPDR, a novel method of using autoencoders for training EBM. MPDR is compatible with any energy functions and gives consistent density parameter estimation with any autoencoders.

We provide a suite of practical strategies for achieving successful anomaly detection with MPDR, including two-stage sampling, energy function design, and ensembling multiple autoencoders.

We demonstrate the effectiveness of MPDR on unsupervised anomaly detection tasks with various data types, including images, representation vectors, and acoustic signals.

2 Preliminaries

Energy-Based Models (EBM) Anenergy-based generative model, or an unnormalized probabilistic model, represents a probability density function using a scalar energy function E θ : X → R , where X denotes the domain of data. The energy function E θ defines a probability density function p θ through the following relationship:

The parameters θ can be learned by maximum likelihood estimation given iid samples from the underlying data distribution p data ( x ) . The gradient of the log-likelihood for a training sample x is well-known [15] and can be written as follows:

where x -denotes a 'negative" sample drawn from the model distribution p θ ( x ) . Typically, x -is generated using Langevin Monte Carlo (LMC). In LMC, points are arbitrarily initialized and then iteratively updated in a stochastic manner to simulate independent sampling from p θ ( x ) . For each time step t , a point x ( t ) is updated by x ( t +1) = x ( t ) -λ 1 ∇ x E θ ( x ( t ) ) + λ 2 ϵ ( t ) , for ϵ ( t ) ∼ N (0 , I ) . The step size λ 1 and the noise scale λ 2 are often tuned separately in practice. Since LMC needs to be performed in every iteration of training, it is infeasible to run negative sampling until convergence, and a compromise must be made. Popular heuristics include initializing MCMC on training data [15], using short-run LMC [12], and utilizing a replay buffer [13, 11].

Recovery Likelihood [22, 16] Instead of directly maximizing the likelihood (Eq. (2)), θ can be learned through the process of denoising data from artificially injected Gaussian noises. Denoising corresponds to maximizing the recovery likelihood p ( x | ˜ x ) , the probability of recovering data x from its noise-corrupted version ˜ x = x + σϵ , where ϵ ∼ N (0 , I ) .

where Bayes' rule is applied. The model parameter θ is estimated by maximizing the log recovery likelihood, i.e., max θ E x , ˜ x [log p θ ( x | ˜ x )] , for x ∼ p data ( x ) , ˜ x ∼ p (˜ x | x ) , where p (˜ x | x ) = N ( x , σ 2 I ) . This estimation is shown to be consistent under the usual assumptions (Appendix A.2 in [16]). DRL [16] uses a slightly modified perturbation ˜ x = √ 1 -σ 2 x + σϵ in training EBM, following [23]. This change introduces a minor modification of Eq. (3).

The recovery likelihood p θ ( x | ˜ x ) (Eq. 3) is essentially a new EBM with the energy ˜ E θ ( x | ˜ x ) = E θ ( x ) + || x -˜ x || 2 / 2 σ 2 . Therefore, the gradient ∇ θ log p θ ( x | ˜ x ) has the same form with the loglikelihood gradient of EBM (Eq. (2)), except that negative samples are drawn from p θ ( x | ˜ x ) instead of p θ ( x ) :

where ∇ θ ˜ E θ ( x | ˜ x ) = ∇ θ E θ ( x ) as the Gaussian noise is independent of θ . Sampling from p θ ( x | ˜ x ) is more stable than sampling from p θ ( x ) , because p θ ( x | ˜ x ) is close to a uni-modal distribution concentrated near x [16, 22]. However, it is questionable whether Gaussian noise is the most informative way to perturb data in a high-dimensional space.

3 Manifold Projection-Diffusion Recovery

We introduce the Manifold Projection-Diffusion Recovery (MPDR) algorithm, which trains EBM by recovering from perturbations that are more informative than Gaussian noise. Firstly, we propose Manifold Projection-Diffusion (MPD), a novel perturbation operation leveraging the low-dimensional structure inherent in the data. Then, we derive the recovery likelihood for MPD. We also provide an efficient sampling strategy and practical implementation techniques for MPDR. The implementation of MPDR is publicly available 1 .

pdata

Data

x

& Manifold

MPD-Perturbed

x

After Latent Chain

x

O

Negative Sample

x

Autoencoders MPDR assumes that a pretrained autoencoder approximating the training data manifold is given. The autoencoder consists of an encoder f e : X → Z and a decoder f d : Z → X , both assumed to be deterministic and differentiable. The latent space is denoted as Z . The dimensionalities of X and Z are denoted as D x and D z , respectively. We assume f e and f d as typical deep neural networks jointly trained to reduce the training data's reconstruction error l ( x ) , where l ( x ) is typically an l 2 error, l ( x ) = || x -f d ( f e ( x )) || 2 .

3.1 Manifold Projection-Diffusion

We propose a novel perturbation operation, Manifold Projection-Diffusion (MPD). Instead of adding Gaussian noise directly to a datum x + σϵ -- → ˜ x as in the conventional recovery likelihood, MPD first encodes x using the autoencoder and then applies a noise in the latent space:

where z = f e ( x ) , ˜ z = z + σϵ , and ˜ x = f d (˜ z ) . The noise magnitude σ is a predefined constant and ϵ ∼ N (0 , I ) . The first step projects x into Z , and the second step diffuses the encoded vector z . When decoded through f d , the output ˜ x always lies on the decoder manifold M = { x | x = f d ( z ) , z ∈ Z} , a collection of all possible outputs from the decoder f d ( z ) . The process is visualized in the left panel of Fig. 1.

Since z serves as a coordinate of M , a Gaussian noise in Z corresponds to perturbation of data along the manifold M , reflecting more relevant modes of variation in data than Gaussian perturbation in X (Fig. 1). Note that MPD reduces to the conventional Gaussian perturbation if we set Z = X and set f e and f d as identity mappings.

3.2 Manifold Projection-Diffusion Recovery Likelihood

We define the recovery likelihood for MPD as p θ ( x | ˜ z ) , which is evaluated as follows:

In (i), we apply Bayes' rule. In (ii), we treat p ( z | x ) as δ z ( · ) , a Dirac measure on Z at z = f e ( x ) , as our encoder is deterministic. Equality (iii) results from the property of Dirac measure ∫ f ( x ) δ y ( z )d z = f ( y ) . Using Gaussian perturbation simplifies the energy to ˜ E θ ( x | ˜ z ) = E θ ( x ) + 1 2 σ 2 || ˜ z -f e ( x ) || 2 .

Now, E θ ( x ) is trained by maximizing log p θ ( x | ˜ z ) . The gradient of log p θ ( x | ˜ z ) with respect to θ has the same form as the conventional maximum likelihood case (Eq. 2) and the Gaussian recovery likelihood case (Eq. 4) but with a different negative sample distribution p θ ( x | ˜ z ) :

Negative samples x -are drawn from p θ ( x | ˜ z ) using MCMC.

Another possible definition for the recovery likelihood is p ( x | ˜ x ) , which becomes equivalent to p ( x | ˜ z ) when f d is an injective function, i.e., no two instances of ˜ z map to the same ˜ x . However, if f d is not injective, additional information loss may occur and becomes difficult to compute. As a result, p ( x | ˜ z ) serves as a more general and also convenient choice for the recovery likelihood.

Algorithm 1 Manifold Projection-Diffusion Recovery

1: while converged do

2: Sample a mini-batch of positive samples x

3: Compute ˜ z = f e ( x ) + σϵ

4: Sample z -from energy ˜ H θ ( z | ˜ z ) using LMC on Z starting from ˜ z

5: Sample x -from energy ˜ E θ ( x | ˜ z ) using -

LMC on X starting from x 0 = f d ( z -)

6: Update θ with the gradient:

-∂ ∂θ E θ ( x ) + ∂ ∂θ E θ ( x -)

7: end while

MPDR-S

E

=

Scalar

l

1 Error: 0.018

MPDR-R

E

=

AE

l

1 Error: 0.012

Consistency Maximizing log p ( x | ˜ z ) leads to the consistent estimation of θ . The consistency is shown by transforming the objective function into KL divergence. The assumptions resemble those for maximum likelihood estimation, including infinite data, identifiablility, and a correctly specified model. Additionally, the autoencoder ( f e , f d ) and the noise magnitude σ should be independent of θ , remaining fixed during training. The consistency holds for any choices of ( f e , f d ) and σ , as long as the recovery likelihood p θ ( x | ˜ z ) is non-zero for any ( x , ˜ z ) . Further details can be found in the Appendix.

3.3 Two-Stage Sampling

A default method for drawing x -from p θ ( x | ˜ z ) is to execute LMC on X , starting from ˜ x , as done in DRL [16]. While this visible chain should suffice in principle with infinite chain length, sampling can be improved by leveraging the latent space Z , as demonstrated in [24, 18, 20, 25]. For MPDR, we propose a latent chain , a short LMC operating on Z that generates a better starting point x -0 for the visible chain. We first define the auxiliary latent energy ˜ H θ ( z ) = ˜ E θ ( f d ( z ) | ˜ z ) , the pullback of ˜ E θ ( x | ˜ z ) through the decoder f d ( z ) . Then, we run a latent LMC that samples from a probability density proportional to exp( -˜ H θ ( z )) . The latent chain's outcome, z -0 , is fed to the decoder to produce the visible chain's starting point x -0 = f d ( z ) , which is likely to have a smaller energy than the original starting point ˜ E θ ( x -0 | ˜ z ) < ˜ E (˜ x | ˜ z ) . Introducing a small number of latent chain steps improves anomaly detection performance in our experiments. A similar latent-space LMC method appears in [24] but requires a sample replay buffer not used in MPDR. Fig.2 illustrates the sampling procedure, and Algorithm1 summarizes the training algorithm.

3.4 Perturbation Ensemble

Using multiple perturbations simultaneously during MPDR training improves performance and stability, while not harming the consistency. Although the consistency of estimation is independent of the specifics of the perturbation, the perturbation design, such as ( f e , f d ) and σ , have a significant impact on performance in practice. Perturbation ensemble alleviates the risk of adhering to a single suboptimal choice of perturbation.

Noise Magnitude Ensemble We randomly draw the perturbation magnitude σ from a uniform distribution over a predefined interval for each sample in a mini-batch. In our implementation, we draw σ from the interval [0 . 05 , 0 . 3] throughout all the experiments.

Manifold Ensemble We can also utilize multiple autoencoder manifolds M in MPD. Given K autoencoders, a mini- batch is divided into K equally sized groups. For each group, negative samples are generated separately using the corresponding autoencoder. Only a minimal increase in training time is observed as K increases, since each autoencoder processes a smaller mini-batch. Memory overhead does exist but is manageable, since MPDR can achieve good performance with relatively smaller autoencoders.

Dz

32

64

128

An effective strategy is to utilize autoencoders with varying latent space dimensionalities D z . For high-dimensional data, such as images, M with different D z tends to capture distinct modes of variation in data. As depicted in Fig. 4, a perturbation on M with small D z corresponds to lowfrequency variation in X , whereas for M with large D z , it corresponds to higher-frequency variation. Using multiple D z 's in MPDR gives us more diverse x -and eventually better anomaly detection performance.

3.5 Energy Function Design

MPDR is a versatile training algorithm for general EBMs, compatible with various types of energy functions. The design of an energy function plays a crucial role in anomaly detection performance, as the inductive bias of an energy governs its behavior in out-of-distribution regions. We primarily explore two designs for energy functions: MPDR-Scalar ( MPDR-S ), a feed-forward neural network that takes input x and produces a scalar output, and MPDR-Reconstruction ( MPDR-R ), the reconstruction error from an autoencoder, E θ ( x ) = || x -g d ( g e ( x )) || 2 , for an encoder g e and a decoder g d . The autoencoder ( g e , g d ) is separate from the autoencoder ( f e , f d ) used in MPD. First proposed in [24], a reconstruction-based energy function has an inductive bias towards assigning high energy values to off-manifold regions (Fig. 3). Training such energy functions using conventional techniques like CD [15] or persistent chains [13, 11] is reported to be challenging [24]. However, MPDR effectively trains both scalar and reconstruction energies. Additionally, in Sec. 4.4, we demonstrate that MPDR is also compatible with an energy function based on a masked autoencoder.

4 Experiment

4.1 Implementation of MPDR

An autoencoder ( f e , f d ) is trained by minimizing the reconstruction error of the training data and remains fixed during the training of E θ ( x ) . When using an ensemble of manifolds, each autoencoder is trained independently. For anomaly detection, the energy value E θ ( x ) serves as an anomaly score which assigns a high value for anomalous x . All optimizations are performed using Adam with a learning rate of 0.0001. Each run is executed on a single Tesla

V100 GPU. Other details, including network architectures and LMC hyperparameters, can be found in the Appendix.

Spherical Latent Space In all our implementations of autoencoders, we utilize a hyperspherical latent space Z = S D z -1 [28, 29, 30]. The encoder output is projected onto S D z -1 via division by its norm before being fed into the decoder. Employing S D z -1 standardizes the length scale of

Z , allowing us to use the same value of σ across various data sets and autoencoder architectures. Meanwhile, the impact of S D z -1 on the reconstruction error is minimal.

Regularization For a scalar energy function, MPDR-S, we add L reg = E θ ( x ) 2 + E θ ( x -) 2 to the loss function, as proposed by [11]. For a reconstruction energy function, MPDR-R, we add L reg = E θ ( x -) 2 , following [24].

Scaling Perturbation Probability Applying regularization to an energy restricts its scale, causing a mismatch in scales between the two terms in the recovery likelihood (Eq. (7)). To remedy this mismatch, we heuristically introduce a scale factor γ < 1 to log p ( z | ˜ z ) , resulting in the modified recovery energy ˜ E ( γ ) θ ( x | ˜ z ) = E θ ( x ) + γ 2 σ 2 || ˜ z -f e ( x ) || 2 . We use γ = 0 . 0001 for all experiments.

4.2 2D Density Estimation

We show MPDR's ability to estimate multi-modal density using a mixture of eight circularly arranged 2D Gaussians (Fig. 2). We construct an autoencoder with S 1 latent space, which approximately captures the circular arrangement. The encoder and the decoder are MLPs with two layers of 128 hidden neurons. To show the impact of the design of energy functions, we implement both scalar energy and reconstruction energy. Three-hidden-layer MLPs are used for the scalar energy function, and the encoder and the decoder in the reconstruction energy function. Note that the network architecture of the reconstruction energy is not the same as the autoencoder used for MPD. The density estimation results are presented in Fig. 3. We quantify density estimation performance using l 1 error. After numerically normalizing the energy function and true density on the visualized bounded domain, we compute the l 1 error at 10,000 grid points. While both energies capture the overall landscape of the density, the reconstruction energy achieves a smaller error by suppressing probability density at off-manifold regions.

4.3 Image Out-of-Distribution Detection

MNIST Hold-Out Digit Detection Following the protocol of [26, 27], we evaluated the performance of MPDR on MNIST hold-out digit detection benchmark, where one of the ten digits in the MNIST dataset is considered anomalous, and the remaining digits are treated as in-distribution. This is a challenging task due to the diversity of the in-distribution data and a high degree of similarity between target anomalies and inliers. In particular, selecting digits 1, 4, 5, 7, and 9 as anomalous is known to be especially difficult. The results are shown in Table 1.

In MPDR, we use a single autoencoder ( f e , f d ) with D z = 32 . The energy function of MPDR-S is initialized from scratch, and the energy function of MPDR-R is initialized from the ( f e , f d ) used in MPD. Even without a manifold ensemble, MPDR shows significant improvement over existing algorithms, including ones leveraging an autoencoder in EBM training [19, 24]. The performance of MPDR is stable over a range of D z , as demonstrated in Appendix B.1.

MNIST OOD Detection To ensure that MPDR is not overfitted to the hold-out digit, we test MPDR in detecting five non-MNIST outlier datasets (Table 2). The results demonstrated that MPDR excels in detecting a wide range of outliers, surpassing the performance of naive algorithms such as autoencoders (AE) and scalar EBMs (IGEBM). Although MPDR achieves high overall detection performance, MPDR-R exhibits slightly weaker performance on EMNIST and Constant datasets. This can be attributed to the limited flexibility of the autoencoder-based energy function employed in MPDR-R.

CIFAR-10 OOD Detection We evaluate MPDR on the CIFAR-10 inliers, a standard benchmark for EBM-based OOD detection. The manifold ensemble includes three convolutional autoencoders, with D z = 32 , 64 , 128 . MPDR-S uses a ResNet energy function used in IGEBM [11]. MPDR-R adopts the ResNet-based autoencoder architecture used in NAE [24].

Table 3 compares MPDR to state-of-the-art EBMs. MPDR-R shows competitive performance across five OOD datasets, while MPDR-S also achieves high AUROC on SVHN and Constant. As both MPDR-R and NAE use the same autoencoder architecture for the energy, the discrepancy in performance can be attributed to the MPDR training algorithm. MPDR-R outperforms NAE on four out of five OOD datasets. Comparison between MPDR-S and DRL demonstrates the effectiveness of non-Gaussian manifold-aware perturbation used in MPDR. CLEL shows strong

overall performance, indicating that learning semantic information is important in this benchmark. Incorporating contrastive learning into MPDR framework is an interesting future direction.

CIFAR-100 OOD Detection on Pretrained Representation In Table 4, we test MPDR on OOD detection with CIFAR-100 inliers. To model a distribution of diverse images like CIFAR-100, we follow [34] and apply generative modeling in the representation space from a large-scale pretrained model. As we assume an unsupervised setting, we use pretrained representations without finetuning. Input images are transformed into 768D vectors by ViT-B_16 model [36]. ViT outputs are normalized with its norm and projected onto a hypersphere. We observe that adding a small Gaussian noise of 0.01 to training data improves stability of all algorithms. We use MLP for all energy functions and autoencoders. In MPDR, the manifold ensemble comprises three autoencoders with D z = 128 , 256 , 1024 . We also implement supervised baselines (MD [34] and RMD [35]). The spherical projection is not applied for MD and RMD to respect their original implementation.

MPDR demonstrates strong anomaly detection performance in the representation space, with MPDRS and MPDR-R outperforming IGEBM and AE/NAE, respectively. This success can be attributed to the low-dimensional structure often found in the representation space of in-distribution data, as observed in [37]. MPDR's average performance is nearly on par with supervised methods, MD and RMD, which utilize class information. Note that EBM inputs are no longer images, making previous EBM training techniques based on image transformation [14, 33] inapplicable.

Ablation Study Table 3 and 4 also report the results from single-manifold MPDR-R with varying latent dimensionality D z to show MPDR's sensitivity to a choice of an autoencoder manifold. Manifold ensemble effectively hedges the risk of relying on a single autoencoder which may not be optimal for detecting all types of outliers. Furthermore, manifold ensemble often achieves better AUROC score than MPDR with a single autoencoder. Additional ablation studies can be found in the Appendix. In Sec. B.2, we examine the sensitivity of MPDR to noise magnitude σ and the effectiveness of the noise magnitude ensemble. Also in Sec. B.4.4, we investigate the effect to scaling parameter γ and show that the training is unstable when γ is too large.

Multi-Class Anomaly Detection on MVTec-AD Using Pretrained Representation MVTec-AD [38] is also a popular anomaly detection benchmark dataset, containing images of 15 objects in their normal and defective forms. We follow the 'unified' experimental setting from [39]. Normal images from 15 classes are used as the training set, where no label information is provided. We use the same feature extraction procedure used in [39]. Each image is transformed to a 272 × 14 × 14 feature map using EfficientNet-b4. When running MPDR, we treat each spatial dimension of a feature map

as an input vector to MPDR, transforming the task into a 272D density estimation problem. We normalize a 272D vector with its norm and add a small white noise with a standard deviation of 0.01 during training. We use the maximum energy value among 14 × 14 vectors as an anomaly score of an image. For the localization task, we resize 14 × 14 anomaly score map to 224x224 image and compute AUROC for each pixel with respect to the ground true mask. We compare MPDR-R with UniAD [39] and DRAEM [40]. The results are shown in Table 12. Details on MPDR implementation can be found in B.5.

MPDR achieves AUROC scores that are very close to that of UniAD, outperforming DRAEM. The trend is consistent in both detection and localization tasks. The performance gap between MPDR and UniAD can be attributed to the fact that UniAD leverages spatial information of 14 × 14 feature map while our implementation of MPDR processes each pixel in the feature map separately.

4.4 Acoustic Anomaly Detection

We apply MPDR to anomaly detection with acoustic signals, another form of non-image data. We use DCASE 2020 Challenge Task 2 dataset [41], containing recordings from six different machines, with three to four instances per machine type. The task is to detect anomalous sounds from deliberately damaged machines, which are unavailable during training. We applied the standard preprocessing methods in[41] to obtain a 640-dimensional vector built up of consecutive mel-spectrogram features. Many challenge submissions exploit dataset-specific heuristics and ensembles for high performance, e.g., [42, 43]. Rather than competing, we focus on demonstrating MPDR's effectiveness in improving common approaches, such as autoencoders and Interpolation Deep Neural Networks (IDNN) [44]. IDNN is a variant of masked autoencoders which predicts the middle (the third) frame given the remaining frames. Similarly to autoencoders, IDNN predicts an input as anomaly when the prediction error is large. We first train AE and IDNN for 100 epochs and then apply MPDR by treating the reconstruction (or prediction) error as the energy. Manifold ensemble consists of autoencoders with D z = 32 , 64 , 128 . More training details can be found in the Appendix.

Table 5 shows that MPDR improves anomaly detection performance for both AE and IDNN. The known failure mode of AE and IDNN is producing unexpected low prediction (or reconstruction) error for anomalous inputs. By treating the error as the energy and applying generative training, the error in OOD region is increased through training, resulting in improved anomaly detection performance.

4.5 Anomaly Detection on Tabular Data

We test MPDR on ADBench [45], which consists 47 tabular datasets for anomaly detection. We compare MPDR with 13 baselines from ADBench. The baseline results are reproduced using the official ADBench repository. We consider the setting where the training split does not contain anomalies. For each dataset, we run each algorithm on three random splits and computed the AUROC on the corresponding test split. We then ranked the algorithms based on the averaged AUROC. In Table A, we present a summary table with the average rank across the 47 datasets.

We employ MPDR-R with a single manifold. For both the manifold and the energy, the same MLP-based autoencoder architecture is used. The encoder and the decoder are MLPs with two 1024-hidden-neuron layers. If the input space dimensionality is smaller than or equal to 100, the latent space dimensionality is set to have the same dimensionality D z = D x . If D x > 100 , we set D z as 70% of D x . We employ 1 step of Langevin Monte Carlo in the latent space and 5 steps

in the input space. The step sizes are 0.1 for the latent space and 10 for the input space. All the hyperparameters except D z are fixed across 47 datasets.

As shown in Table 6, MPDR achieves highly competitive performance on ADBench, demonstrating a higher average rank than the isolation forest, with some overlap of confidence interval. This result indicates that MPDR is a general-purpose anomaly detection algorithm capable of handling tabular data. We empirically observe that MPDR is most effective when AUROC from its autoencoder is low, meaning that the outliers are near the autoencoder manifold. When the autoencoder already achieves good AUROC, the improvement from MPDR training is often marginal.

5 Related Work

Leveraging autoencoders in training, MPDR is closely related to other EBM training algorithms that incorporate auxiliary modules. While the use of a single variational autoencoder in EBM training is explored in [18, 19, 46], MPDR employs multiple autoencoders. MPDR is also compatible with nonvariational autoencoders, offering greater flexibility. Normalizing flows [17, 25, 47] provide informative negative samples and a latent space for EBM training, but unlike MPDR, they do not exploit the low-dimensional structure of data. EBMs can also be trained with additional generator modules [48, 49], which plays a similar role to the decoder of an autoencoder. A contrastive representation learning module [33] improves EBM performance but relies on domain-specific data augmentations and is only demonstrated on images. In contrast, MPDR is applicable to a wider range of data, utilizing a more general assumption of low-dimensionality in data.

MPDR presents a novel objective function for EBM by extending recovery likelihood framework [22, 16]. Investigating its connection to other previously studied objective functions, such as f -divergence [50], pseudo-spherical scoring rule [51], divergence triangle [46], and Stein discrepancy [52], would also be an interesting future direction.

MPDR contributes to the field of anomaly detection [45, 39, 40, 53, 54]. The promising results from MPDR demonstrates that learning the distribution of data is a principled and effective approach for detecting anomalies [24].

6 Conclusion

Contributions We propose MPDR, a novel method of utilizing autoencoders for training EBM. An autoencoder in MPDR provides an informative starting point for MCMC, offers the latent space for the effective traversal of a high-dimensional space, and guides the drift of an MCMC sampler (Eq. 7). MPDR performs competitively on various anomaly detection benchmarks involving diverse types of data, contributing to the enhancement of generative modeling for anomaly detection with high-dimensional data.

Limitations First, the practical performance of MPDR is still sensitive to the specifics of autoencoders used in MPD. Second, some data, such as high-resolution images or texts, may not exhibit a clear low-dimensional structure. In these cases, MPDR may require a separate representation learning stage, as demonstrated in our CIFAR-100 experiment with ViT (Sec. 4.3). Third, MPDR is not optimized for generating samples starting from a simple distribution, such as a Gaussian, while DRL is. We may resort to longer-chain MCMC to generate samples from MPDR.

Acknowledgments and Disclosure of Funding

S. Yoon would like to thank Luigi Favaro and Tilman Plehn for their helpful discussions during the early development of the algorithm. S. Yoon is supported by a KIAS Individual Grant (AP095701) via the Center for AI and Natural Sciences at Korea Institute for Advanced Study. Y.-K. Noh was partly supported by NRF/MSIT (No. 2018R1A5A7059549, 2021M3E5D2A01019545) and IITP/MSIT (IITP-2021-0-02068, 2020-0-01373, RS-2023-00220628). This work was supported in part by IITPMSIT grant 2021-0-02068 (SNU AI Innovation Hub), IITP-MSIT grant 2022-0-00480 (Training and Inference Methods for Goal-Oriented AI Agents), KIAT grant P0020536 (HRD Program for Industrial Innovation), ATC+ MOTIE Technology Innovation Program grant 20008547, SRRC NRF grant RS-2023-00208052, SNU-AIIS, SNU-IAMD, SNU BK21+ Program in Mechanical Engineering, and SNU Institute for Engineering Research.

References

Paul Bergmann, Michael Fauser, David Sattlegger, and Carsten Steger. Mvtec ad - a comprehensive real-world dataset for unsupervised anomaly detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , June 2019.

Yuma Koizumi, Shoichiro Saito, Hisashi Uematsu, Noboru Harada, and Keisuke Imoto. Toyadmos: A dataset of miniature-machine operating sounds for anomalous sound detection. In 2019 IEEE Workshop on Applications of Signal Processing to Audio and Acoustics (WASPAA) , pages 313-317, 2019.

Harsh Purohit, Ryo Tanabe, Kenji Ichige, Takashi Endo, Yuki Nikaido, Kaori Suefusa, and Yohei Kawaguchi. Mimii dataset: Sound dataset for malfunctioning industrial machine investigation and inspection. arXiv preprint arXiv:2206.14225 , 2019.

Barry M Dillon, Luigi Favaro, Tilman Plehn, Peter Sorrenson, and Michael Krämer. A normalized autoencoder for lhc triggers. arXiv preprint arXiv:2206.14225 , 2022.

Christopher M Bishop. Novelty detection and neural network validation. IEE Proceedings-Vision, Image and Signal processing , 141(4):217-222, 1994.

Diederik P Kingma and Max Welling. Auto-encoding variational bayes. International Conference on Learning Representations (ICLR) , 2014.

Aaron van den Oord, Nal Kalchbrenner, and Koray Kavukcuoglu. Pixel recurrent neural networks. arXiv preprint arXiv:1601.06759 , 2016.

Durk P Kingma and Prafulla Dhariwal. Glow: Generative flow with invertible 1x1 convolutions. In Advances in Neural Information Processing Systems , pages 10215-10224, 2018.

Dan Hendrycks, Mantas Mazeika, and Thomas Dietterich. Deep anomaly detection with outlier exposure. In International Conference on Learning Representations , 2019.

Eric Nalisnick, Akihiro Matsukawa, Yee Whye Teh, Dilan Gorur, and Balaji Lakshminarayanan. Do deep generative models know what they don't know? In International Conference on Learning Representations , 2019.

Yilun Du and Igor Mordatch. Implicit generation and modeling with energy based models. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d Alche-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems 32 , pages 3608-3618. Curran Associates, Inc., 2019.

Erik Nijkamp, Mitch Hill, Song-Chun Zhu, and Ying Nian Wu. Learning non-convergent non-persistent short-run mcmc toward energy-based model. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems , volume 32, pages 5232-5242. Curran Associates, Inc., 2019.

Tijmen Tieleman. Training restricted boltzmann machines using approximations to the likelihood gradient. In Proceedings of the 25th international conference on Machine learning , pages 1064-1071, 2008.

Yilun Du, Shuang Li, B. Joshua Tenenbaum, and Igor Mordatch. Improved contrastive divergence training of energy based models. In Proceedings of the 38th International Conference on Machine Learning (ICML-21) , 2021.

Geoffrey E Hinton. Training products of experts by minimizing contrastive divergence. Neural computation , 14(8):1771-1800, 2002.

Ruiqi Gao, Yang Song, Ben Poole, Ying Nian Wu, and Diederik P Kingma. Learning energy-based models by diffusion recovery likelihood. In International Conference on Learning Representations , 2021.

Ruiqi Gao, Erik Nijkamp, Diederik P Kingma, Zhen Xu, Andrew M Dai, and Ying Nian Wu. Flow contrastive estimation of energy-based models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 7518-7528, 2020.

Zhisheng Xiao, Karsten Kreis, Jan Kautz, and Arash Vahdat. {VAEBM}: A symbiosis between variational autoencoders and energy-based models. In International Conference on Learning Representations , 2021.

Tian Han, Erik Nijkamp, Linqi Zhou, Bo Pang, Song-Chun Zhu, and Ying Nian Wu. Joint training of variational auto-encoder and latent energy-based model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , June 2020.

Bo Pang, Tian Han, Erik Nijkamp, Song-Chun Zhu, and Ying Nian Wu. Learning latent space energy-based prior model. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems , volume 33, pages 21994-22008. Curran Associates, Inc., 2020.

Arash Vahdat and Jan Kautz. NVAE: A deep hierarchical variational autoencoder. In Neural Information Processing Systems (NeurIPS) , 2020.

Yoshua Bengio, Li Yao, Guillaume Alain, and Pascal Vincent. Generalized denoising auto-encoders as generative models. In C.J. Burges, L. Bottou, M. Welling, Z. Ghahramani, and K.Q. Weinberger, editors, Advances in Neural Information Processing Systems , volume 26. Curran Associates, Inc., 2013.

Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems , volume 33, pages 6840-6851. Curran Associates, Inc., 2020.

Sangwoong Yoon, Yung-Kyun Noh, and Frank Park. Autoencoding under normalization constraints. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning , volume 139 of Proceedings of Machine Learning Research , pages 12087-12097. PMLR, 18-24 Jul 2021.

Erik Nijkamp, Ruiqi Gao, Pavel Sountsov, Srinivas Vasudevan, Bo Pang, Song-Chun Zhu, and Ying Nian Wu. MCMC should mix: Learning energy-based model with neural transport latent space MCMC. In International Conference on Learning Representations , 2022.

Rithesh Kumar, Sherjil Ozair, Anirudh Goyal, Aaron Courville, and Yoshua Bengio. Maximum entropy generators for energy-based models. arXiv preprint arXiv:1901.08508 , 2019.

Houssam Zenati, Chuan Sheng Foo, Bruno Lecouat, Gaurav Manek, and Vijay Ramaseshan Chandrasekhar. Efficient gan-based anomaly detection. arXiv preprint arXiv:1802.06222 , 2018.

Tim R Davidson, Luca Falorsi, Nicola De Cao, Thomas Kipf, and Jakub M Tomczak. Hyperspherical variational auto-encoders. In 34th Conference on Uncertainty in Artificial Intelligence 2018, UAI 2018 , pages 856-865. Association For Uncertainty in Artificial Intelligence (AUAI), 2018.

Jiacheng Xu and Greg Durrett. Spherical latent spaces for stable variational autoencoders. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing , pages 4503-4513, 2018.

Deli Zhao, Jiapeng Zhu, and Bo Zhang. Latent variables on spheres for autoencoders in high dimensions. arXiv , pages arXiv-1912, 2019.

Tim Salimans, Andrej Karpathy, Xi Chen, and Diederik P Kingma. Pixelcnn++: Improving the pixelcnn with discretized logistic mixture likelihood and other modifications. arXiv preprint arXiv:1701.05517 , 2017.

Will Grathwohl, Kuan-Chieh Wang, Joern-Henrik Jacobsen, David Duvenaud, Mohammad Norouzi, and Kevin Swersky. Your classifier is secretly an energy based model and you should treat it like one. In International Conference on Learning Representations , 2020.

Hankook Lee, Jongheon Jeong, Sejun Park, and Jinwoo Shin. Guiding energy-based models via contrastive latent variables. In International Conference on Learning Representations , 2023.

Stanislav Fort, Jie Ren, and Balaji Lakshminarayanan. Exploring the limits of out-of-distribution detection. Advances in Neural Information Processing Systems , 34:7068-7081, 2021.

Jie Ren, Stanislav Fort, Jeremiah Liu, Abhijit Guha Roy, Shreyas Padhy, and Balaji Lakshminarayanan. A simple fix to mahalanobis distance for improving near-ood detection. arXiv preprint arXiv:2106.09022 , 2021.

Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations , 2021.

Haoqi Wang, Zhizhong Li, Litong Feng, and Wayne Zhang. Vim: Out-of-distribution with virtual-logit matching. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 4921-4930, June 2022.

Paul Bergmann, Kilian Batzner, Michael Fauser, David Sattlegger, and Carsten Steger. The mvtec anomaly detection dataset: a comprehensive real-world dataset for unsupervised anomaly detection. International Journal of Computer Vision , 129(4):1038-1059, 2021.

Zhiyuan You, Lei Cui, Yujun Shen, Kai Yang, Xin Lu, Yu Zheng, and Xinyi Le. A unified model for multiclass anomaly detection. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems , volume 35, pages 4571-4584. Curran Associates, Inc., 2022.

Vitjan Zavrtanik, Matej Kristan, and Danijel Skoˇ caj. Draem - a discriminatively trained reconstruction embedding for surface anomaly detection. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) , pages 8330-8339, October 2021.

Yuma Koizumi, Yohei Kawaguchi, Keisuke Imoto, Toshiki Nakamura, Yuki Nikaido, Ryo Tanabe, Harsh Purohit, Kaori Suefusa, Takashi Endo, Masahiro Yasuda, and Noboru Harada. Description and discussion on dcase2020 challenge task2: Unsupervised anomalous sound detection for machine condition monitoring. arXiv preprint arXiv:2006.05822 , 2020.

Ritwik Giri, Srikanth V. Tenneti, Karim Helwani, Fangzhou Cheng, Umut Isik, and Arvindh Krishnaswamy. Unsupervised anomalous sound detection using self-supervised classification and group masked autoencoder for density estimation. Technical report, DCASE2020 Challenge, July 2020.

Pawel Daniluk, Marcin Gozdziewski, Slawomir Kapka, and Michal Kosmider. Ensemble of auto-encoder based systems for anomaly detection. Technical report, DCASE2020 Challenge, July 2020.

Kaori Suefusa, Tomoya Nishida, Harsh Purohit, Ryo Tanabe, Takashi Endo, and Yohei Kawaguchi. Anomalous sound detection based on interpolation deep neural network. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , pages 271-275, 2020.

Songqiao Han, Xiyang Hu, Hailiang Huang, Minqi Jiang, and Yue Zhao. Adbench: Anomaly detection benchmark. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems , volume 35, pages 32142-32159. Curran Associates, Inc., 2022.

Tian Han, Erik Nijkamp, Xiaolin Fang, Mitch Hill, Song-Chun Zhu, and Ying Nian Wu. Divergence triangle for joint training of generator model, energy-based model, and inferential model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , June 2019.

Jianwen Xie, Yaxuan Zhu, Jun Li, and Ping Li. A tale of two flows: Cooperative learning of langevin flow and normalizing flow toward energy-based model. In International Conference on Learning Representations , 2022.

Will Sussman Grathwohl, Jacob Jin Kelly, Milad Hashemi, Mohammad Norouzi, Kevin Swersky, and David Duvenaud. No {mcmc} for me: Amortized sampling for fast and stable training of energy-based models. In International Conference on Learning Representations , 2021.

Michael Arbel, Liang Zhou, and Arthur Gretton. Generalized energy based models. In International Conference on Learning Representations , 2021.

Lantao Yu, Yang Song, Jiaming Song, and Stefano Ermon. Training deep energy-based models with fdivergence minimization. In Hal Daumé III and Aarti Singh, editors, Proceedings of the 37th International Conference on Machine Learning , volume 119 of Proceedings of Machine Learning Research , pages 10957-10967. PMLR, 13-18 Jul 2020.

Lantao Yu, Jiaming Song, Yang Song, and Stefano Ermon. Pseudo-spherical contrastive divergence. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems , 2021.

Will Grathwohl, Kuan-Chieh Wang, Joern-Henrik Jacobsen, David Duvenaud, and Richard Zemel. Learning the stein discrepancy for training and evaluating energy-based models without sampling. In Hal Daumé III and Aarti Singh, editors, Proceedings of the 37th International Conference on Machine Learning , volume 119 of Proceedings of Machine Learning Research , pages 3732-3747. PMLR, 13-18 Jul 2020.

Zongyuan Huang, Baohua Zhang, Guoqiang Hu, Longyuan Li, Yanyan Xu, and Yaohui Jin. Enhancing unsupervised anomaly detection with score-guided network. IEEE Transactions on Neural Networks and Learning Systems , 2023.

Dmitrii Gavrilev and Evgeny Burnaev. Anomaly detection in networks via score-based generative models. In ICML 2023 Workshop on Structured Probabilistic Inference & Generative Modeling , 2023.

Tarin Clanuwat, Mikel Bober-Irizar, Asanobu Kitamoto, Alex Lamb, Kazuaki Yamamoto, and David Ha. Deep learning for classical japanese literature. arXiv preprint arXiv:1812.01718 , 2018.

Gregory Cohen, Saeed Afshar, Jonathan Tapson, and Andre Van Schaik. Emnist: Extending mnist to handwritten letters. In 2017 international joint conference on neural networks (IJCNN) , pages 2921-2926. IEEE, 2017.

Brenden M Lake, Ruslan Salakhutdinov, and Joshua B Tenenbaum. The omniglot challenge: a 3-year progress report. Current Opinion in Behavioral Sciences , 29:97-104, 2019.

Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms. arXiv preprint arXiv:1708.07747 , 2017.

A Krizhevsky. Learning multiple layers of features from tiny images. Master's thesis, University of Tront , 2009.

Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y Ng. Reading digits in natural images with unsupervised feature learning. In NIPS Workshop on Deep Learning and Unsupervised Feature Learning , 2011.

M. Cimpoi, S. Maji, I. Kokkinos, S. Mohamed, , and A. Vedaldi. Describing textures in the wild. In Proceedings of the IEEE Conf. on Computer Vision and Pattern Recognition (CVPR) , 2014.

Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of International Conference on Computer Vision (ICCV) , December 2015.

Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning , pages 1597-1607. PMLR, 2020.

Gareth O Roberts, Richard L Tweedie, et al. Exponential convergence of langevin distributions and their discrete approximations. Bernoulli , 2(4):341-363, 1996.

A Consistency of MPDR

Let us denote our model for recovery likelihood as p ( x | ˜ z ; θ ) . Weassume that this model is identifiable (different model parameters correspond to different distribution) and correctly specified (there exists θ ∗ such that p ( x | ˜ z ; θ ∗ ) = p data ( x | ˜ z ) ). We also assume that p ( x | ˜ z ; θ ) is non-zero for all x , ˜ z , and θ . Our objective is to max θ 1 N ∑ N i =1 log p ( x i | ˜ z i ; θ ) where ( x i , ˜ z i ) ∼ p ( x , ˜ z ) = p data ( x ) p (˜ z | x ) . In the limit of N → ∞ , the average converges to the expectation 1 N ∑ N i =1 log p ( x i | ˜ z i ; θ ) → E ( x , ˜ x ) [log p ( x | ˜ z ; θ )] . If we subtract E ( x , ˜ x ) [log p ( x | ˜ z ; θ ∗ )] which is constant with respect to θ , then the expression can be written with respect to KL divergence as follows:

The maximum of Eq. 10 is 0, as the minimum of KL divergence is 0. The maximum is achieved if and only if θ = θ ∗ . Note that p (˜ z ) is assumed to be constant with respect to θ .

Since we did not rely on the specifics of how a perturbation p (˜ z | x ) is actually performed, this consistency result holds for any choices of the encoder f e and the noise magnitude σ , as long as the recovery likelihood p ( x | ˜ z ; θ ) remains non-zero for all x , ˜ z , and θ .

B Experimental Details and Additional Results

In this section, we provide detailed information on how each experiment is conducted and also provide some additional supporting experimental results. The hyperparameters related to LMC is summarized in Table 7. ConvNet architectures are provided in Table 8. The contents are organized by the training dataset.

B.1 MNIST

B.1.1 Datasets

All input images are 28 × 28 grayscale, and each pixel value is represented as a floating number between [0 , 1] . Models are trained on the training split of MNIST 2 , excluding the digit designated to be held-out. The training split contains 60,000 images, and the hold-out procedure reduces the training set to approximately 90%. We evaluate the models on the test split of MNIST, which contains a total of 10,000 images. For non-MNIST datasets used in evaluation, we only use their test split. All non-MNIST datasets used in the experiment also follow the 28 × 28 grayscale format, similar to MNIST.

KMNIST (KMNIST-MNIST) [55] 3 contains Japanese handwritten letters, pre-processed into the same format as MNIST. The license of KMNIST is CC BY-SA 4.0.

EMNIST (EMNIST-Letters) [56] contains grayscale handwritten English alphabet images. Its test split contains 20,800 samples. No license information is available.

For the Omniglot [57] dataset, the evaluation set is used. The set consists of 13,180 images. No license information is available.

We use the test split of FashionMNIST [58], which contains 10,000 images. The dataset is made public under the MIT license.

Constant dataset is a synthetic dataset that contains 28 × 28 images, where all pixels have the same value. The value is randomly drawn from a uniform distribution over [0 , 1] . We use 4,000 constant images.

B.1.2 Autoencoder Implementation and Training

The encoder and the decoder used in MPDR, f e and f d , have the architecture of MNIST Encoder and MNIST Decoder, provided in Table 8, respectively. We use the spherical latent space S D z -1 where D z = 32 for the main experiment. The autoencoder is trained to minimize the l 2 reconstruction error of the training data for 30 epochs with Adam of learning rate 1 × 10 -4 . The batch size is 128 and no data augmentation is applied. The l 2 norm of the encoder is regularized with the coefficient of 1 × 10 -4 . The same autoencoder is used for both MPDR-S and MPDR-R.

B.1.3 MPDR Implementation and Training

In MPDR-S, the energy function E θ has the architecture of MNIST Encoder (Table 8) with D z = 1 . The network is randomly initialized from PyTorch default setting and the spectral normalization is applied. The energy function is trained with MPDR algorithm for 50 epochs. The learning rate is 1 × 10 -4 . The batch size is 128. The perturbation probability scaling factor γ for the visible LMC chain is set to zero. For an image-like data such as MNIST, the gradient of perturbation probability ∇ x ( || ˜ z -f e ( x ) || 2 ) introduces non-smooth high-frequency patterns resembling adversarial noise, harming the stability of the training. Therefore, we only use non-zero γ in latent chains in MNIST and CIFAR-10 experiments.

For MPDR-R, the energy function is initialized from ( f e , f d ) and then trained through MPDR algorithm. The learning rate is set to 1 × 10 -5 . All the other details are identical to the MPDR-S case.

B.2 Sensitivity to σ

As an ablation study for noise magnitude ensemble, we perform single-noise-magnitude experiment for MPDR and examine MPDR's sensitivity to the noise magnitude σ . Weevaluate the OOD detection performance of MPDR-S on the MNIST hold-out digit 9 setting with varying values of σ . Results are shown in Table 9.

The choice of σ has a significant impact on MPDR's OOD detection performance. In Table 9, σ = 0 . 01 gives poor OOD detection performance, particularly with respect to the hold-out digit. The performance generally improves as σ grows larger, but a large σ is not optimal for detecting EMNIST.

Selecting a single optimal σ will be very difficult, and therefore, we employ noise magnitude ensemble which can hedge the risk of choosing a suboptimal value for σ .

B.2.1 Sensitivity to D z

As an ablation study for manifold ensemble, we investigate the sensitivity of MPDR to the choice of the latent space dimensionality of the autoencoder. We evaluate the OOD detection performance of MPDR-S on the MNIST hold-out digit 9 setting with varying values of D z . Table 10 presents the results. Consequently, MPDR runs stably for a large range of D z , producing decent OOD performance. One hypothesis is that, for MNIST, it is relatively easy for these autoencoders to capture the manifold structure of MNIST sufficiently well.

Meanwhile, we do observe that the choice of D z affects OOD performance in an interesting way. Increasing D z enhances AUPR for certain OOD datasets but deteriorates AUPR for others. For example, AUPR of Omniglot is increased with larger D z , but AUPR of EMNIST, FashionMNIST, and Constant dataset decreases. No single autoencoder is optimal for detecting all outlier datasets. This observation motivates the use of manifold ensemble, employed in non-MNIST MPDR experiments.

B.2.2 Note on Reproduction

For an autoencoder-based outlier detector, denoted as 'AE" in Table 1, we use the same autoencoder used in MPDR with D z .

NAE is reproduced based on its public code base 4 .

We tried to reproduce DRL on MNIST but failed to train it stably. The original paper and the official code base also does not provide a guidance on training DRL on MNIST.

B.3 CIFAR-10 OOD Detection

B.3.1 Datasets

All data used in CIFAR-10 experiment are in the 32 × 32 RGB format. Models are only trained on CIFAR-10 training set, and evaluated on the testing set of each dataset.

CIFAR-10 [59] contains 60,000 training images and 10,000 testing images. Models are trained on the training set. We don't use its class information, as we consider only unsupervised setting. No license information available.

SVHN [60] is a set of digit images. Its test set containing 26,032 is used in the experiment. The dataset is non-commercial use only.

Texture [61] dataset, also called Describable Textures Dataset (DTD), contains 1,880 test images. The images are resized into 32 × 32. No license information available.

CelebA [62] 5 is a dataset of cropped and aligned human face images. The test set contains 19,962 images. The dataset is for non-commercial research purposes only. We center-crop each image into 140 × 140 and then resize it into 32 × 32 .

Constant dataset is a synthetic dataset that contains 4,000 32 × 32 RGB monochrome image. All 32 × 32 pixels have the same RGB value which is drawn uniform-randomly from [0 , 1] 3 .

CIFAR-100 [59] contains 60,000 training images and 10,000 testing images. No license information available.

B.3.2 Autoencoder Implementation and Training

The autoencoders for CIFAR-10 experiment have an architecture of 'CIFAR-10 Encoder 1" and 'CIFAR-10 Decoder 1" in Table 8 with D z = 32 , 64 , 128 . Each autoencoder is trained for 40 epoch with learning rate 1 × 10 -4 and batch size 128. During training the autoencoders, we apply the following data augmentation operations: random horizontal flipping with the probability of 0.5, random resize crop with the scale parameter [0 . 08 , 1] with the probability 0.2, color jittering with probability of 0.2, random grayscale operation with the probability 0.2. The color jittering parameters are the same with the one used in SimCLR [63] (brightness 0.8, contrast 0.8, saturation 0.8, hue 0.4, with respect to torchvision.transorms.ColorJitter implementation). The l 2 norm of an encoder is regularized with the coefficient of 0.00001. The same autoencoder is used for both MPDR-S and MPDR-R. To boost the performance, we heuristically introduce sample replay buffer for negative samples and apply data augmentation in the middle of LMC.

B.3.3 MPDR Implementation and Training

The energy function in MPDR-S is 'CIFAR-10 Encoder 2" with D z = 1 . The energy function in MPDR-R is 'CIFAR-10 Encoder 2" and 'CIFAR-10 Decoder 2" with D z = 1 . In MPDR-R, the energy function is pre-trained by minimizing the reconstruction error of the training data for 40 epochs. Only random horizontal flip is applied and no other data augmentation is used. Similarly to the MNIST case, we set γ = 0 for the visible LMC chain.

B.3.4 Note on Reproduction

For Improved CD, we train a CIFAR-10 model from scratch using the training script provided by the authors without any modification 6 . We use the model with the best Inception Score to compute AUROC scores.

For DRL, we use the official checkpoint for T6 CIFAR-10 model 7 and compute its energy to perform OOD detection.

Also for NVAE, we use the official CIFAR-10 checkpoint provided in the official repository 8 . We use negative log-likelihood as the outlier score. Due to computational budget constraint, we could only set the number of importance weighted sample to be one -num_iw_samples=1 .

As in MNIST, we use the official CIFAR-10 checkpoint of NAE provided by the authors.

B.4 CIFAR-100 OOD Detection on Pretrained Representation

B.4.1 Datasets

CIFAR-100, CIFAR-10, SVHN, and CelebA datasets are used and are described in the previous section. Each image is resized to 224 × 224 and fed to ViT-B_16 to produce a 768-dimensional vector. MD and RMD operate with this vector. For other methods, the 768D vector is projected onto a hypersphere.

B.4.2 Autoencoder Implementation and Training

Each encoder and decoder is an MLP with two hidden layers where each layer contains 1024 hidden neurons. The leaky ReLU activation function is used in all hidden layers. We use D z = 128 , 256 , 1024 . The autoencoders are trained to minimize the reconstruction error of the training data. During training, the Gaussian noise with the standard deviation of 0.01 is added to each training sample. The l 2 norm of the encoder's weights are regularized with the coefficient of 1 × 10 -6 .

B.4.3 MPDR Implementation and Training

The energy functions are also MLPs. The energy function of MPDR-S has the same architecture as the encoder of the autoencoder with D z = 1 . The energy function of MPDR-R is an autoencoder with the latent dimensionality of 1024. The energy functions are trained for 30 epochs with the learning rate of 1 × 10 -4 .

B.4.4 Sensitivity to γ

We examine how γ affects MPDR's performance. As seen in Table 11, MPDR shows the best performance on the small γ regime, roughly from 0.0001 to 0.001. Setting too large γ is detrimental for the performance and often even incurs training instabilities. It is interesting to note that γ = 0 also gives a decent result. One possible explanation is that γ = 0 reduces the negative sample distribution p θ ( x | ˜ z ) to the model distribution p θ ( x ) , which is still a valid negative sample distribution for training an EBM.

B.5 MVTec-AD Experiment

We use MPDR-R with a single manifold (i.e., without the manifold ensemble). Both the manifold and the energy are a fully connected autoencoder of 256D spherical latent space. For input-space Langevin Monte Carlo, the number of MCMC steps, the step size, and the noise standard deviation are 10, 0.1, and 0.1, respectively. No latent chain is used. The manifold is trained for 200 epochs with Adam of learning rate 1e-3, and the energy is trained for 20 epochs with Adam of learning rate 1e-4.

B.6 Acoustic Anomaly Detection

B.6.1 Dataset

The dataset consists of audio recordings with a duration of approximately 10 seconds, obtained through a single channel and downsampled to 16kHz. Each recording includes both the operational sounds of the target machine and background noise from the surrounding environment. The addition of noise was intended to replicate the conditions of real-world inspections, which often occur in noisy factory environments. The dataset covers six types of machinery, including four sampled from the MIMII Dataset (i.e., valve, pump, fan, and slide rail) and two from the ToyADMOS dataset (i.e., toy-car and toy-conveyor).

B.6.2 Preprocessing

We follow the standard preprocessing scheme used in the challenge baseline and many of challenge participants. The approach involves the use of Short Time Fourier Transform (STFT) to transform each audio clip into a spectrogram, which is then converted to Mel-scale. We set the number of Mel bands as 128, the STFT window length as 1024, and the hop length (i.e., the number of audio samples between adjacent STFT columns) as 512. This configuration results in a spectrogram with 128 columns that represented the number of Mel bands. To construct the final spectrogram, the mel spectra of five consecutive frames are collected and combined to form a single row, resulting in a spectrogram with 640 columns. Each row of the spectrogram is sampled and used as input to the models under investigation, with a batch size of 512, meaning that 512 rows were randomly selected from the spectrogram at each iteration. We standardize all data along the feature dimension to zero mean and unit variance.

B.6.3 Performance Measure

We refer to the scoring method introduced in [41] and use the area under the receiver operating characteristic curve (AUROC) and partial-AUROC ( p AUROC) as a quantitative measure of performance. pAUROC measures the AUC over the area corresponding to the false positive rate from 0 to a reasonably small value p , which we set in all our experiments as 0.1. Each measure is defined as follows:

where { x -i } N -i =1 and { x + j } N + j =1 are normal and anomalous samples, respectively, ⌊·⌋ indicates the flooring function, and H ( · ) represents a hard-threshold function whose output is 1 for a positive argument and 0 otherwise.

B.6.4 Implementation of Models

Weutilize a standard autoencoder model provided by DCASE 2020 Challenge organizers and compare it to our approach. The model comprises a symmetrical arrangement of fully-connected layers in the encoder and decoder; 5 fully connected hidden layers in the input and 5 in the output, with 128-dimensional hidden layers and 32-dimensional latent space. In addition, we incorporate the IDNN model, which predicts the excluded frame using all frames except the central one instead of reconstructing the entire sequence. The IDNN model outperforms the autoencoder on non-stationary sound signals, i.e., sounds with short durations. The model consists of a encoder and decoder, which is similar to the components of an autoencoder but has an asymmetric layout; 3 fully-connected hidden layers that contract in dimension (64, 32, 16) are used in the encoder and 3 layers that expand in dimension (16, 32, 64) compose the decoder. The architectural design of each model follows the specifications outlined in their respective papers.

The MPDR models used for the experiment, MPDR-R and MPDR-IDNN, consists of an ensemble of autoencoders and an energy function, where the terms "R" and "IDNN" specify whether an autoencoder of IDNN is used to compute the energy, respectively. Each autoencoder consists of an encoder, spherical embedding layer, decoder; the encoder and decoder consist of 3 fully-connected layers each, with 1024-dimensional hidden space and a latent space with a dimension chosen among 32, 64, or 128. The autoencoder energy function used in the experiment is built using layers 5 fully-connected encoder layers and 5 decoder layers, whose hidden layers have 128 nodes and latent layer is composed of 32 nodes. The layers of IDNN are indentical to that of the autoencoder version, except for the input and output layers whose dimensions differ and add up to the total number of sampled frames. LMC hyperparameters used are listed in Table 7.

C Empirical Guidelines for Implementing MPDR

Here, we present empirical tips and observations we found useful in achieving competitive OOD detection performance with MPDR. The list also includes heuristics that did not work when we tried.

The training progress can be monitored by measuring an OOD detection metric (i.e., AUROC) computed between test (or validation) inliers and synthetic samples uniformly sampled over the autoencoder manifold M . During a successful and stable run, this score tends to increase smoothly.

Metropolis-style adjustment for LMC [64] did not improve OOD performance.

The choice of activation function in the energy function affects the OOD detection performance significantly. We found that ReLU and LeakyReLU provide good results in general.

In image datasets, stopping the training of the autoencoder manifold before convergence improves OOD detection performance.

A longer Markov chains, both visible and latent, do not always lead to better OOD detection performance.

A larger autoencoder does not always lead to better OOD detection performance.

A larger energy function does not always lead to better OOD detection performance.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors introduce a novel algorithm, Manifold Projection-Diffusion Recovery (MPDR), for training energy-based models (EBMs) that improve the performance of anomaly detection tasks. These tasks are highly relevant in real-world applications like industrial surface inspection, machine fault detection, and particle physics.\n\nUnlike conventional EBM training methods, MPDR harnesses low-dimensional structures in the data to generate more informative negative samples. It works by initially perturbing a data point along a manifold approximating the training dataset and then trains the EBM to maximize the probability of recovering the original data.\n\nA significant aspect of this new method is its use of Manifold Projection-Diffusion (MPD), which replaces Gaussian noise with perturbations reflecting the data's low-dimensional structure. This approach provides more meaningful insights into variations within the data.\n\nThe authors show that MPDR provides consistent density parameter estimation under standard assumptions, is compatible with any energy function, and can work with multiple autoencoders - an advantage over existing algorithms. Moreover, it demonstrates good results even with lightweight autoencoders, making it computationally efficient.\n\nThe paper also presents practical strategies for deploying MPDR, such as two-stage sampling, energy function design, and ensemble techniques using multiple autoencoders.\n\nThrough experimental testing on various data types, including images, representation vectors, and acoustic signals, the authors demonstrate that MPDR significantly improves unsupervised anomaly detection tasks, outperforming other deep generative models.\n\nSTRENGTHS:\n**Originality:** \n\nThe paper is highly original in its formulation and approach to training energy-based models (EBMs) for anomaly detection. The development of the Manifold Projection-Diffusion Recovery (MPDR) represents a creative combination of existing methodologies, such as the use of autoencoders for efficient training and manifold projections for capturing low-dimensional data structures. The method of perturbing data points along a low-dimensional manifold that approximates the training dataset and then training the EBM to maximize the recovery of the original data is a fundamentally new approach.\n\n**Quality:** \n\nThe quality of the paper is commendable. The authors present a clear problem statement, propose a novel solution, and provide empirical evidence supporting their claims. They also delve into the theoretical backing of the proposed method, offering a consistent density parameter estimation under standard assumptions. The paper includes detailed experimental results, highlighting the strength of MPDR across diverse anomaly detection tasks and data types.\n\n**Clarity:** \n\nThe exposition in the paper is clear and well-organized. The authors have done an excellent job explaining complex concepts and methodologies, which makes the paper accessible even to readers who may not be experts in the field. The use of illustrative figures and well-explained algorithms further enhances the clarity of the paper.\n\n**Significance:** \n\nThe significance of this work is substantial, given the wide range of practical applications of anomaly detection. By providing a more efficient and effective way to train EBMs, MPDR could significantly improve performance in areas like industrial surface inspection, machine fault detection, and particle physics. Furthermore, the ability of MPDR to perform effectively with lightweight autoencoders means it can be used in scenarios where computational resources are limited, making it relevant to a broader audience.\n\nWEAKNESSES:\nWhile the paper presents a compelling new approach, here are a few areas that could be addressed or clarified further:\n\n**Theoretical Analysis:** While the authors provided a theoretical justification for consistent density parameter estimation under standard assumptions, it would be beneficial to include more analysis of the proposed method's convergence properties. Understanding how the choice of manifold affects the convergence and stability of learning would also be useful.\n\n**Comparison with Other Methods:** The paper could benefit from a more comprehensive comparison with other state-of-the-art methods for anomaly detection. Not only should this include direct quantitative comparisons on common datasets, but also qualitative discussions about when and why one might prefer the proposed method over others.\n\n**Parameter Sensitivity:** It is not clear how sensitive the results are to the choice of parameters within the MPDR framework. It would be beneficial for practical applications to know how much tuning is needed to achieve optimal performance and how robust the method is to variations in these parameters.\n\n**Real-World Applications:** While the experiment demonstrates the effectiveness of the proposed algorithm in various data types like images, vectors, and acoustic signals, applying the model to real-world datasets and providing case studies can strengthen the paper. This will help readers understand its practical implications better.\n\n**Computational Complexity:** The paper mentions that MPDR performs well with lightweight autoencoders, which indicates computational efficiency. However, a more explicit discussion or analysis of the computational complexity of the proposed method, including both training time and inference time, would provide valuable information to practitioners considering this method.\n\nQUESTIONS:\n1. **Theoretical Analysis:** Could the authors provide a more rigorous analysis of the convergence properties of their proposed method? Specifically, how does the choice of manifold affect the stability and speed of learning in MPDR?\n\n2. **Comparison with Other Methods:** It would be beneficial to see a wider comparison with other state-of-the-art anomaly detection methods. Could the authors elaborate on why one might choose MPDR over other established methods in specific scenarios?\n\n3. **Parameter Sensitivity:** How sensitive is the MPDR algorithm to the initial choice of parameters? Is there a recommended procedure for parameter tuning, or guidelines that could assist users in achieving optimal performance?\n\n4. **Real-World Applications:** Could the authors possibly demonstrate the application of their model on real-world datasets or provide case studies? This could help showcase the practical implications of the proposed method.\n\n5. **Computational Complexity:** The paper mentions that MPDR can work well even with lightweight autoencoders, but could you please clarify further on its computational complexity, training time, and inference time? Would there be any scalability issues when applying this method to larger datasets?\n\nLooking forward to the authors' response to these points.\n\nLIMITATIONS:\nFrom the information provided, it does not appear that the authors have explicitly discussed the limitations and potential negative societal impacts of their work. Therefore, here are some suggestions for addressing these points:\n\n**Limitations:**\n \n1. **Robustness to Noise:** How well does the MPDR framework handle noise in the data? Real-world data often contain a significant amount of noise, which may distort the underlying manifold structure that MPDR relies on.\n\n2. **Scalability:** The scalability of the model hasn't been addressed. Can the method be applied efficiently to very large datasets? What would be the computational requirements in such cases?\n\n3. **Multimodality:** How effectively can MPDR handle multimodal or highly dimensional distributions? This is a common challenge in many real-world anomaly detection tasks.\n\n**Potential Negative Societal Impacts:**\n\nWhile this study primarily focuses on improving anomaly detection methods, which generally have positive implications (e.g., defect detection in manufacturing, early detection of diseases, etc.), any technology has the potential to be misused.\n\n1. **Privacy Concerns:** Anomaly detection tools can potentially be used to identify outliers or anomalies in personal behavior or characteristics, leading to potential privacy concerns if misused, especially in contexts like surveillance or social profiling.\n\n2. **False Positives/Negatives:** In critical applications, false positives or negatives can have serious repercussions. For instance, in health care, a false positive might cause unnecessary worry or treatment, whereas a false negative could delay necessary intervention.\n\nIt would be beneficial to see the authors address these potential issues and discuss how they might be mitigated.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 1, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-03"}

comments: SUMMARY:
Paper proposes MPDR, a novel method of using auto-encoders for training EBM. 
Some practical techniques are introduced.
Extensive numerical experiments are done.

STRENGTHS:
Numerical experiments cover a large scope of benchmarks. And it shows superiority on most benchmarks.

WEAKNESSES:
No theoretical guarantee is provided.
And it doesn't show dominant superiority on some datasets.

QUESTIONS:
Can we have some theoretical analysis on when proposed method has significant advantage, when not? Or is the proposed method better in general and some loss are just random?

LIMITATIONS:
No negative societal impact is seen.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2023-07-05

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces an energy-based model based on the manifold of low-dimensional data. To train the EBMs, this paper takes the idea of maximum recovery likelihood and adds a layer of autoencoder to approximate the low-dimensional manifold representing the data. This introduces perturbation along the low-dim manifold representing the dataset. Simulation results are provided to show the performance.\n\nSTRENGTHS:\n- It is easy to follow. \n- Sufficient literature review.\n\nWEAKNESSES:\n- The novelty of this paper is moderate at best. It only adds a deterministic encoder/decoder to the original problem of likelihood recovery. \n- The accuracy of the manifold approximation is as good as the autoencoder approximation of the dataset. \n- The generation of the negative samples concentrated neat the manifold could potentially introduce biases. \n- lack of comparison to some recent advancements in the topic such as:\n   * Anomaly Detection in Networks via Score-Based Generative Models by Gavrilev et. al, 2023. \n   * Enhancing Unsupervised Anomaly Detection with Score-Guided Network by Huang et. al, 2022.\n\nQUESTIONS:\n- How can we mitigate these biases to ensure the trained energy-based model generalizes well to out-of-distribution samples?\n- The encoder in this algorithm is assumed to be deterministic how does the deterministic assumption impact the performance and the ability to capture the full range of variations in the data?\n- Could authors please elaborate more on the conditions under which the consistency of the estimation by maximizing $\\log p(x|z^~)$ holds?- How does the use of a latent chain and latent space improve the sampling process compared to the visible chain? What are the advantages and disadvantages of using a latent LMC?\n- The paper mentions that the perturbation design, including the encoder-decoder pair (fe, fd) and noise magnitude $\\sigma$, significantly impacts the algorithm's performance. How can we effectively select the optimal perturbation design for different datasets and anomaly detection tasks?\n- The paper mentions that the autoencoder (fe, fd) and the noise magnitude $\\sigma$ should be independent of $\\theta$ and remain fixed during training. How does the fixed nature of the autoencoder and noise magnitude impact the algorithm's adaptability to different datasets and anomaly types?\n- Can you provide more details on how the simultaneous use of multiple perturbations enhances the algorithm's performance? Are there any potential trade-offs or challenges associated with this approach?\n- One of the main potential issues is the memory overhead in manifold ensembles. When utilizing multiple autoencoder manifolds in MPD, there is a memory overhead associated with processing multiple groups separately. How can this memory overhead be managed effectively to ensure efficient training while utilizing multiple autoencoders?\n- Can you provide more insights into how the choice of Dz affects the algorithm's ability to detect anomalies? How can we determine the optimal combination of autoencoders with varying Dz for different types of data? Specifically, in high-dimensional data such as images, how does MPDR framework deal with curse of dimensionality and maintain good performance with relatively smaller autoencoders?\n\nLIMITATIONS:\nPlease check weaknesses and questions.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-05"}

comments: SUMMARY:
This paper introduces an EBM-based model for anomaly detection in the latent manifold space. The proposed model first trains an autoencoder that maps a data point $x$ into the low-dimensional $z$, and then two-stage sampling strategy is developed to generate the original data via the LMC algorithm. Several notable designs are developed to complete the proposed model, such as manifold projection diffusion, two-state sampling, and perturbation ensemble. Experiments on images, vectors and acoustic signals show the strong performance of the method.

STRENGTHS:
(1)	The idea that generation of negative samples from the manifold space sound good. This makes sure the starting points highly reflecting information, resulting in more discriminative generation. 

(2)	Several practical strategies such as two-stage sampling and resembling multiple autoencoder are developed to benefit the anomaly detection performance. 

(3)	Extensive empirical results and ablation studies show the efficiency of the model.

WEAKNESSES:
(1)	lack of the results of widely-used anomaly detection dataset MVTec-AD 

(2)	Lack of well-known anomaly detection baselines, such as UniAD [1] and DRAEM [2]

[1] Zhiyuan You et.al. A unified model for multi-class anomaly detection.

[2] Vitjan Zavrtanik et.al. Draem-a discriminatively trained reconstruction embedding for surface anomaly detection.

QUESTIONS:
1. The method artificially inject Gaussian noises into the latent space. What is the correlations between the simulated abnormal data and the real abnormal data in the test dataset?

LIMITATIONS:
yes.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-07

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a novel anomaly detection algorithm utlizing energy-based models (EBMs). The proposed method, Manifold Projection-Diffusion Recovery (MPDR), is based on recovery likelihood, a framework for learning energy functions by denoising data from artifically injected Gaussian noise. MPDR uses deterministic encoder and decoder to first project the data onto a latent space, add Gaussian noise in the latent space, and finally decode the noisy latent representation onto the data manifold. This would capture more relevant modes of variation in the data than Gaussian perturbations on the original data. The recovery likelihood of MPDR is derived and is shown to result in consistent estimation of the energy function. Negative samples are generated via Langevin Monte Carlo in the latent space. The paper also introduces additional variations, including different types of energy functions and the use of ensembles to generate diverse negative samples. Experiments demomstrate the effectiveness of MPDR for out-of-distribution detection on MNIST and CIFAR-100 datasets, as well as anomaly detection for acoustic signals.\n\nSTRENGTHS:\nAnomaly detection is a long standing problem in machine learning with a rich literature, and this paper proposes a novel step in the development of new algorithms. The idea of using recovery likelihood as well as projecting data onto a low-dimensional latent space are not novel, but this paper combines them to produce an original idea. The relevant background and motivation are presented with sufficient clarity, and the proposed method seems logically sound. The toy example presented in Figure 2 empirically verifies that MPDR captures more relevant modes of variation in the data.\n\nWEAKNESSES:\nThe related works section cite most of the relevant previous works, but are not covered in sufficient detail, perhaps owing to the pade limitations. It is important to cover previous work exhaustively to demonstrate where the proposed method stands in relation.\n\nQUESTIONS:\nThe experiments cover two image datasets and acoustic signals, which are relatively high dimensional data. It would be interesting to demonstrate the performance of MPDR on tabular data [1] (both low and high-dimensional) and explore the relevant modes which arise from applying MPDR to such datasets.\n\n[1] Han, S., Hu, X., Huang, H., Jiang, M. and Zhao, Y., 2022. Adbench: Anomaly detection benchmark. *Advances in Neural Information Processing Systems*, *35*, pp.32142-32159.\n\nLIMITATIONS:\nThe paper covers the limitations of MPDR in sufficient detail, namely the sensitivity to the autoencoders used for latent space projection, questions around application to high-dimensional text or image data, and the fact that MPDR is not optimized to generate samples from simple distributions.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-28"}