Energy-Based Models for Anomaly Detection: A Manifold Diffusion Recovery Approach
Authors: Sangwoong Yoon
Young-Uk Jin
Yung-Kyun Noh
Frank C. Park
Venue: NeurIPS 2023 poster (2023)
Keywords: Energy-based Models
Anomaly Detection
Generative Models
Out-of-Distribution Detection
Recovery Likelihood
TL;DR: We present Manifold Projection-Diffusion Recovery, a new training algorithm for energy-based models (EBMs), and show its effectiveness in anomaly detection and out-of-distribution detection tasks.

Abstract: We present a new method of training energy-based models (EBMs) for anomaly detection that leverages low-dimensional structures within data. The proposed algorithm, Manifold Projection-Diffusion Recovery (MPDR), first perturbs a data point along a low-dimensional manifold that approximates the training dataset. Then, EBM is trained to maximize the probability of recovering the original data. The training involves the generation of negative samples via MCMC, as in conventional EBM training, but from a different distribution concentrated near the manifold. The resulting near-manifold negative samples are highly informative, reflecting relevant modes of variation in data. An energy function of MPDR effectively learns accurate boundaries of the training data distribution and excels at detecting out-of-distribution samples. Experimental results show that MPDR exhibits strong performance across various anomaly detection tasks involving diverse data types, such as images, vectors, and acoustic signals.

Energy-Based Models for Anomaly Detection: A Manifold Diffusion Recovery Approach
Sangwoong Yoon Korea Institute for Advanced Study swyoonOkias. re.k.
Young-Uk Jin Samsung Electronics yueric. soorric.jinsaaseen jin@samsung...... com
Yung-Kyun Noh" Hanyang University Korea Institute for Advanced Study nohyungdhanyang. ac kr
Frank C. Park
Scoul National University
Saige Research f fcpesnu. ac kr
Abstract
We present method of training energy-basee models (EBMs) for anomaly detection that leverages -ww-dmmnnsionl structures within data. The proposed algorithm, Manifold Projection- Difffuiinn Recovery (MPDR), first perturbs a data point along a low-dimensional manifold that approximates the training dataset. Then, EBM is trained maximize the probability of recovering the original data. The training involves the generation of negative samples via MCMC, as conventional EBM training, but from different distribution concentrated near the manifold. The resulting near-manin manrfla negative samples are highly informative, reflecting relevant modes of variation in data. An energy function of MPDR effectively learns accurate boundaries of the training data distribution and excels detecting out-of-distribution samples. Experimental results show that MPDR exhibits strong performance across various anomaly detection tasks involving diverse data types, such images, vectors, and acoustic signals.
Introduction
Unsupervised detection of anomalous data appears frequently in practical applications, such as industrial surface inspection [1. machine fault detection 3 and particle physics (4]. Modeling the probability distribution of normal data Pdata (x) is principled approach for anomaly detection [5). Anomalies, often called outliers out-of-distribution (OOD) samples, lie outside of the data distribution and can thus be characterized by low probability density under the distribution. However, many deep generative models that evaluate the likelihood of data, including variational autoencoders (VAE; (6)), autoregressive models [7). and flow-based models g are known to perform poorly on popular anomaly detection benchmarks such as CIFAR-10 (in) SVHN (out), by assigning high likelihood on seemingly trivial outliers [0].
On the other hand, deep energy based models (EBMs) have demonstrated notable improvement in anomaly detection compared to other deep generative models [11] While the specific reason for the superior performance of EBMs has not been formally analyzed, one probable factor is the explicit mechanism employed in EBM's maximum likelihood training that reduces the likelihood of negative samples. These negative samples
generated from the model distribution pe p=(x) using Markov Chain Monte Carlo (MCMC). Since modern EBMs operate in high-dimensional data spaces, is extremely difficult cover the entire space with finite-length Markov chain. The difficulty in
"Corresooning authors
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Perturbation
Perturbation
IGao et
0.05
0.10
0.15
0.20
0.32 0.52
2
+
fe
X
X
X M
Figure I: (Left) An illustration of Manifold Projection- Diffusion. A datum x is first projected into the latent space Z through the encoder fx(x) and then diffused with local noise, such as Gaussian. The perturbed sample & is obtained by projecting it back to the input space X' through the decoder fa(z). (Right) Comparison between the perturbations in MPDR and DRL [66] on CIFAR-10 examples.
Projectio-Diiffusion.
generating negative samples triggered the development of several heuristics such as using truncated chains [12], persistent chains [3 sample replay buffers [ and data augmentation in the middle of the chain [14].
Instead of requiring a Markov chain to cover the entire space, EBM can be trained with MCMC running within the vicinity of each training datum. For example, Contrastive Divergence (CD; (15) uses a short Markov chain initialized on training data to generate negative samples close to the training distribution. Diffusion Recovery Likelihood (DRL; (16)) enables MCMC to focus on the training data's 's neighborhood using a different objective function called recovery likelihood. Recovery likelihood po (x) x) is the conditional probability of training datum x given the observation of its copy x perturbed with Gaussian noise. Training of DRL requires sampling from the recovery likelihood distribution pa xx),,, which easier than sampling from the model distribution px(x). as pe (x|x) is close to uni-modal and concentrated near While CD and DRL significantly stabilize the negative sampling process, the resulting EBMs exhibit limited anomaly detection performance due to the insufficient coverage of negative samples in the input space.
In this paper, we present novel training algorithm for EBM that does not require MCMC covering the entire space while achieving strong anomaly detection performance. The proposed algorithm, Manifold Projection- Diitfooinnnff Recovery (MPDR), extends the recovery likelihood framework by replacing Gaussian noise with Manifold Projection-Diffusiii that reflects -iwddmmnnioonll structure of data. Leveraging a separately trained autoencoder manifold, MPD firstly projects a training datum onto the manifold and perturbs the datum along the manifold. Compared to Gaussian noise, MPD reflects relevant modes of variation in data, such as change in colors shapes in an image, as shown in Fig. [ A MPD-perturbed sample serves an informative starting point for MCMC that generates a negative sample, teaching EBM to discriminate challenging outliers that has partial similarity to training data.
(MPD), novel perturbation operation
Given an MPD-perturbed sample, EBM is trained by maximizing the recovery likelihood. We derive a simple expression for evaluating the recovery likelihood for MPD. We also propose an efficient two-stage MCMC strategy for sampling from the recovery likelihood, leveraging the latent space of the autoencoder. Additionally, we show that the model parameter estimation by MPDR is consistent under standard assumptions.
Compared to existing EBM training algorithms using an autoencoder-like auxiliary module 17 18
19 20). MPDR has two advantages. First, MPDR allows the use of multiple autoencoders for training. The manifold ensemble technique stabilizes the training and improves anomaly detection performance by providing diverse negative samples. Second, MPDR achieves good anomaly detection performance with lightweight autoencoders. For example, autoencoders in our CIFAR-10 experiment has about only 1.5m3 million parameters, which are significantly fewer than that of NVAE
21 (~130 million)
or Glow 8 (~44 million), both of which are previously used in EBM training
17 18
Our contributions can be summarized as follows:
We propose MPDR, novel method of using autoencoders for training EBM. MPDR compatible with any energy functions and gives consistent density parameter estimation with any autoencoders. We provide a suite of practical strategies for achieving successful anomaly detection with MPDR, including two-stage sampling, energy function design, and ensembling multiple autoencoders.
2

---

We demonstrate the effectiveness of MPDR on unsupervised anomaly detection tasks with various data types, including images, representation vectors, and acoustic signals.
2 Preliminaries
Energy-Based Models (EBM) An energy-based generative model, or an unnormalized probabilistic model, represents a probability density function using scalar energy function Eg X - IR where X' denotes the domain of data. The energy function Eg defines a probability density function pa
through the following relationship:
po(x) exp(-Ee(x))
(1)
The parameters 0 can be learned by maximum likelihood estimation given iid samples from the underlying data distribution Pdota (x). The gradient of the log- likelihood for a training sample well-known 15 and can be written as follows:
Ve lo = VxE + Ex-~~p(xd (Vo Eo )}
(2)
where x denotes "negative" sample drawn from the model distribution (x)x). Typically, x is generated using Langevin Monte Carlo (LMC). In LMC, points are arbitrarily initialized and then iteratively updated in stochastic manner to simulate independent sampling from po(x). For each time step t, point (*) is updated by x x(++1) x A V Eg + )ge for e(f) ~ (0, I). The step size Å and the noise scale λ2 are often tuned separately in practice. Since LMC needs to be negative sampling until convergence, and compromise must be made. Popular heuristics include initializing MCMC on training data [15].
performed in every iteration of training, it is infeasible
using short-run LMC [2]. and utilizing replay buffer 11
Instead of directly maximizing the likelihood (Eq. (2) 0 can be learned through the process of denoising data from artificially injected Gaussian noises. Denoising corresponds to maximizing the recovery likelihood p(x 8), the probability of recovering data x from its noise-corrupted version x x oe, where (0.
Recovery Likelihood [22, 16
po (x(x) (x)))((x(x) exp - Fxx(
20 Ix
1
where Bayes' rule is applied. The model parameter 0 is estimated by maximizing the log recovery likelihood, i.c., maxØ Exxx(log ps [x] for x ~ Pdato (x), x ~ p(x|x), where p(x(x) = N(x, o I'I).
This estimation is shown to be consistent under the usual assumptions (Appendix A.2 in [6]]. DRL
[16] uses slightly modified perturbation x /1 σ²x + oee in training EBM, following 23 This change introduces a minor modification of Eq.
The recovery likelihood (x(xx (Eq, 3 essentially new EBM with the energy (x
(x(x) + ||x x Therefore, the gradient Vo log po (x|x) has the same form with the log- likelihood gradient of EBM (Eq. ),), except that negative samples are drawn from po (x(x) instead of (x):
Vp [og = - (x) + Ex Ex ~po(())) ExEx(x }}
as the Gaussian noise is independent of 6. Sampling from pe (x|x) more stable than sampling from (x)).. because (x(x(x) is close a uni-modal distribution concentrated near x [16. [2]. However, it is questionable whether Gaussian noise is the most informative way to perturb data in a high-dimensional space.
where (x Vo
3 Manifold Projection-Diffusion Recovery
We introduce the Manifold DnjjcctionDiifussinn Recovery (MPDR) algorithm, which trains EBM by recovering from perturbations that more informative than Gaussian noise. Firstly, we propose Manifold
(MPD), a novel perturbation operation leveraging the low-dimensional structure inherent in the data. Then, we derive the recovery likelihood for MPD. We also provide an efficient sampling strategy and practical implementation techniques for MPDR. The implementation of MPDR is publicly available
3

---

Palsta
& Manifold
MPD-Perturbed
After Latent Chain
X
Negative Sample
Figure 2: Negative sample generation process in MDPR. Data x (blue dots) are first projected and diffused on autoencoder manifold M (gray líne), resulting in x. The latent chain starting from x generates X ... The visible chain draws negative samples x from p(x|x) starting from X
Autoencoders MPDR assumes that pretrained autoencoder approximating the training data manifold is given. The autoencoder consists of an encoder fe : X - Z and decoder fa : Z X. both assumed to be deterministic and differentiable. The latent space is denoted dimensionalities of X' and 2 are denoted as Dx and D2, respectively. We assume fe and fa as typical deep neural networks jointly trained to reduce the training data's reconstruction error [(x), where [xx) is typically an 22 error, [(x) = ||x fa(f((x))]1.
Z. The
3.1 Manifold Projection- Diifiusionnn
We propose novel perturbation operation, Manifold Iogjection-Diffusiin (MPD), Instead of adding Gaussian noise directly to a datum x +os, as in the conventional recovery likelihood, MPD first encodes using the autoencoder and then applies a noise in the latent space:
x fs toc
X.
where z = fx)), 2 = z de, and x = fa(z). The noise magnitude a is a predefined constant and (). The first step projects into 2, and the second step diffuses the encoded vector z. When decoded through fa. the output & always lies on the decoder manifold M = fx|x = fa(z), 2 2). collection of all possible outputs from the decoder fz(z). The process is visualized in the left panel of Fig.
Since z serves coordinate of M, Gaussian noise in Z corresponds to perturbation of data along the manifold M, reflecting more relevant modes of variation in data than Gaussian perturbation X' (Fig. [.. Note that MPD reduces to the conventional Gaussian perturbation if we set Z X and set fe and fa as identity mappings.
3.2 Manifold Projection- Driffsiion-Diffuuiion Recovery Likelihood
We define the recovery likelihood for MPD as pa (x 2), which is evaluated as follows:
()) Po (x(2)) &pp (x)p(()(2)))) = (x))) (f p(z))))(22))))) (i)
px(x) (f p(212)0 (z)dz
(6)
(iI) Po (x)p((/z = fx(x exp(-Ee(x + ((gg((z))) exp(- (e (x((2))
In (i), we apply Bayes' rule. In (ii), we treat p(z) x)) as (-(-). )), a Dirac measure on Z at z = fx(),, as our encoder is deterministic. Equality (iii) results from the property of Dirac measure / fx)(xx6 (z)dz
fyy. Using Gaussian perturbation simplifies the energy to (x = Eg (x) + 2 2 fx
Now, Ex( is trained by maximizing log pa (x|2). The gradient of log (x|2) ()) with respect to 0 has the same form as the conventional maximum likelihood case (Eq. ( and the Gaussian recovery likelihood case (Eq. 4 but with different negative sample distribution (x Z):
V log (x|2) = - Ex() + (x=xx=(x(()) Vo [x )).
(8)
Negative samples x drawn from pe (x|2) using MCMC.
Another possible definition for the recovery likelihood is p(x x), which becomes equivalent to p(x|2) when fa is injective function, i.e., no two instances of 2 map to the same x. However, if fa is not injective, additional information loss may occur and becomes difficult compute. As a result, p(x(z)
serves more general and also convenient choice for the recovery likelihood.

---

MPDR-S Eg = Scalar / Error: 0.018
MPDR-R Eg AE
Algorithm 1 Manifold Projection Srnjjection-iffusiinn Recovery
Error: 0.012
1: while converged do
2: Sample a mini- bani- of positive samples x
3: Compute
[x() +
4: Sample from energy Ho (z((z)) using
LMC on Z starting from 2
5: Sample x from energy Ex(x using
LMC on starting from x6 fa(z)
6: Update I with the gradient:
Figure 3: 2D density estimation with a scalar energy function (left) and an autoencoder-based energy function (right).
# Ex( + 00 Ex()
7: end while
Consistency Maximizing log (x) leads to the consistent estimation of 0. The consistency is shown by transforming the objective function into KI. divergence. The assumptions resemble those for maximum likelihood estimation, including infinite data, identifiabliitty. and a correctly specified model. Additionally, the autoencoder (fe: fa) and the noise magnitude a should be independent of 0, remaining fixed during training. The consistency holds for any choices of (fe, f(a and o, long as the recovery likelihood po(x(z) is non-zero for any (x, Z). Further details can be found in the Appendix.
3.3 Two-Stage Sampling
default method for drawing x from pa (x is to execute LMC on X, starting from x, as done in DRL [16). While this visible chain should suffice in principle with infinite chain length, sampling can be improved by leveraging the latent space Z, as demonstrated in [24] 18 20 25]. For MPDR, propose a latent chain, short LMC operating Z that generates better starting point x, for the visible chain. We first define the auxiliary latent energy He (z) Es (fa(z)(z) the pullback of (x|z) through the decoder fz).). Then, we run latent LMC that samples from probability density proportional
(z)) The latent chain's 's outcome, z is fed to the decoder to produce the visible chain's starting point Xg = fa(z), which is likely to have smaller energy than the original starting point Ex (2) < (x Introducing a small number of latent chain steps improves anomaly detection performance in our experiments. A similar latent-space LMC method appears in (24 but requires a sample replay buffer not used MPDR. Fig illustrates the sampling procedure, and A summarizes the training algorithm.
3.4 Perturbation Ensemble
D 32
Using multiple perturbations simultancously during MPDR training improves performance and stability, while not harming the consistency. Although the consistency of estimation is independent of the specifics of the perturbation, the perturbation design, such as (fe, fa) and σ, have a significant impact on performance in practice. Perturbation ensemble alleviates the risk of adhering to a single suboptimal choice of perturbation.
64
128
Noise Magnitude Ensemble We randomly draw the per- turbation magnitude a from a uniform distribution over a pre- defined interval for each sample in mini-batch. In imple- mentation, we draw a from the interval [0.05, 0.3] throughout all the experiments.
Figure 4: CIFAR-10 negative sam- ples from manifold ensemble. Negative samples become more vi- sually complex as D2 grows larger.
Manifold Ensemble We can also utilize multiple autoen- coder manifolds M in MPD. Given K autoencoders, mini- batch is divided into K equally sized groups. For each group, negative samples are generated separately using the corresponding autoencoder. Only a minimal increase in training time is observed as K increases, since each autoencoder processes a smaller mini-batch. Memory overhead does exist but is manageable, since MPDR can achieve good performance with relatively smaller autoencoders.
5

---

Table 1: MNIST hold-out digit detection. Performance is measured in AUPR. The standard deviation of AUPR is computed over the last 10 epochs. The largest mean value is marked in bold, while the second-largess is underlined. Asterisks denote that the results are adopted from literature.
Hold-Out Digit
Autoencoder (AE)
0.062 + 0.000
0.204 + 0.003
0.259
0.006
0.125 + 0.003
0.113 + 0.001
IGEBM
0.101 + 0.020
0.106 0.019
0.205
0.108
0.100 0.042
0.079 + 0.015
MEG 26
0.281 + 0.035
0.401 + 0.061
0,402 + 0.062
0.290
0.040
0.324 + 0.034
BiGAN-o 27
0.287 + 0.023
= 0.443 0.029
0.514
1.041
0.307 - 0.028
Latent 20
0.630 0.017
0.619
0.409
0,413 0.010
VAE+EBM 19
0.297 0.033
0.723 : 0.042
0.676 0.041
0.490
0.383 - 0.025
NAE 24
0.802 + 0.079
+ 0.648 0.045
+716+00051
0
0.441 = 0.067
MPDR-S (ours)
0.764 + 0.045
0.823 + 0.018
0.741 + 0.041
0.857 + 0.02
0.478 + 0.048
MPDR-R (ours)
0.844 + 0.030
0.711 + 0.029
0.757 0.024
0.850 + 0.014
0.569 - 0.036
An effective strategy is to utilize autoencoders with varying latent space dimensionalities Dz. For nigh-dimensional data, such images, M with different DN tends to capture distinct modes of variation in data. As depicted in Fig. 4 a perturbation on .M with small D₂ corresponds to low- frequency variation in X, whereas for M with large Dz, it corresponds higher-frequency variation. Using multiple D₂'s 's in MPDR gives us more diverse x and eventually better anomaly detection performance.
3.5 Energy Function Design
MPDR a versatile training algorithm for general EBMs, compatible with various types of energy functions. The design of energy function plays crucial role in anomaly detection performance, as the inductive bias of an energy governs its behavior in outoof-distribution regions. We primarily ex- plore two designs for energy functions: MPDR-Scalar (MPDR-S), feed-forward neural network that takes input x and produces a scalar output, and HPPDRRReconstrucke
(MPDR-R), -)), the reconstruction error from an autoencoder, Ex(x) |x gd(g. (x) for an encoder ge and decoder Ja. The autoencoder (ge. 9a) is separate from the autoencoder (fe. fa) used in MPD. First proposed in (24). a reconstruction based energy function has an inductive bias towards assigning high energy values to off-manifold regions (Fig. 3). Training such energy functions using conventional techniques like CD [15 or persistent chains 13 is reported to be challenging (24). However, MPDR effectively trains both scalar and reconstruction energies. Additionally, in Sec. we demonstrate that MPDR is also compatible with an energy function based on a masked autoencoder.
Experiment
4.1 Implementation of MPDR
An autoencoder (afe is trained by minimizing the reconstruction error of the training data and remains fixed dur- ing the training of (x). When us- ing an ensemble of manifolds, each au- toencoder trained independently. For anomaly detection, the energy value Eş(x) serves as an anomaly score which assigns a high value for anomalous x. All optimizations are performed using Adam with learning rate of 0.0001. Each run is executed on a single Tesla
Table 2: MNIST OOD detection performance measured in AUPR. We test models from hold-out digit 9 experiment (Table 1). The overall performance is high, as detecting these outliers is easier than identifying the hold-out digit.
KMNIST EMNIST Omniglot FashionMNIST Constant
AE
0.999
0.977
0.947
1.000
0.954
IGEBM
0.990
0.923
0.845
0.996
1.000
NAE
1.000
0.993
0.997
1.000
1.000
MPDR-S
0.999
0.995
0.994
0,999
1.000
MPDR-R
0.999
0.989
0.997
0.999
0.990
V100 GPU. Other details, including network architectures and LMC hyperparameters, can be found in the Appendix.
Spherical Latent Space latent space SD.-1 -1 (88 29 30) The encoder output is projected onto SD.-1 via division
In all our implementations of autoencoders, we utilize a hyperspherical
by its norm before being fed into the decoder. Employing DD-1 standardizes the length scale of

---

2, allowing us to use the same value of a across various data sets and autoencoder architectures. Meanwhile, the impact of s on the reconstruction error is minimal.
Regularization For a scalar energy function, MPDR-S, we add Treg = Ex + Ex(x )² to the loss function, as proposed by [1T] For a reconstruction energy function, MPDR-R, we add Lreg = Ex )²² following (24).
Scaling Perturbation Probability Applying regularization to energy restricts its scale, causing a mismatch in scales between the two terms in the recovery likelihood (Eq. (7)) To remedy this mismatch, we heuristically introduce a scale factor y < 1 to log ž). resulting in the modified
recovery energy ( (x|z) = Ex(x) + 2 l| - fe())]l?. We use 7 = 0.0001 for all experiments.
4.2 2D Density Estimation
We show MPDR's 's ability to estimate multi-modal density using a mixture of eight circularly arranged 2D Gaussians (Fig. 2}. We construct an autoencoder with S' latent space, which approximately captures the circular arrangement. The encoder and the decoder are MLPs with two layers of 128 hidden neurons. To show the impact of the design of energy functions, we implement both scalar energy and reconstruction energy. Three-hidden-layee MLPs are used for the scalar energy function, and the encoder and the decoder in the reconstruction energy function. Note that the network architecture of the reconstruction energy is not the same as the autoencoder used for MPD. The density estimation results are presented Fig. 3 We quantify density estimation performance using I error. After numerically normalizing the energy function and true density on the visualized bounded domain, we compute the I error at 10,000 grid points. While both energies capture the overall landscape of the density, the reconstruction energy achieves a smaller error by suppressing probability density at off-manifold regions.
4.3 Image
Detection
MNIST Hold-Out Digit Detection Following the protocol of [26] [26 we evaluated the perfor- mance of MPDR on MNIST hold-out digit detection benchmark, where one of the ten digits in the MNIST dataset is considered anomalous, and the remaining digits are treated as in-distribution. This a challenging task due to the diversity of the in-distribution data and high degree of similarity between target anomalies and inliers. In particular, selecting digits 1, 4, 5, 7, and 9 as anomalous is known to be especially difficult. The results are shown in Table [
In MPDR, we use a single autoencoder (Je, with D2 = 32. The energy function of MPDR-S is initialized from scratch, and the energy function of MPDR-R is initialized from the (fe, fa) used in MPD. Even without a manifold ensemble, MPDR shows significant improvement over existing algorithms, including ones leveraging an autoencoder in EBM training [19 224. The performance of MPDR is stable over a range of Dz. as demonstrated in Appendix B.I.
MNIST OOD Detection To ensure that MPDR is not overfitted to the hold-out digit, we test MPDR in detecting five non-MNIST outlier datasets (Table 2). The results demonstrated that MPDR excels in detecting wide range of outliers, surpassing the performance of naive algorithms such as autoencoders (AE) and scalar EBMs (IGEBM). Although MPDR achieves high overall detection performance, MPDR-R exhibits slightly weaker performance EMNIST and Constant datasets. This can be attributed to the limited flexibility of the autoencoder- based energy function employed in MPDR-R.
CIFAR-10 10 OOD Detection We evaluate MPDR on the CIFAR-IO inlicrs, a standard benchmark for EBM-based OOD detection. The manifold ensemble includes three convolutional autoencoders, with D₂ = 32, 64, 128. MPDR-S uses a ResNet energy function used in IGEBM [T]. MPDR-R adopts the besde autoencoder architecture used in NAE 24
Table 3 compares MPDR to ofate-of-the-art art EBMs. MPDR-R shows competitive performance across five OOD datasets, while MPDR-S also achieves high AUROC on SVHN and Constant. As both MPDR-R and NAE use the same autoencoder architecture for the energy, the discrepancy in performance can be attributed to the MPDR training algorithm. MPDR-R outperforms NAE on four out of five OOD datasets. Comparison between MPDR-S and DRL demonstrates the effectiveness of non- Gon-Gaussian manifold-aware perturbation used in MPDR. CLEL shows strong
7

---

Table 3: OOD detection with CIFAR-10 as in-distribution. AUROC values are presented. The largest value in the column is marked as boldface, and the second and the third largest values are underlined. Asterisks denote that the results are adopted from literature.
Table 4: OOD detection on pretrained ViT-B 16 representation with CIFAR- 100 as in-distribution. Performance is measured in AUROC.
SVIIN
Textures
Constant
CIFARIOO
CelebA
CIFARIO
SVHN ColebA
PixeICNN+
0.32
0.33
0,71
0.63
Supervised
GLOW g
0.24
0.27
0.55
0.57
MD B4
0,8634 0,9638 0.8833
IGEBM
0.63
0.48
0.39
RMD 35
0.9159 0.9685 0.4971
NVAE 21
0.4402
0.4554
0.6478
0.4943
0.6804
VAEBM
0.83
0.62
0.77
Unsupervised
JEM 32
0.67
0.60
0.67
0.75
AE
0.8580
0.9645 0.8103
Improved CD 14
0.7843
0.7275 0.8000
0.5298
0,5399
NAB
0.8041
0.9082 0.8181
NAE 24
0,9352
0.7472 0,9793
0.6316
0.8735
IGEBM I
0,8217 0,9584 0,9004
DRL 16
0.8816
0.4465 0.9884
0.4377
0.6398
DRL [6
0,5730
0.6340 0.7293
CLEL 38
0.9848 0.9437
0.7161
0.7717
MPDR-S (ours)
0.8338
0.9911 0.9183
MPDR-S (ours)
0,9860 0.6583 0.9996
0,5576
0,7313
MPDR- (ours)
0.8626 0.9932 0.8662
MPDR-R (ours)
0.9807 0.7978 0.9996
0.6354
0.8282
MPDR-R Single AE
MPDR-R Single AE
D₂ = 128
0.6965 0.9326 0.9526
D2 32
0.8271 0.6606 0.9877
0.5835
0.7751
D2 256
0,8048 0.9196 0.7772
D2 64
0,9330 0.6631 0,9489
0.6223
0.8272
D2 1024
0.7443 0.9482 0.9247
D2 = 128
0,9886 0.6942 0.9651 0.6531 0,8500
overall performance, indicating that learning semantic information is important in this benchmark. Incorporating contrastive learning into MPDR framework is an interesting future direction.
CIFAR-100 OOD Detection on Pretrained Representation In Table 4 we test MPDR on OOD detection with CIFAR-100 inliers. To model a distribution of diverse images like CIFAR-I0K we follow (34 and apply generative modeling the representation space from a large-scale pretrained model. As we assume an unsupervised setting, we use pretrained representations without fine- tuning. Input images are transformed into 768D vectors by ViT-B 16 model [36]. ViT outputs are normalized with its norm and projected onto hypersphere. We observe that adding small Gaussian noise of 0.01 to training data improves stability of all algorithms. We use MI.P for all energy functions and autoencoders. In MPDR, the manifold ensemble comprises three autoencoders with D2 = 128, 256, 1024. We also implement supervised baselines (MD 34 and RMD (35).. The spherical projection is not applied for MD and RMD to respect their original implementation.
MPDR demonstrates strong anomaly detection performance in the representation space, with MPDR- S and MPDR-R outperforming IGEBM and AE/NAE, respectively. This success can be attributed to the -indeimnnionnll structure often found in the representation space of in-distribution data, as observed in (37). MPDR's 's average performance is nearly on par with supervised methods, MD and RMD, which utilize class information. Note that EBM inputs are no longer images, making previous inapplicable.
EBM training techniques based on image transformation 14 33
Ablation Study Table 3 and 4 also report the results from single-manifold MPDR-R with varying latent dimensionality Dz to show MPDR's s sensitivity to a choice of an autoencoder manifold. Manifold ensemble effectively hedges the risk of relying on single autoencoder which may not be optimal for detecting all types of outliers. Furthermore, manifold ensemble often achieves better AUROC score than MPDR with a single autoencoder. Additional ablation studies can be found in the Appendix. In Sec. B.2 we examine the sensitivity of MPDR to noise magnitude and the effectiveness of the noise magnitude ensemble. Also Sec. B.4.4, we investigate the effect scaling parameter 7 and show that the training is unstable when 7 is too large.
Multi-Class Anomaly Detection on MYTec-AD Using Pretrained Representation MVTec-AD 38 is also a popular anomaly detection benchmark dataset, containing images of 15 objects in their normal and defective forms. We follow the "unified" experimental setting from [39 Normal images
from 15 classes are used as the training set, where no label information is provided. We use the same feature extraction procedure used [39]. Each image is transformed to a 272x 14 14 feature map using EfficientNet-b4, When running MPDR, we treat each spatial dimension of feature map
8

---

Table 5: Acoustic anomaly detection on DCASE 2020 Track 2 Dataset. AUROC and PAUROC (in parentheses) are displayed per cent. pAUROC is defined as AUROC computed over a restricted false positive rate interval [0, pl. where we set p = 0.1, following 41
Toy Car
Toy Conveyor
Fan
Pump
Slider
Valve
AE 441
75.40 (62.03)
77.38 (63.02)
66. 46.44(6 (53.40)
71.42 (61.77)
89.65 (74.69)
72.52 (52.02) 75.23 (51.04)
MPDR-R
81.54 (68.21)
78.61 (63.99)
71.72 (55.95)
78.27 (68.14)
90.91 (76.58)
IDNN 45
76.15 (72.36)
78.87 (62.50)
72.74 (54.30)
73.15 (61.25)
90.83 (74.16)
90.27 (69.46)
MPDR-IDNN
78.53 (73.34)
79.54 (65.35)
73.27 (54.57)
76.58 (66.49)
91.56 (75.19)
91.10 (70.87)
input vector to MPDR, transforming the task into 272D density estimation problem. We normalize a 272D vector with its norm and add a small white noise with a standard deviation of 0.01 during training. We use the maximum energy value among 1 14x 14 vectors as anomaly score of an image. For the localization task, we resize x4x 14 anomaly score map to 224x224 image and compute AUROC for each pixel with respect to the ground true mask. We compare MPDR-R with UniAD 39 and DRAEM [40]. The results are shown in Table I2 Details on MPDR implementation can be found in B.5
MPDR achieves AUROC scores that are very close to that of UniAD, outperforming DRAEM. The trend is consistent in both detection and localization tasks. The performance gap between MPDR and UniAD can be attributed to the fact that UniAD leverages spatial information of 14x 14 feature map while our implementation of MPDR processes each pixel in the feature map separately.
4.4 Acoustic Anomaly Detection
We apply MPDR to anomaly detection with acoustic signals, another form of non-image data. We use DCASE 2020 Challenge Task 2 dataset [11. containing recordings from six different machines, with three to four instances per machine type. The task is detect anomalous sounds from deliberately damaged machines. which are unavailable during training. We applied the standard preprocessing methods in 41 to obtain 540-dimensiona, vector built up of consecutive mel-speettogram features. Many challenge submissions exploit dataset-specifie heuristics and ensembles for high performance, e.g. 43 Rather than competing, we focus on demonstrating MPDR's effectiveness in improving common approaches, such as autoencoders and Interpolation Deep Neural Networks (IDNN) [44]. IDNN is variant of masked autoencoders which predicts the middle (the third) frame given the remaining frames. Similarly to autoencoders, IDNN predicts an input as anomaly when the prediction error is large. We first train AE and IDNN for 100 epochs and then apply MPDR by treating the reconstruction (or prediction) error the energy. Manifold ensemble consists of autoencoders with D2 = -2, 64, 128. More training details can be found in the Appendix.
5 shows that MPDR improves anomaly detection performance for both AE and IDNN. The known failure mode of AE and IDNN is producing unexpected low prediction (or reconstruction) error for anomalous inputs. By treating the eтror as the energy and applying generative training, the error in OOD region is increased through training, resulting in improved anomaly detection performance.
4.5 Anomaly Detection Tabular Data
We test MPDR on ADBench compare MPDR with 13 baselines from ADBench. The baseline results are reproduced using the official ADBench repository. We consider the setting where the training split does not contain anomalies. For each dataset, we run each algorithm on three random splits and computed the AUROC on the corresponding test split. We then ranked the algorithms based on the averaged AUROC. In Table A, we present a summary table with the average rank across the 47 datasets.
45 which consists 47 tabular datasets for anomaly detection. We
We employ MPDR-R with single manifold. For both the manifold and the energy, the same MLP-based autoencoder architecture used. The encoder and the decoder are MLPs with two 1024-hidden- layers. If the input space dimensionality is smaller than or equal to 100, the latent space dimensionality set to have the same dimensionality D2 = Dxx Dx > 100, we set D2 as 70% of Dx. We employ I step of Langevin Monte Carlo in the latent space and 5 steps

---

in the input space. The step sizes are 0.1 for the latent space and 10 for the input space. All the hyperparameters except D, are fixed across 47 datasets.
As shown in 6 MPDR achieves highly competitive performance on ADBench, demonstrating a higher average rank than the isolation forest, with some overlap of confidence interval. This result indicates that MPDR is a general-purpee purpose anomaly detection algorithm capable of handling tabular data. We empirically observe that MPDR is most effective when AUROC from its autoencoder is low, meaning that the outliers are near the autoencoder manifold. When the autoencoder already achieves good AUROC, the improvement from MPDR training is often marginal.
5 Related Work
Leveraging autoencoders in training, MPDR is closely related to other EBM training algorithmss that incorporate auxiliary modules. While the use of single variational autoencoder in EBM training is explored in [18] 19 46), MPDR employs multiple autoencoders. MPDR is also compatible with non- variational autoencoders, offering greater flexibility. Normal- izing flows I17 25 47 provide informative negative samples and a latent space for EBM training, but unlike MPDR, they do not exploit the low-dimensional structure of data. EBMs can also be trained with additional generator modules [48] 49) which plays similar role to the decoder of an autoencoder. A contrastive representation learning module (33] improves EBM
Table The rank of AUROC aver- aged over 47 datasets in ADBench. A smaller value indicates the al- gorithm achieves higher AUROC score compared to other algorithms on average. The standard errors are also displayed.
Method
Average Rank
MPDR (ours)
4.43 + 0.50
performance but relies on -omaii--peccfic data augmentations and is only demonstrated on images. In contrast, MPDR is applicable to a wider range of data, utilizing more general assumption of low- low-dimensionality in data.
IForest
5.28 + 0.39
OCSVM
7.94 + 0.47
CBLOF
5,98 + 0.53
COF
9.23 0.58
COPOD
7.10
MPDR presents novel objective function for EBM by extend- ing recovery likelihood framework 122 [6]. Investigating its
ECOD
6.97
HBOS
+.51
connection to other previously studied objective functions, such
KNN
6.64 0.56
as f-divergence [50]. pseudo-spherical scoring rule [51], diver-
LOF
9.01
gence triangle [46), and Stein discrepancy [52] would also be
PCA
an interesting future direction.
SOD
7
DeepSVDD
MPDR contributes to the field of anomaly detection 45 39 40
DAGMM
10.09 = 0.52
53
The promising results from MPDR demonstrates that learning the distribution of data is principled and effective approach for detecting anomalies (24].
6 Conclusion
Contributions We propose MPDR, novel method of utilizing autoencoders for training EBM. An autoencoder in MPDR provides an informative starting point for MCMC, offers the latent space for the effective traversal of digh-dimensiionl space, and guides the drift of an MCMC sampler (Eq. 7.. MPDR performs competitively various anomaly detection benchmarks involving diverse types of data, contributing to the enhancement of generative modeling for anomaly detection with high-dimensional data.
First, the practical performance of MPDR is still sensitive to the specifics of autoen- coders used in MPD. Second, some data, such as high-resolution images or texts, may not exhibit clear low-dimensional structure. these cases, MPDR may require a separate representation learning stage, demonstrated in our CIFAR-100 experiment with ViT (Sec. 4.3). Third, MPDR is not optimized for generating samples starting from a simple distribution, such as Gaussian, while DRI. is. We may resort to longer-chain MCMC to generate samples from MPDR.
Limitations
10

---

Acknowledgments and Disclosure of Funding
Yoon would like thank Luigi Favaro and Tilman Plchn for their helpful discussions during the early development of the algorithm. S. Yoon is supported by KIAS Individual Grant (AP095701) via the Center for AI and Natural Sciences at Korea Institute for Advanced Study. Y.-K. Noh was partly supported by NRF/MSIT (No. 2018R1A 1SRRAAA705955999 202 M335D22000095550 1-20-0-0373,, RS MSIT grant 2021-0-02068 (SNU AI Innovation Hub), IIT--MSIT grant 2022-0-00433 (Training and Inference Methods for Goal-Oriented Al Agents), KIAT grant P0020536 (HRD Program for Industrial Innovation), ATC+ MOTIE Technology Innovation Program grant 20008547, SRRC NRF grant RS-2023-00208052, SNU-AIIS, SNU-IAMD, SNU BK21+ Program in Mechanical Engineering, and SNU Institute for Engineering Research.
and ITPP/MSIT This work was supported in part by IITP-
(ITTP-2021- 10002210-000 1-0-0200-
References
[1] Paul Bergmann, Michael Fauser, David Sattlegger, and Carsten Steger. Mvtec ad a comprehensive real-world dataset for unsupervised anomaly detection. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2019.
[2] Yuma Koizumi, Shoichiro Saito, Hisashi Uematsu, Noboru Harada, and Keisuke Imoto. Toyadmos: A dataset of miniature-machine operating sounds for anomalous sound detcction. 2019 JEEE Workshop on Applications of Signal Processing to Audio and Acoustics (WASPAA), pages 313-317, 2019.
[3] Harsh Purohit, Ryo Tanabe. Kenji Ichige, Takashi Endo, Yuki Nikaido, Kaori Suefusa, and Yohei Kawaguchi. Mimii dataset: Sound dataset malfunctioning industrial machine investigation and inspection. arXiv preprint wrXiv:2206.1422. 2019.
Barry M Dillon, Luigi Favaro, Tilman Plehn, Peter Sorrenson, and Michael Krämer. A normalized autoencoder lhe triggers. arXiv preprint arXiv:2206. /4225, 2022.
[5] Christopher M Bishop. Novelty detection and neural network validation. IEE Proceedings Vision, Image and Signal processing, 141(4):217- 222, 1994.
[6] Diederik P Kingma and Max Welling, Auto-encoding variational bayes. International Conference on Learning Reprexentations (ICLR), 2014.
[7] Aaron van den Oord, Nal Kalehbrenner, and Koray Kavukcuoglu. Pixel recurrent neural networks. arXiv preprint arviv: htt::6660 /eee:6001000000 06759, 2016.
[8] Durk P Kingma and Prafulla Dhariwal. Glow:
Generative flow with invertible 1x1 convolutions. In Advances in Neural Information Processing Systems, pages 10215-10224, 2018.
[9] Dan Hendrycks, Mantas Mazcika, and Thomas Dictterich. Deep anomaly detection with outlier exposure. In International Conference on Learning Representations, 2019.
[10] Eric Nalisnick, Akihiro Matsukawa, Yee Whye Teh, Dilan Gorur, and Balaji Lakshminarayanan. deep generative models know what they don't know? In International Conference on Learning Representations, 2019.
[11] Yilun Du and Igor Mordatch. Implicit generation and modeling with energy based models, In H. Wallach, H. Larochelle, A. Beygelzimer, F. Alche-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems 32, pages 3608-3618. Curran Associates, Inc., 2019.
[12] Erik Nijkamp, Mitch Hill, Song Chun Zhu, and Ying Nian Wu, Learning non non-convergent non- persistent short-run run meme toward energy-based based model. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d' d'Alché Alch E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 32, pages 5232-5242. Curran Associates, Inc., 2019.
[13] Tijmen Tieleman. Training restricted boltzmann machines using approximations to the likelihood gradient. In Proceedings of the 25th international conference on Machine learning, pages 1064 1071, 2008
[14] Yilun Du, Shuang Li, B. Joshua Tenenhaum, and Igor Mordatch. Improved contrastive divergence training of energy based models. In Proceedings of the 38th International Conference on Machine Learning (ICML-21), 2021.
[15] Geoffrey E Hinton. Training products of experts by minimizing contrastive divergence. Neural computation. 14(8):1771-1800 2002.

---

[16] Ruiqi Gao, Yang Song, Ben Poole, Ying Nian Wu, and Diederik P Kingma. Learning mnergy-based, models by diffusion recovery likelihood. In International Conference Learning Representations, 2021.
[17] Ruiqi Gao, Erik Nijkamp, Diederik Kingma, Zhen Xu, Andrew M Dai, and Ying Nian Wu. Flow contrastive estimation of energy-based models. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7518 8-7228,, 2020.
[18] Zhisheng Xiao, Karsten Kreis, Jan Kautz, and Arash Vahdat. {VAEBM}; A symbiosis between variational autoencoders and energy-based models. In International Conference Learning Representations, 2021.
[19] Tian Han, Erik Nijkamp, Linqi Zhou, Bo Pang. Song- Chun Zhu, and Ying Nian Wu. Joint training of variational auto-encoder and latent energy- based model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.
[20] Bo Pang, Tian Han, Erik Nijkamp. Song-Chun Zhu, and Ying Nian Wu. Learning latent space energy-based prior model. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 21994-22008. Curran Associates, Inc., 2020.
[21] Arash Vahdat and Jan Kautz. NVAE: A deep hierarchical variational autoencoder. In Neural Information Processing Systems (NeurIPS), 2020.
[22] Yoshua Bengio. Li Yao, Guillaume Alain, and Pascal Vincent. Generalized denoising auto-encoders as generative models. In C.J. Burges, L. Bottou, M. Welling. Z. Ghahramani, and K.Q. Weinberger, editors, Advancex in Neural Information Processing Systems, volume 26. Curran Associates, Inc., 2013.
[23] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 6840 6851. Curran Associates, Inc., 2020.
[24] Sangwoong Yoon, Yung-Kynn Noh, and Frank Park. Autoencoding under normalization constraints. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 12087-12097. PMLR, 18-24 Jul 2021.
[25] Erik Nijkamp, Ruiqi Gao, Pavel Sountsov, Srinivas Vasudevan, Bo Pang, Song-Chun Zhu, and Ying Nian Wu. MCMC should mix: Learning energy-based model with neural transport latent space MCMC. In International Conference on Learning Representations, 2022.
[26] Rithesh Kumar, Sherjil Ozair, Anirudh Goyal, Aaron Courville, and Yoshua Bengio. Maximum entropy generators for energy-based models. arXiv preprint 1911:01901.8858, 2019.
[27] Houssam Zenati, Chuan Sheng Foo, Bruno Lecouat, Gaurav Manek, and Vijay Ramaseshan Chandrasekhar. Efficient gan-based anomaly detection. arXiv preprint 100 2018
[28] Tim Davidson, Luca Falorsi. Nicola De Cao, Thomas Kipf, and Jakub Tomezak. Hyperspherical variational auto-cncoders. In 34th Conference on Uncertainty in Artificial Intelligence 2018, UAI 20/8, pages 856-865. Association For Uncertainty in Artificial Intelligence (AUAI), 2018
[29] Jiacheng Xu and Greg Durrett. Spherical latent spaces for stable variational autoencoders. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 4503-4513, 2018.
[30] Deli Zhao, Jiapeng Zhu, and Bo Zhang. Latent variables spheres for autoencoders in high dimensions. arXiv, pages arXiv 1912,191.22. 2019.
[31] Tim Salimans, Andrej Karpathy, Xi Chen, and Diederik Kingma. Pixelenn++: Improving the pixelcnn with discretized logistic mixture likelihood and other modifications. arXiv preprint arXiv:1701.0555/ 2017.
[32] Will Grathwohl, Kuan-Chich Wang, Heern-Henrik Jacobsen, David Duvenaud, Mohammad Norouzi, and Kevin Swersky. Your classifier is secretly an energy based model and you should treat like one. In International Conference on Learning Representations. 2020.
[33] Hankook Lec, Jongheon Jeong, Sejun Park, and Jinwoo Shin. Guiding energy-based models via contrastive latent variables. In International Conference on Learning Representations, 2023.
[34] Stanislav Fort, Jie Ren, and Balaji Lakshminarayanan. Exploring the limits of Advances in Neural Information Processing Systems, 34:7068-700 7081, 2021.
detection.
12

---

[35] Jie Ren, Stanislav Fort, Jeremiah Liu, Abhijit Guha Roy, Shreyas Padhy, and Balaji Lakshminarayanann A simple fix mahalanobis distance for improving near-ood detection. arXiv preprint 1rXiv:200.00002. 2021.
[36] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua 7hai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at seale. In International Conference on Learning Representations, 2021.
[37] Haoqi Wang. Zhizhong Li, Litong Feng, and Wayne Zhang. Vim: Out-of-distribution with virtual-logit matching. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 4921-4930, June 2022.
[38] Paul Bergmann, Kilian Batzner, Michael Fauser, David Sattlegger, and Carsten Steger. The mytee anomaly detection dataset: a comprehensive well-world dataset for unsupervised anomaly detection. International Journal of Computer Vision, 129(4): 1038- 1059,, 2021.
[39] Zhiyuan You, Lei Cui, Yujun Shen, Kai Yang, Xin Lu, Yu Zheng. and Xinyi Le. A unified model for multi- class anomaly detection. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems, volume 35, pages 4571-4584, Curran Associates, Inc., 2022.
[40] Vitjan Zavrtanik, Matej Kristan, and Danijel Skočaj. Dracm a discriminatively trained reconstruction embedding for surface anomaly detection. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 8330-8339, October 2021.
[41] Yuma Koizumi, Yohei Kawaguchi. Keisuke Imoto. Toshiki Nakamura, Yuki Nikaido, Ryo Tanabe, Harsh Purohit, Kaori Suefusa, Takashi Endo, Masahiro Yasuda, and Noboru Harada. Description and discussion on dcase2020 challenge task2: Unsupervised anomalous sound detection for machine condition monitoring. arXiv preprint 2ifi:2000.08822. 2020.
[42] Ritwik Giri, Srikanth V. Tenneti, Karim Helwani, Fangzhou Cheng, Umut Isik, and Arvindh Krish- naswamy. Unsupervised anomalous sound detection using self supervised classification and group masked autoencoder for density estimation. Technical report, DCASE2020 Challenge, July 2020.
[43] Pawel Daniluk, Marcin Gozdziewski, Slawomir Kapka, and Michal Kosmider. Ensemble of auto-encoder based systems for anomaly detection. Technical report, DCASE2020 Challenge, July 2020.
[44] Kaori Suefusa, Tomoya Nishida, Harsh Purohit, Ryo Tanabe, Takashi Endo, and Yohei Kawaguchi. Anomalous sound detection based on interpolation deep neural network. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). pages 271-275, 2020.
[45] Songqiao Han, Xiyang Hu, Hailiang Huang, Minqi Jiang, and Yue Zhao. Adbench: Anomaly detection benchmark. Koyejo. S. Mohamed, A. Agarwal, D. Belgrave, K. Cho. and Oh, editors, Advances in Neural Information Processing Systems, volume 35, pages 32142-32158 Curran Associates, Inc., 2022.
[46] Tian Han, Erik Nijkamp, Xiaolin Fang, Mitch Hill, Song-Chun 7hu, and Ying Nian Wu. Divergence triangle for joint training of generator model, energy-based model, and inferential model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2019.
[47] Jianwen Xie, Yaxuan Zhu, Jun Li, and Ping Li. A tale of two flows: Cooperative learning of langevin flow and normalizing flow toward energy-based model. In International Conference on Learning Representa- tions, 2022.
[48] Will Sussman Grathwohl, Jacob Jin Kelly, Milad Hashemi, Mohammad Norouzi, Kevin Swersky, and David Duvenaud. No (meme) for me: Amortized sampling for fast and stable training of energy- based models. In International Conference on Learning Representations, 2021.
[49] Michael Arbel, Liang Zhou, and Arthur Gretton. Generalized energy based models. In International Conference on Learning Representations, 2021.
[50] Lantao Yu, Yang Song, Jiaming Song, and Stefano Ermon. Training deep energy- based models with f- divergence minimization. In Hal Daumé III and Aarti Singh, editors, Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 10957-10967.. PMLR, 13-18 Jul 2020.
[51] Lantao Yu, Jiaming Song, Yang Song, and Stefano Ermon. Pseudo sphuro-ppeercal contrastive divergence. In A. Beygelzimer. Y. Dauphin, P. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems, 2021.
13

---

[52] Will Grathwohl, Kuan-Chich Wang. Joern- bern-Heeniik Jacobsen, David Duvenaud, and Richard Zemel. Learn- ing the stein discrepancy for training and evaluating energy-based models without sampling. In Hal Daumé and Aarti Singh, editors, Proceedings of the 37th International Conference on Machine Learning. volume 119 Proceedings of Machine Learning Research, pages 3732-3747, PMLR, 13-18 Jul 2020.
[53] Zongyuan Huang, Baohua Zhang. Guoqiang Hu, Longyuan Yanyan Xu, Yaohui Enhancing unsupervised anomaly detection with score- score-guided network. IEEE Transactions on Neural Networks and Learning Systems, 2023.
[54] Dmitrii Gavrilev and Evgeny Burnaev. Anomaly detection in networks via score-based generative models. In ICML 2023 Workshop on Structured Probabilistic Inference & Generative Modeling, 2023.
[55] Tarin Clanuwat, Mikel Bober-Irizar, Asanobu Kitamoto, Alex Lamb, Kazuaki Yamamoto, and David Ha. Deep learning for classical japanese literature. arXiv preprint arXiv:18/2.017/1 2018.
[56] Gregory Cohen, Saeed Afshar, Jonathan Tapson, and Andre Van Schaik. Emnist: Extending mnist to handwritten letters. In 2017 international joint conference neural networks (IJCNN), pages 2921-2926 TEFE, 2017.
[57] Brenden M Lake, Ruslan Salakhutdinov, and Joshua B Tenenbaum. The omniglot challenge: 3-year progress report. Current Opinion in Behavioral Sciences, 29:97- 104, 2019.
[58] Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mann mnist: novel image dataset for benchmarking machine learning algorithms. arXiv preprint 77Xiv:170..0777,, 2017.
[59] A Krizhevsky. Learning multiple layers of features from tiny images, Master's thesis, University of Tront, 2009.
[60] Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y Ng. Reading digits in natural images with unsupervised feature learning. In NIPS Workshop on Deep Learning and Unsupervised Feature Learning. 2011.
[61] M. Cimpoi, S. Maji. L. Kokkinos, S. Mohamed, and A. Vedaldi. Describing textures in the wild. In Proceedings of IEEE Conf. on Computer Vision and Puttern Recognition (CVPR), 2014.
[62] Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of International Conference on Computer Vision (ICCV), December 2015.
[63] Ting Chen, Simon Komblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning visual representations. International conference on machine learning, pages 1597-1607, PMLR, 2020.
[64] Gareth Roberts, Richard Tweedie, al. Exponential convergence langevin distributions and their discrete approximations. Bernoulli, 2(4):341-363 1996.
14

---

A Consistency of MPDR
Let us denote our model for recovery likelihood as p(x|2; 0). We assume that this model identifiable (different model parameters correspond to different distribution) and correctly specified (there exists 8 such that p(x|2; 8*) = Pdata (x(z)). We also assume that p(x/2: 8) is non-zero for all x, 2, and
N Ei=1 log p(x; 2;;; 0) where (x, 2.) ~ p(x, 2) Pdata (x)p(2)x) x). In the limit of N + oc, the average converges to the expectation Ei=l log p(x, [,;; 0) ->
0. Our objective is to maxo 1
(x)x)) llog p(x ž; 0)]. If subtract (x), [log pxx|2; 0*)] which is constant with respect to 0, then the expression can written with respect KL divergence as follows:
E (og log pog((x(2;00) = p(x, 2) log p(x|2; 0)
p(x 22: 0-) dxdē
(9)
= =- p(z) (L(p(x/2; 1p |p(x|z;0) 0))
(10)
The maximum of Eq. i0]is O, the minimum of KL divergence is 0. The maximum is achieved if and only if 0 = 0" Note that p(ž) is assumed to be constant with respect to 0.
Since we did not rely the specifics of how perturbation p(ž)x) is actually performed, this consistency result holds for any choices of the encoder fe and the noise magnitude a, as long as the recovery likelihood p(x/2; () remains non-zero for all x, 2, and 0.
B Experimental Details and Additional Results
In this section, we provide detailed information on how each experiment conducted and also provide some additional supporting experimental results. The hyperparameters related to LMC is summarized in T ConvNet architectures are provided in Table The contents are organized by the training dataset.
Table 7: Hyperparameters for LMC. Latent chain hyperparameters denoted by Z and X' indicates visible chain hyperparameters. "scale (x)" refers to the multiplicative scale factor on the perturbation probability.
Experiment
2 Steps Z Step Size Z Noise 2 scale (7) X' Steps X' Step Size X' Noise X scale (r)
MNIST
MPDR-S
0.05
0.02
0.0001
10
0.005
0
MPDR-R
0.02
0.0001
0.005
CIFAR-10
MPDR-S
10
0.1
0.01
0.0001
20
10
0.005
0
MPDR-R
10
0.1
0.01
0.0001
20
10
0.005
0
CIFAR-100 ViT
MPDR-S
0.005
0.0001
MPDR-R
30
0.005
0.0001
DCASE 2020 (Toy Car, Toy Conveyor, Pump)
MPDR-R
0
10 0.005
0.0001
MPDR-IDNN
0.005
0.0001
DCASE 2020 (Fan, Slider, Valve)
MPDR-R
10
0.005
0.0001
MPDR-IDNN
20
0.1
0.01
0.0001
20
10
0.005
0.0001
MV
MPDR-R
0.1
0.0001
ADBench
MPDR-R
1
0.1
0.05
0.0001
5
10
0.1
0.0001
15

---

B.1 MNIST
B.1.1 Datasets
All input images are 28x28 grayscale, and each pixel value is represented as a floating number between [0, 1]. Models are trained the training split of MNIST excluding the digit designated to be held-out. The training split contains 60,000 images, and the hold-out procedure reduces the training set approximately 90%. We evaluate the models on the test split of MNIST, which contains a total of 10,000 images. For non non-MNIST datasets used in evaluation, we only use their test split. All non-MNIST datasets used in the experiment also follow the 28 x28 grayscale format, similar to MNIST.
KMNIST (KMNNIS-MNNIST) [55] contains Japanese handwritten letters, pre-processed into the
same format MNIST. The license of KMNIST is CC BY-SA 4.0.
EMNIST (EMNIST-Letters) [56] contains grayscale handwritten English alphabet images. Its test split contains 20,800 samples. No license information is available.
For the Omniglot (57 dataset, the evaluation set is used. The set consists of 13,180 images. No license information available.
We use the test split of FashionMNIST 558 which contains 10,000 images. The dataset made
public under the MIT license.
Constant dataset is a synthetic dataset that contains 28 x images, where all pixels have the same value. The value randomly drawn from uniform distribution over [0, 1]. We use 4,000 constant images.
B.1.2 Autoencoder Implementation and Training
The encoder and the decoder used in MPDR, fe and fa, have the architecture of MNIST Encoder and MNIST Decoder, provided in T respectively. We use the spherical latent space S where D₂ 32 for the main experiment. The autoencoder is trained to minimize the l2 reconstruction error of the training data for 30 epochs with Adam of learning rate 1 x 10-4, The batch size is 128 and no data augmentation is applied. The la norm of the encoder is regularized with the coefficient of 1 x 10 The same autoencoder is used for both MPDR-S and MPDR-R.
B.1.3 MPDR Implementation and Training
In MPDR-S, the energy function Fg has the architecture of MNIST Encoder (Table X with D₂ The network is randomly initialized from PyTorch default setting and the spectral normalization is applied. The energy function trained with MPDR algorithm for 50 epochs. The learning rate is 1 1x 10-4, The batch size is 128. The perturbation probability scaling f'ccorr y for the visible LMC chain is set to zero. For an image-like data such as MNIST, the gradient of perturbation probability Vx(I|ž [x)][²) introduces non-smooth high-frequency patterns resembling adversarial noise, harming the stability of the training. Therefore, we only use non-zero in latent chains in MNIST and CIFAR- experiments.
For MPDR-R, the energy function is initialized from (fe, fa) and then trained through MPDR algorithm. The learning rate is to 10-5, All the other details are identical to the MPDR-S case.
B.2 Sensitivity to
As an ablation study for noise magnitude ensemble, we perform gegll--nose-maanniinee experiment for MPDR and examine MPDR's sensitivity the noise magnitude We evaluate the OOD detection performance of MPDR-S on the MNIST hold-out digit 9 setting with varying values of a. Results are shown in Table
The choice of a has significant impact on MPDR's OOD detection performance. Table 9 0.01 gives poor OOD detection performance, particularly with respect to the hold-out digit. The performance generally improves as a grows larger, but a large or is not optimal for detecting EMNIST.
https:://githu..comm/// www.:ow.oooooooaaeiae..a//aaaiat codh/ http:///hhhuu..omm/o/oooooddd///
16

---

Table 8: Convolutional neural network architectures used in experiments. The parenthesis following the network name indicates the activation function used in the network.
MNIST Encoder (ReLU)
MNIST Decoder (ReLU)
Conva(L. 32)
ConVTa 128)
CIFAR-10 10 Encoder
CIFAR-I0 Decoder
C3nv 64)
MaxPool(2x)
Con9Ta(128, 64)
Upsample(2x)
C 32, stride22)
ConvTs(D) (Da... 256) C255 128, stride=2, pad=1) C1 64, stride=2, pud=1)
C3nv 64, stride=2)
Conva 64)
(on 64)
C6nv 128, stride=
Conv 128)
Upsample(2x)
Coneg(128 256, stride=2)
ConvT; (64, 3) Sigmoid()
MaxPool((2))
Con/Tg164, 32)
FCG266 D2)
(128, 1024)
C32VT=(32. 1)
FC(1224 Dz)
Sigmoid()
CIAAR- 10 Encoder
C3. 128, pad=l)
CIFAR- Decoder
ResBlock(128, 128, down=Iruee
ResBlock(125 128)
ConTTy(DD, 128)
ResBlock(128, 256, down=/Tree
ResBlock( 128, 128, up=True)
ResBlock(256, 256)
ResBlock(125 128, up=TTuee)
ResBlock(256, 256, down-True)
ResBloock(12) 128, up=Tmee)
ResBlock(256, 256)
Convy(128, 3, pad=l)
GlobalAvgPPol()
FC(256, D2)
Selecting single optimal a will be very difficult, and therefore, we employ noise magnitude ensemble which can hedge the risk of choosing a suboptimal value for a.
B.2.1 Sensitivity to DN
As an ablation study for manifold ensemble, we investigate the sensitivity of MPDR to the choice of the latent space dimensionality of the autoencoder. We evaluate the OOD detection performance of MPDR-S on the MNIST hold-out digit 9 setting with varying values of Dz. Table I0 presents the results. Consequently, MPDR runs stably for a large range of Da, producing decent OOD performance. One hypothesis is that, for MNIST, it is relatively easy for these autoencoders to capture manifold structure of MNIST sufficiently well.
Mcanwhile, we do observe that the choice of D2 affects OOD performance in an interesting way. Increasing D3 enhances AUPR for certain OOD datasets but deteriorates AUPR for others. For example, AUPR of Omniglot is increased with larger Dz, but AUPR of EMNIST, FashionMNIST, and Constant dataset decreases. No single autoencoder optimal for detecting all outlier datasets. This observation motivates the use of manifold ensemble, employed in non-MNIST MPDR experiments.
B.2.2
Note on Reproduction
For an autoencoder-based outlier detector, denoted "AE" in Table 1 we use the same autoencoder used in MPDR with Dx-
NAE is reproduced based on its public code base
We tried to reproduce DRL MNIST but failed train it stably. The original paper and the official code base also does not provide a guidance on training DRL on MNIST.
B.3
CIFAR-10 OOD Detection
B.3.1
Datasets
All data used in CIFAR-10 experiment in the 32 32 RGB format. Models are only trained on CIFAR-10 training set, and evaluated on the testing set of each dataset.
contains 60,000 training images and 10,000 testing images. Models are trained on the training set. We don't it use its class information, as we consider only unsupervised setting. No license information available.
CIFAR-I0 10
17

---

Table 9: Sensitivity to a. MPDR-S is run with an autoencoder with varying values of noise magnitude a. AUPR against various outlier datasets are presented. For MNIST we present the standard deviation computed over the last 10 epochs.
MNIST 9
KMNIST
EMNIST
Omniglot
FashionMNIST
Constant
0.01
0.098 + 0.009
0.667
0.827
0,940
0.944
0.895
0.1
0.330 0.063
0.983
0.995
0,971
0.994
0.998
0.2
0.522 + 0.053
0.999
0,993
0,997
0.999
1.000
0.3
0.558 0.039
1.000
0.990
0.997
1.000
1.000
Table 10: Sensitivity to Dz. MPDR-S is run with an autoencoder with varying values of Dz. AUPR against various outlier datasets presented. For MNIST 9, present the standard deviation computed over the last 10 epochs. Noise magnitude ensemble is applied.
D₂
MNIST
KMNIST EMNIST
Omniglot
FashionMNIST
Constant
16
0.611 + 0.041
0.979
0.996
0.958
0,999
1.000
32
0.525 + 0.039
0.999
0.994
0.994
0,999
1.000
64
0.512 + 0.048
0.999
0.993
0.998
0.998
0.999
128
0.505 + 0.051
0.999
0.991
0.999
0.988
0.946
256
0.590 + 0.041
0.996
0.983
0.996
0.958
0.866
a set of digit images. Its test set containing 26,032 is used in the experiment. The dataset is non-commercial use only.
SVHN 60
Texture [61] dataset, also called Describable Textures Dataset (DTD), contains 1,880 test images. The images resized into 32x32. No license information available.
CelebA 162 is a dataset of cropped and aligned human face images. The test set contains 19,962 images. The dataset is for non-commereial research purposes only. We center-crop each image into 140 x 140 and then resize it into 32 x 32.
Constant dataset is synthetic dataset that contains 4,000 32x 32 RGB monochrome image. All 32x32 pixels have the same RGB value which drawn uniform-randee from [0. 1|3. CIFAR-100 [59] contains 60,000 training images and 10,000 testing images. No license information available.
B.3.2 Autoencoder Implementation and Training
The autoencoders for CIFAR-10 experiment have an architecture of "CIFAR-10 Encoder 1" and "CIFAR-10 Decoder 1" in Table with D2 32, 64, 128. Each autoencoder trained for 40 epoch with learning rate x 10-1 and batch size 128. During training the autoencoders, we apply the following data augmentation operations: random horizontal flipping with the probability random resize crop with the scale parameter [0.08, 1] with the probability 0.2, color jittering with probability of 0.2, random grayscale operation with the probability 0.2. The color jittering parameters are the same with the one used in SimCLR 63 (brightness 0.8, contrast 0.8, saturation 0.8, hue 0.4, with respect to torchvision transorms. ColorJitter implementation). The l2 norm of an encoder is regularized with the coefficient of 0,00001. The same autoencoder is used for both MPDR-S and MPDR-R. boost the performance, we heuristically introduce sample replay buffer for negative samples and apply data augmentation in the middle of L.C..
0.5,
B.3.3 MPDR Implementation and Training
The energy function in MPDR-S is "CIFAR-10 Encoder 2" with D, 1. The energy function in MPDR-R is "CIFAR-10 Encoder 2" and "CIFAR-10 10 Decoder with D,, 1. In MPDR-R, the energy function is pre-trained by minimizing the reconstruction error of the training data for 40 epochs. Only random horizontal flip is applied and no other data augmentation used. Similarly to the MNIST case, we set 7 0 for the visible LMC chain.
https://mmlab.i cuhk. edu. hi/ www.h//ooe/t///eeeeeeeeeeeeeeeee html
18

---

B.3.4 Note on Reproduction
For Improved CD, train CIFAR-10 model from scratch using the training script provided by the authors without any modification We use the model with the best Inception Score to compute AUROC scores.
For DRI,, we use the official checkpoint T6 CIFAR-10 model and compute its energy to perform OOD detection.
Also for NVAE, use the official CIFAR- 10 checkpoint provided in the official repository We use negative log-likelihood as the outlier score. Due to computational budget constraint, we could only set the number of importance weighted sample be one -num_ iw
As in MNIST, we use the official CIFAR-10 checkpoint of NAE provided by the authors.
B.4
CIFAR-100 OOD Detection on Pretrained Representation
B.4.1 Datasets
CIFAR-100, CIFAR-10, SVHN, and CelebA datasets are used and are described in the previous section. Each image is resized 224 224 and fed ViT-B_ to produce 768-dimensional vector. MD and RMD operate with this vector. For other methods, the 768D vector is projected onto hypersphere.
B.4.2 Autoencoder Implementation and Training
Each encoder and decoder an MLP with two hidden layers where each layer contains 1024 hidden neurons. The leaky ReLU activation function is used in all hidden layers. We use D, = 128, 256, 1024. The autoencoders are trained to minimize the reconstruction error of the training data. During training, the Gaussian noise with the standard deviation of 0.01 is added to each training sample. The l2 norm of the encoder's weights are regularized with the coefficient of 1 x 10-6
B.4.3 MPDR Implementation and Training
The energy functions are also MLPs. The energy function MPDR-S has same architecture as the encoder of the autoencoder with D₂ 1. The energy function of MPDR-R R is an autoencoder with the latent dimensionality 1024. The energy functions are trained for 30 epochs with the learning rate of 1 10-4
B.4.4 Sensitivity to
We examine how ? affects MPDR's performance. As seen in Table 11 MPDR shows the best performance on the small y regime, roughly from 0.0001 to 0.001. Setting too large is detrimental for the performance and often even incurs training instabilities. It is interesting note that ; 0 also gives decent result. One possible explanation is that 7 0 reduces the negative sample distribution (x())ž) the model distribution pe (x).. which still valid negative sample distribution for training an EBM.
Table 11: Sensitivity of 7, demonstrated in CIFAR-100 experiment. AUROC values displayed.
7
CIFAR10
SVHN
CelebA
O
0.8580
0.9931
0.8456
0.0001
0.8626
0.9932
0.8662
0.001
0.8639
0.9918
0.8625
0.01
0.8496
0.9894
0.8576
0.1
0.8186
0.9424
0.8511
http:///ggcabbccmmrrrrrgggeeeeee divergence
http::///gthabb.ooo/////aaa////
19

---

Table 12: MVTec-AD detection and localization task in the unified setting. AUROC scores (percent)
are computed for each class. UniAD and DRAEM results are adopted from 39 a task is marked as boldface.
The largest value in
Detection UniAd
Localization UniAd 39
MPDR (ours)
DRAEM 40
MPDR (ours)
DRAEM 40
Bottle
100.0
99.7
97.5
98.5
98.1
87.6
Cable
95.5
95.2
57.8
95.6
97.3
71.3
Capsule
86.4
86.9
65.3
98.2
98.5
50.5
Hazclnut
99.9
99.8
93.7
98.4
98.1
96.9
Metal Nut
99.9
99.2
72.8
94.5
94.8
62.2
Pill
94.0
93.7
82.2
94.9
95.0
94.4
Screw
92.0
98.
98.3
95.5
Toothbrush
89.6
94.2
90.6
98.7
98.4
97.7
Transistor
98.3
99,8
74,8
95.4
97.9
65.5
Zipper
95.3
95.8
98.8
96.2
96.8
98.3
Carpet
98.0
98.8
98.5
98.6
Grid
97.9
98.2
99.3
96.9
96.5
98.7
Leather
100.0
100
98.7
98.5
98.8
97.3
Tile
100.0
99.3
98.7
94.6
91.8
98.0
Wood
97.9
98.6
99.8
93.8
93.2
96.0
Mean
96.0
96.5
88.1
96.7
96.8
87.2
B.5 MVTec-AD Experiment
We use MPDR-R with single manifold (i.e., without the manifold ensemble). Both the manifold and the energy are fully connected autoencoder 256D spherical latent space. For input-space Langevin Monte Carlo, the number of MCMC steps, the step size, and the noise standard deviation are 10, 0.1, and 0.1, respectively. No latent chain is used. The manifold is trained for 200 epochs with Adam of learning rate le-3, and the energy is trained for 20 epochs with Adam of learning rate le-4.
B.6 Acoustic Anomaly Detection
B.6.1
Dataset
The dataset consists of audio recordings with a duration of approximately 10 seconds, obtained through a single channel and downsampled 16kHz. Each recording includes both the operational sounds of the target machine and background noise from the surrounding environment. The addition of noise was intended to replicate the conditions of real-world inspections, which often occur noisy factory environments. The dataset covers six types of machinery, including four sampled from the MIMII Dataset (i.e., valve, pump, fan, and slide rail) and two from the ToyADMOS dataset (i.e., toy-car and toy-conveyor).
B.6.2 Preprocessing
We follow the standard preprocessing scheme used in the challenge baseline and many of challenge participants. The approach involves the use of Short Time Fourier Transform (STFT) transform each audio clip into spectrogram, which is then converted to Mel-scale. We set the number of Mel bands 128, the STFT window length as 1024, and the hop length (i.e., the number of audio samples between adjacent STFT columns) as 512. This configuration results in spectrogram with 128 columns that represented the number of Mel bands. To construct the final spectrogram, the mel spectra of five consecutive frames are collected and combined to form single row, resulting in spectrogram with 640 columns. Each row of the spectrogram sampled and used as input to models under investigation, with batch size of 512, meaning that 512 rows were randomly selected from the spectrogram each iteration. We standardize all data along the feature dimension to zero mean and unit variance.
20

---

B.6.3 Performance Measure
We refer to the scoring method introduced in 41 and use the area under the receiver operating characteristic curve (AUROC) and partial-AUROC (pAUROC) as quantitative measure of perfor- mance. PAUROC measures the AUC over the area corresponding to the false positive rate from 0 to a reasonably small value p. which we set our experiments as 0.1. Each measure defined as follows:
N
N
AUROC-
Σ
M(Ao(1))) Ao(2)))
(11)
N_
i=1
LpN N
pAUROC = LNN-JN+
Σ
Σ
H(AAo()))- Ao(27))
(12)
where {-,))= and {xt) J=1 are normal and anomalous samples, respectively, [.] indicates the flooring function, and H(-) represents hard-threshoid function whose output is 1 for positive argument and otherwise.
B.6.4 Implementation of Models
We utilize standard autoencoder model provided by DCASE 2020 Challenge organizers and compare our approach. The model comprises a symmetrical arrangement of fully-connected layers in the encoder and decoder; 5 fully connected hidden layers in the input and 5 in the output, with 128-dimensional hidden layers and 32-dimensional latent space. addition, we incorporate the IDNN model, which predicts the excluded frame using all frames except the central one instead of reconstructing the entire sequence. The IDNN model outperforms the autoencoder on ota-stationary sound signals, i.e., sounds with short durations. The model consists of encoder and decoder, which similar to the components hidden layers that contract in dimension (64, 32, 16) are used in the encoder and 3 layers that expand dimension (16, 32, 64) compose the decoder. The architectural design of each model follows the specifications outlined in their respective papers.
an autoencoder but has an asymmetric layout; 3 fully-connected
The MPDR models used for the experiment, MPDR-R and MPDR-IDNN, consists of an ensemble of autoencoders and an energy function, where the terms "R" and "IDNN" specify whether an autoencoder of IDNN used compute the energy, respectively. Each autoencoder consists of an encoder, spherical embedding layer, decoder; the encoder and decoder consist of 3 fully-connected layers each, with 1024-dimensional hidden space and a latent space with dimension chosen among 32, 64, 128. The autoencoder energy function used in the experiment is built using layers 5 fully-connected encoder layers and decoder layers, whose hidden layers have 128 nodes and latent layer is composed of 32 nodes. The layers of IDNN are indentical to that of the autoencoder version, except for the input and output layers whose dimensions differ and add up to the total number of sampled frames. LMC hyperparameters used are listed in Table 7
C Empirical Guidelines for Implementing MPDR
Here, present empirical tips and observations we found useful in achieving competitive OOD detection performance with MPDR. The list also includes heuristics that did not work when we tried.
The training progress can be monitored by measuring an OOD detection metric (i.e., AUROC) computed between test (or validation) inliers and synthetic samples uniformly sampled over the autoencoder manifold M. During a successful and stable run, this score tends increase smoothly. Metropolis-sty.e adjustment for LMC [64] did not improve OOD performance.
The choice of activation function in the energy function affects the OOD detection performance significantly. We found that ReLU and LeakyReLU provide good results in general.
In image datasets, stopping the training of the autoencoder manifold before convergence improves
OOD detection performance.
longer Markov chains, both visible and latent, do not always lead to better OOD detection performance.
21

---

A larger autoencoder does not always lead to better OOD detection performance.
larger energy function does not always lead better OOD detection performance.
22


Reviews:
[{"review_id": "SxPe3qdYsZ", "reviewer": "Reviewer_ekma", "summary": "This paper proposes a novel anomaly detection algorithm utlizing energy-based models (EBMs). The proposed method, Manifold Projection-Diffusion Recovery (MPDR), is based on recovery likelihood, a framework for learning energy functions by denoising data from artifically injected Gaussian noise. MPDR uses deterministic encoder and decoder to first project the data onto a latent space, add Gaussian noise in the latent space, and finally decode the noisy latent representation onto the data manifold. This would capture more relevant modes of variation in the data than Gaussian perturbations on the original data. The recovery likelihood of MPDR is derived and is shown to result in consistent estimation of the energy function. Negative samples are generated via Langevin Monte Carlo in the latent space. The paper also introduces additional variations, including different types of energy functions and the use of ensembles to generate diverse negative samples. Experiments demomstrate the effectiveness of MPDR for out-of-distribution detection on MNIST and CIFAR-100 datasets, as well as anomaly detection for acoustic signals.", "questions": "The experiments cover two image datasets and acoustic signals, which are relatively high dimensional data. It would be interesting to demonstrate the performance of MPDR on tabular data [1] (both low and high-dimensional) and explore the relevant modes which arise from applying MPDR to such datasets.\n\n[1] Han, S., Hu, X., Huang, H., Jiang, M. and Zhao, Y., 2022. Adbench: Anomaly detection benchmark. *Advances in Neural Information Processing Systems*, *35*, pp.32142-32159.", "limitations": "The paper covers the limitations of MPDR in sufficient detail, namely the sensitivity to the autoencoders used for latent space projection, questions around application to high-dimensional text or image data, and the fact that MPDR is not optimized to generate samples from simple distributions. ", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "Anomaly detection is a long standing problem in machine learning with a rich literature, and this paper proposes a novel step in the development of new algorithms. The idea of using recovery likelihood as well as projecting data onto a low-dimensional latent space are not novel, but this paper combines them to produce an original idea. The relevant background and motivation are presented with sufficient clarity, and the proposed method seems logically sound. The toy example presented in Figure 2 empirically verifies that MPDR captures more relevant modes of variation in the data.", "weaknesses": "The related works section cite most of the relevant previous works, but are not covered in sufficient detail, perhaps owing to the pade limitations. It is important to cover previous work exhaustively to demonstrate where the proposed method stands in relation. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer ekma,\n\nWe're truly grateful for your insightful feedback. Your time and effort in evaluating our work is deeply appreciated. Please allow us to address any questions you have.\n\n\n> Related works are not covered in sufficient detail.\n\nWe apologize that we couldn't discuss all related work in sufficient detail due to page constraints. If we are given an opportunity to update the manuscript with an extra page, we will expand the \"Related Work\" section to provide a deeper discussion. If you have a particular set of literature that you think we should cite and discuss, please let us know. We will include them in the updated manuscript.\n\n> Demonstrating the performance of MPDR on tabular data, such as Adbench.\n\nThank you for the suggestion. Based on your suggestion, we benchmarked MPDR using Adbench and provided the results below. MPDR was run on 47 tabular datasets provided by Adbench, and we compared MPDR with 13 baselines in Adbench. We reproduced the baseline results using the official repository of Adbench.\n\nFor each dataset, we ran each algorithm on three random splits and computed the AUROC on the corresponding test split. We then ranked the algorithms based on the averaged AUROC. In Table A, we present a summary table with the average rank across the 47 datasets.\n\n\n\n**Table A: Rank of each algorithm's AUROC score averaged over 47 datasets. The $\\pm$ sign indicates the standard error.**\n\n|     | Average Rank (The lower the better)|\n|:----|-----------:|\n| MPDR (ours) | **4.43** $\\pm$ 0.50 |\n| IForest | 5.28 $\\pm$ 0.39 |\n| OCSVM | 7.94 $\\pm$ 0.47 |\n| CBLOF | 5.98 $\\pm$ 0.53 |\n| COF | 9.23 $\\pm$ 0.58 |\n| COPOD | 7.10 $\\pm$ 0.61 |\n| ECOD | 6.97 $\\pm$ 0.58 |\n| HBOS | 6.53 $\\pm$ 0.51 |\n| KNN | 6.64 $\\pm$ 0.56 |\n| LOF | 9.04 $\\pm$ 0.62 |\n| PCA | 6.45 $\\pm$ 0.54 |\n| SOD | 7.91 $\\pm$ 0.52 |\n| DeepSVDD | 11.43 $\\pm$ 0.42 |\n| DAGMM | 10.09 $\\pm$ 0.52 |\n\n\nFor MPDR, we used MPDR-R, where the energy is an autoencoder. We do not employ the manifold ensemble, instead, a single manifold is used for perturbation. For both the manifold and the energy, the same MLP-based autoencoder architecture was used. The encoder and the decoder were MLPs with two 1024-hidden-neuron layers. If the input space dimensionality $D_x$ is smaller than or equal to 100, the latent space dimensionality is set to have the same dimensionality $D_z = D_x$. If $D_x>100$,  we set $D_z$ as 70% of of $D_x$. We employed 1 step of Langevin Monte Carlo in the latent space and 5 steps in the input space. The step sizes are 0.1 for the latent space and 10 for the input space. All the hyperparameters except $D_z$ are fixed across 47 datasets.\n\nAs shown in Table A, MPDR achieves highly competitive performance on Adbench, demonstrating a higher average rank than the isolation forest (although with some overlap of confidence interval). This result indicates that MPDR is a general-purpose anomaly detection algorithm capable of handling tabular data.\n\n\n\nOnce more, we're truly grateful for the time and effort you've dedicated to reviewing our paper. Should you have any additional questions, please don’t hesitate to leave a comment. If your concerns have been addressed well, we kindly ask if you'd consider raising your score.\n\nBest regards,\n\nAuthors.\n"}, {"review_id": "80XtnxxU8n", "reviewer": "Reviewer_dsqA", "summary": "This paper introduces an EBM-based model for anomaly detection in the latent manifold space. The proposed model first trains an autoencoder that maps a data point $x$ into the low-dimensional $z$, and then two-stage sampling strategy is developed to generate the original data via the LMC algorithm. Several notable designs are developed to complete the proposed model, such as manifold projection diffusion, two-state sampling, and perturbation ensemble. Experiments on images, vectors and acoustic signals show the strong performance of the method.", "questions": "1. The method artificially inject Gaussian noises into the latent space. What is the correlations between the simulated abnormal data and the real abnormal data in the test dataset?", "limitations": "yes.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "(1)\tThe idea that generation of negative samples from the manifold space sound good. This makes sure the starting points highly reflecting information, resulting in more discriminative generation. \n\n(2)\tSeveral practical strategies such as two-stage sampling and resembling multiple autoencoder are developed to benefit the anomaly detection performance. \n\n(3)\tExtensive empirical results and ablation studies show the efficiency of the model.\n", "weaknesses": "(1)\tlack of the results of widely-used anomaly detection dataset MVTec-AD \n\n(2)\tLack of well-known anomaly detection baselines, such as UniAD [1] and DRAEM [2]\n\n[1] Zhiyuan You et.al. A unified model for multi-class anomaly detection.\n\n[2] Vitjan Zavrtanik et.al. Draem-a discriminatively trained reconstruction embedding for surface anomaly detection.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer dsqA,\n\nThank you for taking the time to review our paper. We deeply value your feedback. Below, we will address your concerns and questions.\n\n> MVTec-AD dataset and comparison with UniAD and DRAEM\n\nThank you for your suggestion. MPDR also demonstrates promising performance on MVTec-AD. We present the anomaly detection performance of MPDR on MVTec-AD, along with an empirical comparison to UniAD and DRAEM. We will incorporate the MVTec-AD results in the updated manuscript and ensure that both papers are cited.\n\n**Table: MVTec-AD detection task in the unified setting. AUROC (percent) scores are computed for each class. UniAD and DRAEM results are adopted from You et al., 2022.**\n\n|         |     MPDR (ours)         |    UniAD     | DRAEM   |\n| -------|---------------------|---------------|---------------|\n| bottle | **100.0**$\\pm$0.00 | 99.7    |  97.5 |\n| cable | **95.5**$\\pm$0.01 | 95.2     | 57.8|\n| capsule | 86.4$\\pm$0.01 | **86.9**  | 65.3 |\n| hazelnut | **99.9**$\\pm$0.00 | 99.8 | 93.7 |\n| metal_nut | **99.9**$\\pm$0.00 | 99.2 | 72.8 |\n| pill | **94.0**$\\pm$0.01 | 93.7 | 82.2 |\n| screw | 85.9$\\pm$0.01 | **87.5** | 92.0 |\n| toothbrush | 89.6$\\pm$0.01 | **94.2** | 90.6 |\n| transistor | 98.3$\\pm$0.01 | **99.8** | 74.8 |\n| zipper | 95.3$\\pm$0.01 | 95.8 | **98.8** |\n| carpet | **99.9**$\\pm$0.00 | 99.8 | 98.0 |\n| grid | 97.9$\\pm$0.01 | 98.2 | **99.3** |\n| leather | **100.0**$\\pm$0.00 | **100** | 98.7 |\n| tile | **100.0**$\\pm$0.00 | 99.3 | 98.7 |\n| wood | 97.9$\\pm$0.00 | 98.6 | **99.8** |\n| mean | 96.0$\\pm$0.00 | **96.5** | 88.1 |\n\n\n\n**Table: MVTec-AD localization task in the unified setting. AUROC scores (percent) are computed for each class.  UniAD and DRAEM results are adopted from You et al., 2022.**\n\n|         |     MPDR  (ours)    |    UniAD     | DRAEM   |\n| -------|---------------------|---------------|---------------|\n| bottle | **98.5**$\\pm$0.00 | 98.1     | 87.6|\n| cable | 95.6$\\pm$0.00 | **97.3** |  71.3|\n| capsule | 98.2$\\pm$0.00 | **98.5** | 50.5|\n| hazelnut | **98.4**$\\pm$0.00 | 98.1 | 96.9|\n| metal_nut | 94.5$\\pm$0.00 | **94.8**| 62.2|\n| pill | 94.9$\\pm$0.00 | **95.0**| 94.4|\n| screw | 98.1$\\pm$0.00 | **98.3**| 95.5|\n| toothbrush | **98.7**$\\pm$0.00 | 98.4| 97.7|\n| transistor | 95.4$\\pm$0.00 | **97.9**| 65.5|\n| zipper | 96.2$\\pm$0.00 | 96.8| **98.3**|\n| carpet | **98.8**$\\pm$0.00 | 98.5| 98.6|\n| grid | **96.9**$\\pm$0.00 | 96.5| 98.7|\n| leather | 98.5$\\pm$0.00 | **98.8**| 97.3|\n| tile | 94.6$\\pm$0.00 |91.8| **98.0**|\n| wood | 93.8$\\pm$0.00 | 93.2| **96.0**|\n| mean | 96.7$\\pm$0.00 | **96.8**| 87.2 |\n\n\n\nWe follow the “unified” experimental setting of UniAD paper (You et al., 2022). Normal images from 15 classes are used as the training set, where no label information is provided. We use the same feature extraction procedure used in UniAD. Each image is transformed to a 272x14x14 feature map using EfficientNet-b4. When running MPDR, we treat each spatial dimension of a feature map as an input vector to MPDR, transforming the task into a 272D density estimation problem. We normalize a 272D vector with its norm and add a small white noise with a standard deviation of 0.01 during training. We use the maximum energy value among 14x14 vectors as an anomaly score of an image. For the localization task, we resize 14x14 anomaly score map to 224x224 image and compute AUROC for each pixel with respect to the ground true mask.\n\nWe use MPDR-R with a single manifold (i.e., without the manifold ensemble). Both the manifold and the energy are a fully connected autoencoder of 256D spherical latent space.\nFor input-space Langevin Monte Carlo, the number of MCMC steps, the step size, and the noise standard deviation are 10, 0.1, and 0.1, respectively. No latent chain is used. The manifold is trained for 200 epochs with Adam of learning rate 1e-3, and the energy is trained for 20 epochs with Adam of learning rate 1e-4.\n\nIn both MVTec-AD detection and localization tasks, MPDR achieves AUROC score that is very close to that of UniAD, outperforming other baselines including DRAEM. Note that MPDR-R and UniAD are based on the same approach that anomalies are characterized by a large reconstruction error. These two approaches may be combined to improve the result further.\n\n\n\n> What are the correlations between the simulated abnormal data and the real abnormal data in the test dataset?\n\n\nThank you for your question. In principle, the simulated abnormal data, which we refer to as negative samples in the manuscript, are not correlated with the actual test outliers. Firstly, no test outliers are utilized during any stage of the training process, including both the construction of the manifold and the learning of the energy. Moreover, as MPDR reaches convergence, the distribution of these negative samples should be close to the distribution of the training data, as in typical EBMs. We present some examples of these negative samples in Figure 4. These images bear minimal resemblance to test outliers like SVHN digits.\n\n\n\nOnce again, we appreciate the time and effort you've invested in reviewing our work. Please let us know if you have any further concerns or questions. If your concerns have been addressed satisfactorily, we kindly ask you to consider revising the score upward.\n\nBest regards,\n\nAuthors.\n"}, {"review_id": "UUxKcPrCh3", "reviewer": "Reviewer_gSqM", "summary": "This paper introduces an energy-based model based on the manifold of low-dimensional data. To train the EBMs, this paper takes the idea of maximum recovery likelihood and adds a layer of autoencoder to approximate the low-dimensional manifold representing the data. This introduces perturbation along the low-dim manifold representing the dataset. Simulation results are provided to show the performance.", "questions": "- How can we mitigate these biases to ensure the trained energy-based model generalizes well to out-of-distribution samples?\n- The encoder in this algorithm is assumed to be deterministic how does the deterministic assumption impact the performance and the ability to capture the full range of variations in the data?\n- Could authors please elaborate more on the conditions under which the consistency of the estimation by maximizing $\\log p(x|z^~)$ holds?- How does the use of a latent chain and latent space improve the sampling process compared to the visible chain? What are the advantages and disadvantages of using a latent LMC?\n- The paper mentions that the perturbation design, including the encoder-decoder pair (fe, fd) and noise magnitude $\\sigma$, significantly impacts the algorithm's performance. How can we effectively select the optimal perturbation design for different datasets and anomaly detection tasks?\n- The paper mentions that the autoencoder (fe, fd) and the noise magnitude $\\sigma$ should be independent of $\\theta$ and remain fixed during training. How does the fixed nature of the autoencoder and noise magnitude impact the algorithm's adaptability to different datasets and anomaly types?\n- Can you provide more details on how the simultaneous use of multiple perturbations enhances the algorithm's performance? Are there any potential trade-offs or challenges associated with this approach?\n- One of the main potential issues is the memory overhead in manifold ensembles. When utilizing multiple autoencoder manifolds in MPD, there is a memory overhead associated with processing multiple groups separately. How can this memory overhead be managed effectively to ensure efficient training while utilizing multiple autoencoders?\n- Can you provide more insights into how the choice of Dz affects the algorithm's ability to detect anomalies? How can we determine the optimal combination of autoencoders with varying Dz for different types of data? Specifically, in high-dimensional data such as images, how does MPDR framework deal with curse of dimensionality and maintain good performance with relatively smaller autoencoders?", "limitations": "Please check weaknesses and questions.", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- It is easy to follow. \n- Sufficient literature review.", "weaknesses": "- The novelty of this paper is moderate at best. It only adds a deterministic encoder/decoder to the original problem of likelihood recovery. \n- The accuracy of the manifold approximation is as good as the autoencoder approximation of the dataset. \n- The generation of the negative samples concentrated neat the manifold could potentially introduce biases. \n- lack of comparison to some recent advancements in the topic such as:\n   * Anomaly Detection in Networks via Score-Based Generative Models by Gavrilev et. al, 2023. \n   * Enhancing Unsupervised Anomaly Detection with Score-Guided Network by Huang et. al, 2022.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer gSqM,\n\nThank you for dedicating your time to reviewing our work. We sincerely appreciate your detailed feedback. Here, we would love to answer your questions in depth.\n\n**The novelty of the paper**\n\nWe would like to highlight that simplicity does not always equate triviality. MPDR (our method) is the first example of training an energy-based model using the recovery likelihood framework with non-Gaussian perturbation. Even though MPDR is based on autoencoders and the recovery likelihood, it achieves significantly better out-of-distribution detection performance than both of the base algorithms, as repeatedly shown in our experiments (Table 1, 3, and 4, where DRL is an algorithm based on the recovery likelihood). Please consider that the paper also provides novel techniques, such as perturbation ensemble and the latent space LMC, which deliver stable training and improved performance.\n\n\n**Accuracy of the manifold approximation is as good as the autoencoder approximation**\n\nWe are afraid that this statement does not accurately reflect what is presented in the paper.  MPDR can learn the correct data distribution (and thus the correct data manifold) even when an autoencoder provides a crude approximation. In Figure 2, the manifold learned by an autoencoder (the gray line) does not reflect the cluster structure of data, being a crude approximation. However, the resulting energy function correctly captures the underlying clusters, as shown in Figure 3.\n\nMPDR can learn from an imperfect autoencoder because input-space LMC is not confined to the autoencoder’s manifold and can generate off-manifold samples.\n\n**Potential bias due to near-manifold negative samples / How can we reduce bias**\n\nIn theory,MPDR provides asymptotically unbiased estimation of probability density, as we show in Section 3.2 and Appendix A. This is possible because an ideal MCMC sampling can cover the entire input space.\n\nIn practice, the bias may exist, and the root cause of the bias is finite-length MCMC which does not mix over the entire space. The training techniques we introduced contributes to mitigate this bias. For example, the manifold ensemble provides diverse starting points for MCMC. Also, the use of autoencoder-based energy function (MDPR-R) automatically assigins high energy on points that are far from data. \n\nMeanwhile, the inductive bias from the near-manifold negative samples could be useful.  Other energy-based models, such as IGEBM, also suffer from biases from finite MCMC. When compared to them, MPDR achieves stronger OOD detection performance, as shown in Table 1, 3, and 4.\n\n**Comparison to other recent advancements**\n\n> Anomaly Detection in Networks via Score-Based Generative Models by Gavrilev et. al, 2023.\n\nUnfortunately, Gavrilev et al., 2023 is not directly comparable with MPDR, because it is designed for a different task, graph node anomaly detection. We will cite and mention this work in the updated manuscript, as extending MPDR to graph data is an interesting future direction.\n\n> Enhancing Unsupervised Anomaly Detection with Score-Guided Network by Huang et. al, 2022.\n\nThank you for letting us discover an interesting work. We compare MPDR with SG-AE, and SG-RSRAE, the method proposed by Huang et al., 2022. We run them on MNIST, as both papers contain MNIST experiments. We use the digit 4 as inliers and the rest of the digits as outliers. We use MPDR-R model. All the hyperparameters are the same as in the MNIST experiment presented in the main manuscript. We will cite this work and include this result in the updated manuscript.\n\n|       | AUC-ROC| AUC-PR|\n|:---|----:|----:|\n|SG-AE| 0.939±0.005| 0.563±0.015|\n|SG-RSRAE| 0.951±0.010| 0.736±0.062|\n| MPDR (ours) | **0.975**±0.004 | **0.997**±0.001|\n\n**Other questions**\n\n> how does the deterministic assumption impact the performance and the ability to capture the full range of variations in the data?\n\nThe deterministic assumption does not play a critical role in MPDR but is employed mostly for convenience. An autoencoder, either deterministic or stochastic, usually captures the overall variation of data well, achieving low reconstruction error for the training data. Changing a deterministic encoder requires us to solve the integral $\\int p(\\tilde z | z) p(z|x) dz$ in Eq.6. This integral is easy when we are using Gaussian distributions in the Euclidean space but can be difficult in other situations.\n\n> Conditions for consistency\n\nThe consistency of MPDR requires several assumptions: the infinite number of data, correctly specified and identifiable model, and non-zero recovery probability, i.e., $p(x|\\tilde{z})>0$ for all $x$ and $\\tilde{z}$. Please find more detail in Appendix A. If you are curious about a particular aspect of the condition, please let us know.\n\n> How does the latent chain improve the sampling process? The advantages and disadvantages of using a latent LMC?\n\nThe latent chain facilitates the exploration of MCMC. A small step in the latent space corresponds to a much larger distance in the input space. However, it requires additional computation time. Also, operating only in the latent space, a latent LMC can not explore the off-manifold region in the input space. Therefore, the input space LMC is always necessary for successful training.\n\n> How can we select the optimal perturbation design, optimal combination of autoencoders, and Dz?\n\nAs in other outlier detection algorithms, those hyperparameters can be selected using a separate validation OOD dataset, as proposed in Appendix A of Hendrycks et al., 2018.\n\nHendrycks et al., Deep Anomaly Detection with Outlier Exposure, 2018.\n\nDue to the rebuttal's space constraint, we will address the rest of the questions during the discussion period.\n\nYour comprehensive review of our paper is deeply appreciated. Please share any lingering questions or concerns. If they have been resolved, we would be grateful for your consideration in updating the review score.\n\nBest regards,\n\nAuthors.\n"}, {"review_id": "yJ8784u9qv", "reviewer": "Reviewer_Pi2D", "summary": "Paper proposes MPDR, a novel method of using auto-encoders for training EBM. \nSome practical techniques are introduced.\nExtensive numerical experiments are done.", "questions": "Can we have some theoretical analysis on when proposed method has significant advantage, when not? Or is the proposed method better in general and some loss are just random?", "limitations": "No negative societal impact is seen.", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "Numerical experiments cover a large scope of benchmarks. And it shows superiority on most benchmarks. ", "weaknesses": "No theoretical guarantee is provided.\nAnd it doesn't show dominant superiority on some datasets. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer Pi2D\n\nThank you so much for your comment. We would like to address your concerns in detail.\n\n\n> No theoretical guarantee is provided.\n\nWe are concerned that this statement does not accurately reflect what is presented in the paper. Please refer to the end of Section 3.2 and Appendix A, where we provide the theoretical guarantee that the proposed method offers an asymptotically unbiased estimation of the underlying density. Consistent estimation of density can lead to accurate recovery of the distribution’s support (Cuevas and Fraiman, 1997), enabling successful out-of-distribution detection.\n\nAntonio Cuevas. Ricardo Fraiman. \"A plug-in approach to support estimation.\" Ann. Statist. 25 (6) 2300 - 2312, 1997.\n\n> The method does not show dominant superiority on some datasets. When does the proposed method has significant advantage?\n\nQualitatively speaking, the proposed method is designed to leverage the inductive bias that the data possesses a pronounced low-dimensional structure. This manifold assumption is both useful and effective since it approximately holds true for a wide range of data. However, this assumption may not be as effective in complex datasets, such as high-resolution images.\n\nIs there a specific set of results that concerns you? Please let us know. We will try to provide an explanation for the result.\n\n> Can we have some theoretical analysis on when proposed method has significant advantage, when not?\n\nAlmost all the generative anomaly detection methods compared in our paper offer consistent density estimation, making them nearly equivalent from a theoretical perspective. The difference in empirical performance originates from the inductive biases of the algorithms and their implementation details. As with many other deep learning algorithms, it is challenging to quantify the contribution of these aspects. However, exploring this could be a highly rewarding avenue for future work.\n\n\nAgain, thank you for your time in reviewing our work. Should you have any additional questions, please leave us a comment. If your concerns have been addressed, we kindly ask if you would consider enhancing the score.\n\nBest regards,\n\nAuthos.\n"}, {"review_id": "YZB0COX9RB", "reviewer": "Reviewer_Tcts", "summary": "The authors introduce a novel algorithm, Manifold Projection-Diffusion Recovery (MPDR), for training energy-based models (EBMs) that improve the performance of anomaly detection tasks. These tasks are highly relevant in real-world applications like industrial surface inspection, machine fault detection, and particle physics.\n\nUnlike conventional EBM training methods, MPDR harnesses low-dimensional structures in the data to generate more informative negative samples. It works by initially perturbing a data point along a manifold approximating the training dataset and then trains the EBM to maximize the probability of recovering the original data.\n\nA significant aspect of this new method is its use of Manifold Projection-Diffusion (MPD), which replaces Gaussian noise with perturbations reflecting the data's low-dimensional structure. This approach provides more meaningful insights into variations within the data.\n\nThe authors show that MPDR provides consistent density parameter estimation under standard assumptions, is compatible with any energy function, and can work with multiple autoencoders - an advantage over existing algorithms. Moreover, it demonstrates good results even with lightweight autoencoders, making it computationally efficient.\n\nThe paper also presents practical strategies for deploying MPDR, such as two-stage sampling, energy function design, and ensemble techniques using multiple autoencoders.\n\nThrough experimental testing on various data types, including images, representation vectors, and acoustic signals, the authors demonstrate that MPDR significantly improves unsupervised anomaly detection tasks, outperforming other deep generative models.", "questions": "1. **Theoretical Analysis:** Could the authors provide a more rigorous analysis of the convergence properties of their proposed method? Specifically, how does the choice of manifold affect the stability and speed of learning in MPDR?\n\n2. **Comparison with Other Methods:** It would be beneficial to see a wider comparison with other state-of-the-art anomaly detection methods. Could the authors elaborate on why one might choose MPDR over other established methods in specific scenarios?\n\n3. **Parameter Sensitivity:** How sensitive is the MPDR algorithm to the initial choice of parameters? Is there a recommended procedure for parameter tuning, or guidelines that could assist users in achieving optimal performance?\n\n4. **Real-World Applications:** Could the authors possibly demonstrate the application of their model on real-world datasets or provide case studies? This could help showcase the practical implications of the proposed method.\n\n5. **Computational Complexity:** The paper mentions that MPDR can work well even with lightweight autoencoders, but could you please clarify further on its computational complexity, training time, and inference time? Would there be any scalability issues when applying this method to larger datasets?\n\nLooking forward to the authors' response to these points.", "limitations": "From the information provided, it does not appear that the authors have explicitly discussed the limitations and potential negative societal impacts of their work. Therefore, here are some suggestions for addressing these points:\n\n**Limitations:**\n \n1. **Robustness to Noise:** How well does the MPDR framework handle noise in the data? Real-world data often contain a significant amount of noise, which may distort the underlying manifold structure that MPDR relies on.\n\n2. **Scalability:** The scalability of the model hasn't been addressed. Can the method be applied efficiently to very large datasets? What would be the computational requirements in such cases?\n\n3. **Multimodality:** How effectively can MPDR handle multimodal or highly dimensional distributions? This is a common challenge in many real-world anomaly detection tasks.\n\n**Potential Negative Societal Impacts:**\n\nWhile this study primarily focuses on improving anomaly detection methods, which generally have positive implications (e.g., defect detection in manufacturing, early detection of diseases, etc.), any technology has the potential to be misused.\n\n1. **Privacy Concerns:** Anomaly detection tools can potentially be used to identify outliers or anomalies in personal behavior or characteristics, leading to potential privacy concerns if misused, especially in contexts like surveillance or social profiling.\n\n2. **False Positives/Negatives:** In critical applications, false positives or negatives can have serious repercussions. For instance, in health care, a false positive might cause unnecessary worry or treatment, whereas a false negative could delay necessary intervention.\n\nIt would be beneficial to see the authors address these potential issues and discuss how they might be mitigated.", "rating": 5, "confidence": 1, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "**Originality:** \n\nThe paper is highly original in its formulation and approach to training energy-based models (EBMs) for anomaly detection. The development of the Manifold Projection-Diffusion Recovery (MPDR) represents a creative combination of existing methodologies, such as the use of autoencoders for efficient training and manifold projections for capturing low-dimensional data structures. The method of perturbing data points along a low-dimensional manifold that approximates the training dataset and then training the EBM to maximize the recovery of the original data is a fundamentally new approach.\n\n**Quality:** \n\nThe quality of the paper is commendable. The authors present a clear problem statement, propose a novel solution, and provide empirical evidence supporting their claims. They also delve into the theoretical backing of the proposed method, offering a consistent density parameter estimation under standard assumptions. The paper includes detailed experimental results, highlighting the strength of MPDR across diverse anomaly detection tasks and data types.\n\n**Clarity:** \n\nThe exposition in the paper is clear and well-organized. The authors have done an excellent job explaining complex concepts and methodologies, which makes the paper accessible even to readers who may not be experts in the field. The use of illustrative figures and well-explained algorithms further enhances the clarity of the paper.\n\n**Significance:** \n\nThe significance of this work is substantial, given the wide range of practical applications of anomaly detection. By providing a more efficient and effective way to train EBMs, MPDR could significantly improve performance in areas like industrial surface inspection, machine fault detection, and particle physics. Furthermore, the ability of MPDR to perform effectively with lightweight autoencoders means it can be used in scenarios where computational resources are limited, making it relevant to a broader audience.", "weaknesses": "While the paper presents a compelling new approach, here are a few areas that could be addressed or clarified further:\n\n**Theoretical Analysis:** While the authors provided a theoretical justification for consistent density parameter estimation under standard assumptions, it would be beneficial to include more analysis of the proposed method's convergence properties. Understanding how the choice of manifold affects the convergence and stability of learning would also be useful.\n\n**Comparison with Other Methods:** The paper could benefit from a more comprehensive comparison with other state-of-the-art methods for anomaly detection. Not only should this include direct quantitative comparisons on common datasets, but also qualitative discussions about when and why one might prefer the proposed method over others.\n\n**Parameter Sensitivity:** It is not clear how sensitive the results are to the choice of parameters within the MPDR framework. It would be beneficial for practical applications to know how much tuning is needed to achieve optimal performance and how robust the method is to variations in these parameters.\n\n**Real-World Applications:** While the experiment demonstrates the effectiveness of the proposed algorithm in various data types like images, vectors, and acoustic signals, applying the model to real-world datasets and providing case studies can strengthen the paper. This will help readers understand its practical implications better.\n\n**Computational Complexity:** The paper mentions that MPDR performs well with lightweight autoencoders, which indicates computational efficiency. However, a more explicit discussion or analysis of the computational complexity of the proposed method, including both training time and inference time, would provide valuable information to practitioners considering this method.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer Tcts,\n\nWe would like to express our sincere gratitude for taking the time and effort to review our paper. We greatly appreciate your highly detailed and constructive comment and are happy to answer your questions.\n\n**Theoretical Analysis**\n\n> How does the choice of manifold affect the stability and speed of learning in MPDR from a theoretical perspective?\n\nThank you for the insightful question. Indeed, the theoretical analysis can shed light on the choice of a manifold. According to our analysis on the asymptotic normality of estimation error, an encoder with a higher compression rate tends to yield better estimations. That is, the more information lost during encoding, the smaller the estimation error becomes. This understanding aligns with our empirical observation that L2 regularization on the encoder weights often yields improved results. We will incorporate this analysis into the updated manuscript.\n\nMore detailed argument is given as follows:\nIn the limit of infinite data, the parameter estimation error from maximum recovery likelihood follows a zero-mean normal distribution as in maximum likelihood estimation. The covariance of this distribution is determined by the inverse of a term $I(\\theta)=E_{p(x,\\tilde{z})}[-\\nabla^2_\\theta \\log p_\\theta (x|\\tilde{z}) ]$. Since $\\nabla^2_\\theta \\log p_\\theta (x|\\tilde{z})= \\nabla^2_\\theta \\log p_\\theta(x) - \\nabla^2_\\theta \\log p_\\theta(\\tilde{z})$, this term is decomposed as follows:\n$$ I(\\theta) = I_0 (\\theta) + E_{p(x,\\tilde{z})}[\\nabla^2_\\theta \\log p_\\theta(\\tilde{z}) ],$$\nwhere $I_0 (\\theta)$ is Fisher information. The term $\\nabla^2_\\theta \\log p_\\theta(\\tilde{z})$ becomes 0 when the encoder compresses all $x$ into a single $z$.\n\nHowever, with an encoder that compresses everything, we could not leverage the inductive bias of the manifold assumption. Therefore, in practice, we need to use a moderately compressing encoder.\n\n\n\n\n**Comparison with Other Methods**\n\n> It would be beneficial to see a wider comparison with other state-of-the-art anomaly detection methods. \n\nDue to page constraints, we primarily compared the proposed method with the closely related generative anomaly detectors. At the request of other reviewers, we have conducted additional comparison experiments. If there's a specific algorithm you'd like to see compared, please let us know. We are happy to incorporate additional comparative experiments in our updated manuscript.\n\n> Could the authors elaborate on why one might choose MPDR over other established methods in specific scenarios?\n\nSince MPDR explicitly leverages low-dimensional representations, we believe MPDR is a recommendable choice when data is expected to have a pronounced manifold structure.\n\n\n**Parameter Sensitivity**\n\n> How sensitive is the MPDR algorithm to the initial choice of parameters? Is there a recommended procedure for parameter tuning, or guidelines that could assist users in achieving optimal performance?\n\nPlease find Appendix B.2 for sensitivity analysis for important hyperparameters such as the noise magnitude and the latent dimensionality. As in other outlier detection algorithms, those hyperparameters can be selected using a separate validation OOD dataset, as proposed in Appendix A of Hendrycks et al., 2018.\n\nHendrycks et al., Deep Anomaly Detection with Outlier Exposure, 2018.\n\n**Real-World Applications**\n\n> Could the authors possibly demonstrate the application of their model on real-world datasets or provide case studies? \n\nPlease find the experiments in Section 4.4 which utilized audio data collected from real-world mechanical devices. MPDR is able to deliver improvement in this dataset as well.\n\n\n**Scalability & Computational Complexity**\n\n> Could you please clarify further its computational complexity, training time, and inference time? Would there be any scalability issues when applying this method to larger datasets?\n\nIn our CIFAR-10 experiment, for example, training of MPDR takes about 5 hours on a single V100 GPU, including the training of autoencoders used in manifold projection-diffusion. Compared to other energy-based models that is typically trained for more than one day, MPDR requires shorter training time. Using the same V100 GPU, the inference over the whole test set takes 3.5 seconds, which corresponds to ~2800 images per second. The inference is lightweight because the autoencoders used in perturbation are only used during training and are not required for making predictions.\n\nWith respect to the number of data, MPDR is as scalable as other deep learning algorithms, as its training is fully based on stochastic gradient descent. \n\n\n**Multimodality**\n\n> How effectively can MPDR handle multimodal distributions?\n\nThis could be a highly interesting future research direction. One possible method is to build a joint latent representation of multiple modalities and run MPDR in that space.\n\n**Potential Negative Impacts**\n\nWe believe that the concerns raised regarding negative impacts are not specific to the proposed algorithm but pertain to machine learning algorithms in general. At the very least, issues related to false positives/negatives can be mitigated by developing a more refined algorithm, and MPDR aims to contribute in that direction.\n\nWe thank you again for your detailed comment. Please let us know if you have any other unresolved concerns. If your concerns are resolved, please kindly consider raising the evaluation score.\n\nBest regards,\n\nAuthors.\n"}]

Author rebuttal:

Meta-review: This paper addresses an energy-based model for anomaly detection. It presents the algorithm, referred to as 'manifold projection-diffusion recovery' which leverages the separately-trained autoencoder and noise perturbation in the latent space. All reviewers feel that the paper is well written and contains substantial contributions. Despite the limited novelty and the lack of theoretical guarantee, the proposed method is logically sound. The authors did a good job in their rebuttal, resolving most of concerns raised by reviewers. All five reviewers agree that the paper is deserved to be presented at the conference.
Decision: Accept (poster)