On Diffusion Modeling for Anomaly Detection
Authors: Victor Livernoche
Vineet Jain
Yashar Hezaveh
Siamak Ravanbakhsh
Venue: ICLR 2024 spotlight (2024)
Keywords: Diffusion based models
Anomaly detection
Probabilistic Inference
Primary area: probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
TL;DR: Identify anomalies in a dataset by estimating the diffusion time, anomalies have higher diffusion times

Abstract: Known for their impressive performance in generative modeling, diffusion models are attractive candidates for density-based anomaly detection. This paper investigates different variations of diffusion modeling for unsupervised and semi-supervised anomaly detection. In particular, we find that Denoising Diffusion Probability Models (DDPM) are performant on anomaly detection benchmarks yet computationally expensive. By simplifying DDPM in application to anomaly detection, we are naturally led to an alternative approach called Diffusion Time Estimation (DTE). DTE estimates the distribution over diffusion time for a given input and uses the mode or mean of this distribution as the anomaly score. We derive an analytical form for this density and leverage a deep neural network to improve inference efficiency. Through empirical evaluations on the ADBench benchmark, we demonstrate that all diffusion-based anomaly detection methods perform competitively for both semi-supervised and unsupervised settings. Notably, DTE achieves orders of magnitude faster inference time than DDPM, while outperforming it on this benchmark. These results establish diffusion-based anomaly detection as a scalable alternative to traditional methods and recent deep-learning techniques for standard unsupervised and semi-supervised anomaly detection settings.

Published as a conference paper at ICLR 2024
ON DIFFUSION MODELING FOR ANOMALY DETEC- TION
Victor Livernoche 13
Vineet Jain
Yashar Hezaveh 23
Siamak Ravanbakhsh"
School of Computer Science, McGill University
Department of Physies, University of Montreal
Mila Quebec AI Institute
Equal contribution
ABSTRACT
Known for their impressive performance in generative modeling. diffusion mod- els attractive candidates for density-based anomaly detection. This paper in- vestigates different variations of diffusion modeling for unsupervised and semi- supervised anomaly detection. In particular, find that Denoising Diffusion Probability Models (DDPM) are performant on anomaly detection benchmarks yet computationally expensive. By simplifying DDPM in application anomaly detection, we are naturally led to an alternative approach called Diffusion Time Estimation (DTEEDDTE estimates the distribution over diffusion time for given input and uses the mode or mean of this distribution the anomaly score. We de- rive an analytical form for this density and leverage a deep neural network to im- prove inference efficiency. Through empirical evaluations on the ADBench bench- mark, we demonstrate that all diffusion- -assed anomaly detection methods perform competitively for both semi-supervised and unsupervised settings. Notably, DTE achieves orders of magnitude faster inference time than DDPM, while outper- forming it on this benchmark. These results establish diffusion-based anomaly detection as a scalable alternative to traditional methods and recent deep-learning techniques for standard unsupervised and semi-supervised anomaly detection set- tings.
INTRODUCTION
Anomaly detection seeks identify observations that differ from the others to such large extent that they are likely generated by a different mechanism (Hawkins) 1980). This is a longstanding research problem in machine learning with applications in various fields ranging from medicine (Pachauri & Sharma 2015] Salem et al. 2013), finance (Ahmed et al. 20166), security (Ahmed) ctt 2016)). manufacturing (Susto et al. 2017), particle physics (Fraser et al. 2022) and geospatial data YYaiii et al. 2006). Despite its significance and potential for impact (e.g., leading to the discov- ery of new phenomena), to this day traditional anomaly detection methods, such as nearest neigh- bours, reportedly outperform deep learning techniques on various benchmarks (Han et al.) (022) by sigmificant margin. This true for unsupervised, sumirvuper and supervised anomaly detection tasks. However, the growing number of applications involving high-dimensional data and massive datasets are beginning challenge the classical, and in particular non-parametric, tech- niques, and there is need for scalable, interpretable, and expressive deep learning techniques for anomaly detection.
In recent years, denoising diffusion probabilistic models (DDPMs) (Ho et al. 2020) have received much attention as powerful class of generative models. While these models have been successfully utilized for anomaly detection in domain-spedie image datasets (Wolleb et al., 2022 Zhang al.
2023a Wyatt et al. 2022). a comprehensive exploration their applicability for general-purpose anomaly detection across diverse tabular, image, and natural language datasets is notably absent.
Our starting point the observation that DDPM exhibits competitive performance compared to previous approaches for unsupervised and semi-supervised anomaly detection. These are some of
Code available https///gihhuhcco////iiiDt

---

Published as a conference paper at ICLR 2024
the most challenging settings, where either unlabelled mix of normal and anomalous samples are available for training or, at best, the training data only includes normal samples. However, the expressivity and interpretability of DDPM come with considerable computational cost. This computational complexity poses challenges for anomaly detection tasks involving large datasets or data streams.
In anomaly detection using DDPM, we deterministically "denoise" the input and measure the dis- tance its denoised reconstruction; a large distance indicates an anomaly. Since we only use this distance for outlier identification, in order to reduce the complexity of the diffusion-based approach, we propose to directly estimate this distance, which is correlated with diffusion time.
More precisely, we estimate the posterior distri- bution of diffusion time (or noise variance) for given input. This estimated distribution serves as guide for identifying anomalies, as they are anticipated to exhibit higher posterior density at larger time steps compared to normal sam-
DTE (Non Parametrici
10
*
OCS
AD DDPM
GDAD
LOr
DeeDSVDD
COROD
PlanarFlow PCA
ples. In particular, we use the mode or mean of this distribution as the anomaly score. We de-
DA DOA
MCD
20-1
rive analytical form for this posterior distri- bution, enabling its non-parametric estimation. We see that the non-parametric approximation produces ranking for anomalies that is identi- cal to k-Nearest Neighbours (kNN) for anomaly detection. We then propose parametric model, a deep neural network, allowing us to leverage the generalization capability and efficient infer- ence time of deep learning.
0
GANomally
HBOS
ERLOF
DTE (Categorical)
DTE (Inverse Gamma)
Figure 1: Average inference time vs. aver- age AUC ROC for all 57 ADBench datasets in the semi-supervised setting. Lower right is bet- ter (DTE Categorical). Colour scheme: red (diffusion-based), green (deep learning), blue (classical).
We provide an extensive evaluation compared to classical and other deep models for differ- ent anomaly detection settings on more than 57
datasets from ADBench (Han et al. 2022). Our empirical results suggest that using a single deep neural network architecture across all datasets and settings makes the diffusion model competitive with classical and other deep models. Figure [ shows the efficiency and effectiveness of differ- anomaly detection algorithms across all datasets in ADBench. Notably, our proposed method surpasses the direct application of DDPMs, achieving substantial improvements in inference time.
The contributions of our work are summarized follows:
Evaluation of denoising diffusion probabilistic models on various anomaly detection tasks encompassing tabular data and embeddings of images and natural language datasets.
Development of a simplified approach that models the posterior distribution over diffusion time as a proxy for anomaly detection.
Derivation of an analytical form of the posterior distribution of diffusion time and develop- ment of non-parametric estimator that leads us to kNN.
Introduction of a parametric approach utilizing a deep neural network for improved gener- alization and scalability.
Implementation of additional baselines and extensive evaluation on 57 datasets from AD- Bench, showcasing competitive performance compared to classical and existing deep- learning-basee anomaly detection algorithms.
Investigation into the interpretability of crrtusion-basedd methods, including our novel ap- proach, highlighting their strengths and limitations.
Exploration of optimal representation selections for image datasets with diffusion methods.
2 PRELIMINARIES
classification of anomaly detection methods is based on the availability of labelled data. Super- vised setting is similar binary classification with unbalanced classes since the number of anomalies
2

---

Published as a conference paper at ICLR 2024
in the data is generally small fraction of the total number of samples. This setup is limited to the identification of known anomalies. The more challenging unsupervised setting assumes that the data is mix of normal and anomalies, without access to labels. Methods in this category often make assumptions about the data-generation process. Therefore, embedding techniques and deep genera- tive models are prime candidates. However, challenge for deep models the fact that they tend to model the anomalies within the input data more easily, making the task of identifying them harder. A middle ground between supervised and unsupervised is semii-uperviied or one-class classification setting, where one has access to purely normal samples during training, yet anomalies of unknown nature can exist inference time. Perhaps confusingly, the term semi-supervised is also used when partial labelling of anomalies is available during the training. In this work, we are interested in identifying anomalies with an unknown distribution and therefore do not assume access to any bel information for outliers. That is we consider both unsupervised and the one-class classification version of semi-supervised anomaly detection.
2.1 DIFFUSION PROBABILISTIC MODELS
A diffusion process is a stochastic process characterized by probability distribution that evolves over time, governed by the diffusion equation. Diffusion probabilistic models (Sohl-Dickstein et al,) 2015 Ho et al., 2020) are latent variable probabilistic models where the state time steps larger than zero are considered latent variables. Let X ((xo) denote the data and x...... . denote the corresponding latent variables. The forward diffusion process is generally fixed to add Gaussian noise at each timestep according to variance schedule $1. Br. The approximate posterior
((x.TT Xo) given by,
(XxIT Xo)
II ((xx|Xx11)) ((x,(x)-1) == N(x; V 1,,
Choosing the transitions as Gaussian distributions enables sampling x, at any time in closed form. Let 0+ := 1 - Be and ā = 1I. then,
q(x, (x)) = Nxxx / Vaxoo (1 (1-
(2)
Diffusion probabilistic models then learn transitions that reverse the forward diffusion process. Start- ing at p(xx) = (xr;; 0, I), the joint distribution of the reverse process pe (xx:r) is given by,
T
(=(x).)) == p( (x) II
Po (x, 1x xx)),
Po (x (xx) = (xx: 140 (xt,1) (), Eg ())
(3)
This parameterized Markov chain also called the reverse process, can produce samples matching the data distribution after finite number of transition steps.
3 DIFFUSION TIME ESTIMATION
Denoising diffusion probabilistic models (DDPM), as introduced in (Ho et al. 2020). can be used generate samples matching the data distribution even in high-dimensional spaces. The reverse diffusion process implicitly learns the score function of the data distribution and can be used for the likelihood-based identification of anomalies. A common approach used in prior works on anomaly detection using diffusion models (Wolleb et al. 2022 Zhang et al., 2023a Wyatt et al. 2022) is to reconstruct input samples by simulating the reverse diffusion chain and then using the reconstruction distance to identify anomalies. This is particularly useful where anomalies are localized in the image, and the difference between the input and its reconstruction identifies this localized anomaly. While all previous works focus this scenario in image data, we consider the broader problem of identification anomalous samples without assumptions on data type or the nature of the anomaly.
Toward this objective, we evaluate the reconstruction-basse approach using DDPMs on the AD- Bench benchmark, which comprises 57 datasets, including tabular, image, and natural language data. We observe that the choice of timestep at the start of reverse diffusion is arbitrary, yet it can significantly affect the anomaly detection performance. We found that using 25% of the maximum timestep globally leads to good results; the Appendix A for an ablation.
3

---

Published as a conference paper at ICLR 2024
(a) Data
Diffused Gaussian mix.
(c) DDPM vector-field
(d) DTE Postr. mode
(c) Gradient of [d]
(f) Denoising using [c]
Figure 2: DDPM and DTE on toy dataset shown in (a). (b) shows the Gaussian density function associated with the lowest timestep of DDPM and (c) shows the vector field corresponding to the gradient of this density. (d) plots the mode of the DTE posterior distribution over diffusion time, which we show in subsequent sections is an inverse Gamma distribution. (e) shows the gradient of (d), and (f) shows the flow associated with this gradient, showing that random samples are mapped toward the data manifold.
As anticipated, the expressivity of these models allows them to perform competitively compared prior work. However, inference for single data point involves simulating the reverse diffusion chain in its entirety, making this approach computationally expensive. By quantifying the disparity between the reconstructed output and the original input, the objective to effectively capture the deviations of anomalous samples from the underlying data manifold. We contend that modeling the score function by learning the reverse process is unnecessary if the objective is only the identification of anomalies.
Building upon this idea, we propose a much simpler approach that does not require modeling the reverse diffusion process but instead models the distribution over diffusion time corresponding to noisy input samples. Assuming anomalies are distanced from the data manifold, the density for larger timesteps should have higher value for anomalies, enabling their probabilistic identification. This can be seen as a direct estimation of reconstruction error.
simulate anomalous samples using diffusion process and train neural net- work to predict the diffusion time corresponding to the noisy samples. Provided that the noisy samples cover the entire feature space, this procedure should also capture potential anomalies. Fig- ure 2 contrasts DDPM and DTE on a toy dataset. The success of our method in using diffusion for anomaly detection is due to the space-filling property of the diffusion process; different regions of the space are sampled at different rates, depending on their proximity to the data manifold. To our knowledge, this the first setting that uses this property of diffusion beyond its application in learn- ing time-de -eeeddepndentt score functions for generative modelling. While in that setting, the estimated score is able meaningfully approximate the true score over the entire space, we show that we are able to approximate the diffusion time for arbitrary points, including normal or anomalous points.
More concretely,
3.1 POSTERIOR DISTRIBUTION OF DIFFUSION TIME
Assuming X, E Rd produced through a diffusion process, starting from the data manifold, our goal in this section is to identify the distribution over its diffusion time, surrogate for its dis- tance from the manifold. The diffusion process described by Equation (2) specifies a distribution corresponding to each timestep. First, let assume the dataset consists of single data point the
4

---

Published as a conference paper at ICLR 2024
origin. Denote the variance time as of dgg, and consider the d-dimensional mean
Gaussian distribution at each timestep .V(0, of) The posterior distribution over of given x, is:
| 20
x()) o p(xx(o)) p(o)) Nxxx;; 0, of) x of exp
This is an inverse Gamma distribution alo
(1 I(a) of
a+1
exp
(-
with parameter
values d/2 1 and b /xx112/2.
$333
20222
2012
0005
00075
250
11
158
U0p
203
50
(a) Analytical posterior p(o7) (r.)
(b) Non- Non-paramctric estimate
Figure 3: Posterior timestep distribution p(o) where x, produced using diffusion with dif- ferent time steps ... T}. averaged over the vertebral dataset. (a) shows the analytical distribution computed by placing Gaussian distributions of different variances at each point in the dataset, and (b) shows the inverse Gamma distribution with scale parameter value depending on the average distance to the k-nearest neighbours (I- 32).
If instead of single data point at the origin, have dataset D, with the corresponding data distribution p(x), we have
p(o x.) p(x.loip(o)) = Σ p(xx
Σ N((x; XiED
(4)
We refer to Equation (4) as the analytic estimator in subsequent sections since it is the exact poste- rior distribution. The posterior distribution can be interpreted adding the likelihoods Gaussian distributions centered around data points Xy E 'D with different (time-dependent)) variances. Substi- tuting the Gaussian density function and simplifying, we get
p(o)(x.) Eoj exp
||x=
xo 207
exp log
Σ X.
||x X 112
of
exp
20
X,ED
We can approximate the log sum-sum-exp term using max function:
p(o,lx.) 2 or exp
max X
X. xol13
= exp
(-17
min X
X,
X 11
(5)
20
The posterior over diffusion time approximately has the form of an inverse Gamma distribution with the shape parameter d/2 depending only on the dimensionality of the data and the seale parameter b ming,ED ||x, xo depending the distance of the input point to the closest point 2
the dataset. Note that, as a > = > 2, this analysis is only valid for three or higher dimensions.

---

Published as a conference paper at ICLR 2024
3.2 NON-PARAMETRIC MODEL
The posterior over diffusion time given by Equation (5) can potentially be used as a non-parametric approach anomaly detection. The approximation of log-sum-exp using the maximum value (near- est neighbour) becomes less accurate for larger timesteps, in which point has a comparable distance several points in the dataset. We found that instead of setting the scale parameter based on the distance to the closest point, approximating log-sum-exp using the average distance to k-nearest neighbours of the input point works better in practice. The non-parametric estimator is then:
1
X, Σ XoEKNN(x.)
-xoll
plo,1x,) 2 of exp
(6)
of
2
Figure 3 shows the analytical posterior distribution obtained using Equation and the non- parametric estimator given in Equation (6) for real dataset.
The upshot is that, given a point xs, this method approximates the scale parameter of the inverse Gamma distribution using the average distance to its k-nearest neighbours. The anomaly score is the mean of this distribution over diffusion time. As seen Figure 3 points X, that are produced using diffusion with larger time-steps also have higher posterior mean, on average, enabling to identify them as points that are far from the manifold. Interestingly, this method closely resembles the classical k-nearest neighbours (kNN). In fact, the anomaly rankings given by these methods are identical. In our experiments, the difference in score comes from the distance calculation: for DTE non-parametric, take the mean distance from the k-nearest neighbours opposed to (a variation of) KNN that takes the distance from the kth-nearest neighbour.
3.3 PARAMETRIC MODEL
The porammrmetr estimator of diffusion time becomes compute and memory when deal- ing with large datasets due to the need to find the k-nearest neighbours for each input sample in the entire datasct. To tackle the scalability problem, we employ deep neural networks to estimate the posterior distribution, which also enhances generalization capabilities. The full training procedure for both parametric models is available in Appendix D.2
Thmaaoo
Trianoo
Timares
(a) thyroid
(b) breastw
(c) vertebral
(d) shuttle
Figure 4: Predicted diffusion time against ground truth diffusion time for Gaussian model (E2- regression), Inverse Gamma model, and categorical model (with seven bins) on the test for various datasets. The maximum length of the diffusion Markov chain is T 300. The shaded region indicates the standard deviation in predictions across the dataset.
that the posterior distribution over time-dependent variance has the form of am inverse Gamma distribution. We train a deep neural network parameter- ized by which denote by So. predict the scale parameter b of the inverse Gamma distribution, given the noisy sample x,. Since the shape parameter a depends only on the dimensionality of the data, is known fixed parameter. We minimize the negative log log-likeldoodd given by:
Inverse Gamma model In Section 3.1
((0) = - Er log fe(x;) - (a + 1) log of fo(x:)/o?1
(7)
The expectation is over data samples Xn ~ p(x) and timesteps u(1 T]. The mode of the distribution is used as anomaly score.

---

Published as a conference paper at ICLR 2024
Figure 4 shows the predicted timestep for the inverse Gamma model applied to different datasets, with the length of Markov chain T 300. Compared to standard lg regression which assumes that the output variable is Gaussian distributed, the inverse Gamma model has much lower bias for diffusion time prediction for smaller timesteps, which empirically validates our analysis. However, this model suffers from high bias and high variance for larger timesteps. The high bias can be attributed to the approximation error of log-sum-exp using k-nearest neighbours, which becomes inaccurate for larger timesteps. The high variance is a consequence of the shape of the inverse Gamma distribution, which becomes flat for large values of the scale parameter (see Figure B).
Categorical model The inverse Gamma model while analytically accurate, can restrict the ex- pressivity of the neural network. In order to provide more flexibility in learning the diffusion time distribution, we can model it as a categorical distribution over T classes, where T is the length of the Markov chain associated with the diffusion process. This approach does not assume any parametric distribution over diffusion time and requires the model to accurately predict the full distribution. Let y/+ E {0, 1)" denote the one-hot vector with one at coordinate t, and fo denote the deep neural network that predicts the class probabilities, fo X' - 1]" We minimize the cross-entropy loss function, which is equivalent to maximizing the log- lielikkelihood of the categorical distribution:
K
(()) = Ei,wo
(M) Ve
log
)
In practice, we simplify the learning task by combining timesteps into bins and training model to predict the correct bin. If B denotes the number of bins, then the corresponding bin for a timestep t would be [P Figure shows the predicted timestep for the categorical model on different datasets. Compared to the inverse Gamma model, it suffers from significantly less bias across the entire range of timesteps. The score calculation is described in Appendix D.3 with the training algorithm in Appendix D.2
4 EXPERIMENTS
Setting We perform experiments on the ADBench benchmark (Han et al. 2022). which comprises set of popular tabular anomaly detection datasets as well as newly created tabular datasets made from images and natural language tasks, all described in Appendix D.1 The implementation details are provided in Appendix [ with the training algorithm, model architecture. hyperparameters, and comparison of the run-time. Some ablation studies are in Appendix A We implement and compare the results of the various approaches proposed in Section 3 the non-parametric, the parametric inverse Gamma, and the parametric categorical DTE.
D.1
Baselines We compare against all the unsupervised learning methods included in ADBench. These include classical methods, namely CBLOF (He et al. 2003), COPOD (Li et al. 2020), ECOD (Li
et al. 2022). FeatureBagging (Lazarevic & Kumar 2005).
HBOS (Goldstein & Dengel 2012).
FForest (Liu al.) 2008). kNN (Ramaswamy al. 2000), LODA (Pevnyl 2016). LOF (Breunig
et al. 2000), MCD (Fauconnier & Haesbroeck] 2009). OCSVM (Schölkopf et al. 1999), and PCA
(Shyu et al. 2003). The deep learning-based methods include DeepSVDD (Ruff al. 2018). and DAGMM (Zong et 2018). Outside of ADBench, we also compare against some more recently proposed deep learning-based approaches such as DROCC (Goyal et al. 2020), GOAD (Bergman & Hoshen, 2020), ICL (Shenkar & Wolf, 2022), SLAD (Xu et al.) 2023b) and DIF (Xu et al. 2023a): see Section 5 for brief overview. For each method, we picked the best pest-performing set of hyperparameters given in their original paper. We also have four additional generative baselines: normalizing flows with planar flows (Rezende & Mohamed] 2015) to identify anomalies based on the log-likelihood, DDPM, VAE (Kingma & Welling 2013) and GAN (Goodfellow et al. 2014 to reconstruct the input and compare it with the original input to identify anomalies.
Results Figure 5 shows the overall performance of these different methods 57 tasks in AD- Bench, each limited to 50,000 data points. The results for each individual dataset are provided in Appendix(F F. We report the mean AUC ROC and its standard deviation over five different seeds for each method. For the unsupervised setting, we used bootstrapping over the whole dataset for train- ing, while inference is made the full dataset. For the semi-supervised setting, we used 50% of
7

---

Published as a conference paper at ICLR 2024
DTE (Catagorical) (inverse
(Catagorical
DTE (Non Perametrc)
DTE (Non Parametac
DF
DC
GANomdl
GANooal
Ranerfiw
Hanarfiv
ICL
DROC:
DR.C
DAGVR
DAGMI
OCCMM
MCD
LoD4
LOD4
ffoeed
Ioreed
Featurelagging
featurellagging
ECoD
ECOO
CSLDF
CBLDF
(a) Semi-supervised
(b) Unsupervised
Figure 5: AUC ROC means and standard deviations on the 57 datasets from ADBench over five different seeds for a) the semi-supervised setting using normal samples only for training and b) the unsupervised setting with bootstrapped training instances. Colour scheme: red green (deep learning methods), blue (classical methods). DTE outperforms all baselines for the semi-superrvied setting apart from kNN. It is also competitive in the unsupervised setting.
the normal samples for training, while the test set contains the rest of the normal samples and all anomalous samples. The proposed method is among the few competitive in both semi-supervised and unsupervised settings. particular, our method outperforms all previous deep learning- bbased approaches in both settings significantly and also outperforms the DDPM model. Unsurprisingly, deep learning methods have a higher variance than non-parametric methods. Using bagging can be way to help reduce the variance the cost more training and inference time.
Figure ] compares our method's performance and inference time with the other baselines. In some applications, such medical and network monitoring, fast inference time crucial as the algorithm must detect anomalies in real time. Our method uses forward pass through simple neural network for predictions, which gives the shortest inference time over all the methods considered here. Training time, inference time and compute amounts are available in Appendix D.4
Choice of representation ADBench's image datasets use vector representation derived from pre- trained ImageNet embeddings. We investigated the impact of representation quality for semi- supervised anomaly detection across several datasets: VisA (Zou et al., 2022). CIFAR-10, and MNIST, We observe that different methods, including DDPM, kNN and DTTE, perform better when applied to image embeddings rather than raw images. In particular, embeddings produced through self-supervision are generally of higher quality when compared to those produced for elassification, and the embeddings that are specialized or fined-tuned to the target dataset produce the best results. The results are reported in Appendix
We also observe that KNN remains a pop--eeffrming algorithm for anomaly detection, where its only disadvantage remains its scalability. As explained Section 3.2 the non-parametri method gives the same anomaly ranking as kNN. DTE can thus be approximately interpreted as a parametric k- nearest neighbours algorithm which can beneficial for large datasets that require smaller inference time. To understand the anomalies, both DDPM and DTE are able to identify "denoised" data point; DDPM depends on an initial time step hyper-parameter, whereas DTE does not, by using deterministic ODE flow. However, DDPM outperforms in denoising, being explicitly trained for it. Further interpretability discussion, illustrated with a toy example, in Appendix B
RELATED WORK
We refer the reader the following surveys for comprehensive review (Pang et al. 2021 Chandola jet al.; |2099 Ruff et al.) 2021 Hodge Austin 2004). Although recently the spotlight has shifted towards deep learning methodologies, classical techniques such as kNN (Ramaswamy et al. 2000)
8

---

Published as a conference paper at ICLR 2024
persistently exhibit strong performance. We compared our method with some of these techniques in Section Clustering and nearest neighbour algorithms use the distance to score instances, making them eastly interpretable. Clustering algorithms, such CBLOF (He et all, 2003) and k-means [MacQueen 1967). assume that anomalies are either not part of cluster, are part of smaller clusters than normal instances, or lie further away from the cluster centroid. In contrast, nearest neighbour algorithms use the distance between points or relative density with respect to their neighbourhood.
As anomalies can be more difficult detect in high-dimensional spaces and complex data distribu- tions (Pang et al. 202T), the development of deep anomaly detection algorithms has been increasing over the past few years (Ruff et al., 202I). In particular, several works combine autoencoders with other classical techniques (Zhou & Paffenroth
2017: Kim et al.) [220]] An & Cho 2015 Erfani
VDD jet af.] 2016; Sakurada & Yairi] 2014] Xia et al. 2015). Other notable methods include DeDDSVDD
(Ruff et al. 2018), DAGMM (Zong et al.] 2018);
1); Lunar (Goodge et al. 2022). DROCC (Goyyl et al.,
2020), GOAD (Bergman & Hoshen, 2020p, SO-GAAL and MO-GAAL (Liu et al. 2019), SLAD
(Xu et al. 2023b) and DIF (Xu et al. 2023a). Deep kNN methods (Pang al. 2018 Sun et al.,
2022 learn representations to apply KNN. ICL (Shenkar & Wolf 2022), which uses contrastive rep-
resentation learning reported competitive results foD datasets, for the semi-supervised setting.
Diffusion-based Techniques While diffusion models have been previously used for anomaly de- tection in image and video (Yan et al. 2023: Flaborea al. 2023) Tur et al., 2023) data for a one- class setting (semi-supervised), their application in the context of tabular data and the unsupervised setting was unexplored. Wolleb al. (2022) proposed an encoding method using a diffusion process followed by denoising procedure guided by classifier. Zhang et al. (2023a) synthesizes anomaly samples to train the denoising network for anomaly repair. AnoDDPM employs a specific diffusion noise to train a denoising network for normal image reconstruction (Wyatt et al. (022). Similarly, Graham et al. (2023) utilized a DDPM to reconstructs an image for multiple different timesteps combined together to make anomaly scores. [iu et al. (2023) introduced diffusion method that reconstruct an image by in in-painting the input masked by a checkerboard pattern. Lastly, Zhhng et al. (023b) used a latent diffusion model trained with simulated anomalous samples images.
6 CONCLUSION
This paper investigates the applicability of diffusion modelling for unsupervised and semi- supervised anomaly detection. We observe that specific design choices in DDPMs, although some- what arbitrary, significantly influence their performance. Despite the expressivity and interpretabil- ity of DDPMs, they come with notable computational overhead compared to existing parametric techniques. For anomaly detection, DDPM essentially estimates the distance between the input and its "denoised reconstruction", we observe that one could directly produce this estimate, or equiv- alently estimate the diffusion time. We first observe that the distribution of diffusion time given noisy input, follows an inverse Gamma distribution. This forms the basis for our non-parametric approach that accurately predicts the diffusion time and turns out to create the same anomaly score ranking as kNN. A subsequent parametric strategy leverages deep neural network, hamnsssig its generalization and rapid inference capabilities for large datasets. We evaluate the effectiveness of DTE on ADBench, a benchmark comprising popular anomaly detection datasets. Our results demonstrate competitive performance compared to prior work while improving the inference time by several orders of magnitude. Furthermore, we find that using pre-trained embeddings for images considerably improves the performance of diffusion- feffusioo-bbsedd methods, showing the potential advan- tage of using latent space diffusion.
7 LIMITATIONS AND FUTURE WORK
While our approach, DTE, achieves excellent performance with low inference time, is important to acknowledge that in terms of interpretability, DTE falls behind DDPM as we explain in Appendix B and Section 4 This may pose challenges for practitioners seeking understand the underlying mechanisms and behaviours of the data. Evaluating DTE in handling larger and more complex real-world datasets remains an avenue for future exploration. While here, only address point anomalies, applications of diffusion modelling for group and contextual anomalies remain a high- impact unexplored area that we plan to investigate in the future.

---

Published as a conference paper at ICLR 2024
8 REPRODUCIBILITY STATEMENT
We have made efforts to ensure that our method is reproducible. Appendix D.1 scription of all datasets included in ADBench, along with the preprocessing steps. Appendix D.2 presents formal algorithm for parametric DTE and Appendix [ providds detailed description of the network architecture and hyperparameters. We provide full results for both the unsupervised and semi-supervisee settings with additional metrics, for all individual datasets and baselines in Ap- pendix [ as a reference for researchers to reproduce our experimental results. We are releasing the code as part of the supplemental material with detailed explanations to ran the experiments.
provides a de-
ACKNOWLEDGEMENTS
We want thank Mehran Shakerinava for his input in the early stages of this project and Katelin Schutz for helpful discussion. The NSERC NFRF program and CIFAR AI Chairs partly support this research. Mila and the Digital Research Alliance of Canada provide computational resources.
REFERENCES
Mohiuddin Ahmed, Abdun Naser Mahmood, and Jiankun Hu. survey of network anomaly detec- tion techniques. Journal of Network and Computer Applications, 60:19-31, 2016a.
Mohiuddin Ahmed, Abdun Naser Mahmood, and Md Rafiqul Islam. A survey of anomaly detection techniques in financial domain. Future Generation Computer Systems, 55:278-288, 2016b.
Jinwon An and Sungzoon Cho. Variational autoencoder based anomaly detection using reconstruc- tion probability. 2015.
Lion Bergman and Yedid Hoshen.
Classification-batt anomaly detection for general data. In International Conference on Learning Representations, 2020. /R///ttpss://perrevieew
net fort/Eorum3 ieen?iddei 1d=H11K 1BtVS
Markus M. Breunig, Hans-Peter Kriegel, Raymond T. Ng, and Jörg Sander. Lof: Identifying density- based local outliers. In Proceedings of the 2000 ACM SIGMOD International Conference on Management of Data, SIGMOD '00, pp. 93 1041 New York, NY, USA, 2000. Association for Computing Machinery. ISBN 1521132174, doi: 10.1145/3420993333888
URL https://
doi. ooi.orr//0..11 111511..1114/24200 R000442000033338888
Varun Chandola, Arindam Banerjee, and Vipin Kumar. Anomaly detection: A survey, ACM Comput. Swrv, 41(3), jul 2009. ISSN 0360-0300. doi: 10.1145/1541880.15418882
orq/ 100//00.200 11t.111111111111111
1541882
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In North American Chapter of the Associ- ation,fo for Computational Linguistics, httpa////oo/////////o.ooooooaoor BtRphttps://
CorpusID::29677 ID:
Sarah M. Erfani, Sutharshan Rajasegarar, Shanika Karunasekera, and Christopher Leckie. High- dimensional and large-scale anomaly detection using a linear one-class svm with deep leamn ing. Pattern Recognition, 58:121 134, 2016. ISSN 0031-3203. doi:
028. URL https: www. www.sceeiiicccccccc.. rect com/ haamm//cieenne/aticlle/ 0/0/00003132001630000
C. Fauconnier and Gentiane Haesbroeck. Outliers detection with the minimum covariance determi- nant estimator in practice. Statistical Methodology, 6:363-379, 07 2009. doi: 10.1016/j.stamet.. 2008.12.005.
Alessandro Flaborea, Luca Collorone, Guido Maria D' Amely di Melendugno, Stefano D' D'Ariioo, Bardh Prenkaj, and Fabio Galasso. Multimodal motion conditioned diffusion model for skeleton- based video anomaly detection. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) pp. 10318-10329, October 2023.
10

---

Published as a conference paper at ICLR 2024
Katherine Fraser, Samuel Homiller, Rashmish K. Mishra, Bryan Ostdiek, and Matthew D. Schwartz. Challenges for unsupervised anomaly detection in particle physics. Journal of High Energy Physics. 2022(3). mar 2022. doi: 1011007/hhc003322220666
| httph: /tttps://do.... /ot..///..........
(0073222hhee0333222222222220
Markus Goldstein and Andreas Dengel. Histogram- -aaeaad-bbaedd-- outlier score (hbos): A fast unsupervised anomaly detection algorithm. 09 2012.
Ian J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks, 2014.
Adam Goodge, Bryan Hooi, See Kiong Ng. and Wee Siong Ng. Lunar: Unifying local outlier detection methods via graph neural networks. 2022.
Yury Gorishniy, Ivan Rubachev, Valentin Khrulkov, and Artem Babenko. Revisiting deep learning models for tabular data. In Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, 2021. httpp:://pppeeeeeeeeete
net/forum?id- 111yr0egly
Sachin Goyal, Aditi Raghunathan, Moksh Jain, Harsha Vardhan Simhadri, and Prateek Jain. DROCC: Deep robust one-class classification. In Hal Daumé III and Aarti Singh (eds.), Pro- ceedings of the 37th International Conference on Machine Learning, volume 119 of Proceed- ings of Machine Learning Research, pp. 371 3711-1 PMLR, 13-18 Jul 2020. URL https: //proceedings.o iangggsmml. mlr. wtt../ppeaas////yellllleea. haml
Mark Graham, Walter H.I., Pinaya, Petru-Daniel Tudosiu, Parashkev Nachev, Sebastien Ourselin, and Jorge Cardoso. Denoising diffusion models for -iittriiitiiitoonn detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, pp. 2947-2956, June 2023.
Songqiao Han, Xiyang Hu, Hailiang Huang, Minqi Jiang, and Yue Zhao. ADBench: Anomaly detection benchmark. In Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2022. URL https: httt:::://penreeiew.netttt http:/ewww.e..eeeoeeeeee wwwmmfof///oiddd.. EoA SFQ9zo0]
Douglas Hawkins. Identification of outliers, volume 11. Springer, 1980.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition, 2015. URL htt.p::://axii..rg///////// 113.//33111300555
Zengyou He, Xiaofei Xu, and Shengehun Deng. Discovering cluster-based local outliers. Pattern Recogn. Lett. 2409-10):161116655 jun 2003. ISSN 0167-8655 doi: 1011016//0167-8655000 00003-5. httlht org/ 101111110011111655555555 (C3) 0000 00003-5
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 33:6840 6851, 2020.
Victoria Hodge and Jim Austin. survey of outlier detection methodologies. Artificial Intelligence Review, 22:85-126, 10 2004. doi: 10. 10.1122...A1RE..0000555212......
Ki Hyun Kim, Sangwoo Shim, Yongsub Lim, Jongseob Jeon, Jeongwoo Choi, Byungchan Kim, and Andre S. Yoon. Rapp: Novelty detection with reconstruction along projection pathway. In ICLR. OpenReview.ned. 2020. URL
1clr2020 ht1.00..mm+4440
2013. www.stttttiiimmmmmc....
Diederik P. Kingma and Max Welling. Auto-encoding variational bayes. CoRR, abs/1312.6114 http:://ggp.....aaa...cccoooootr
Akim Kotelnikov, Dmitry Baranchuk, Ivan Rubachev, and Artem Babenko. Tabddpm: Modelling tabular data with diffusion models, 2022. _ htt.:/ttt////axiii wttpabg//absa.//// 15421
15421

---

Published as a conference paper at ICLR 2024
Aleksandar Lazarevic and Vipin Kumar. Feature bagging for outlier detection. In Proceedings of the Eleventh ACM SIGKDD International Conference on Knowledge Discovery in Data Min- ing, KDD '05, pp. 157-166, New York, NY. USA, 2005, Association for Computing Machinery. ISBN 15959315XX. doi: 10. 00.1145/108188 187018800.100 10110088177001111111 108180181199 1891. URL htttps://don 10..:/i.or//11111//1000
1081870 10200...8111 1081891
Zheng Li, Yue Zhao, Nicola Botta, Cezar Ionescu, and Xiyang Hu. COPOD: Copula-based outlier detection. In 2020 IEEE International Conference on Data Mining (ICDM). IEEE, nov 2020. doi: 10. 01119/cddm50008220000000001 2F Fdicm00118 2020. 00135
URL https: otreii.or///0//
Zheng Li, Yue Zhao, Xiyang Hu, Nicola Botta, Cezar Ionescu, and George Chen. ECOD: Unsu- pervised outlier detection using empirical cumulative distribution functions. IEEE Transactions on Knowledge and Data Engineering, pp. 1-1, 2022. doi: 10.1 (0.109992e2222231555580 URL
https: //o.i .rgg 10. 10982Ft 2022 .000022..111999 3159580
Fei Tony Liu, Kai Ming Ting, and Zhi-Hua Zhou. Isolation forest. In 2008 Eighth IEEE Interna- tional Conference on Data Mining, pp. 413-422, 2008. doi: 10. 109/000DM
Yezheng Liu, Zhe Li, Chong Zhou, Yuanchun Jiang, Jianshan Sun, Meng Wang, and Xiangnan He. Generative adversarial active learning for unsupervised outlier detection, 2019.
Zhenzhen Liu, Jinjie Zhou, Yufan Wang, and Kilian Q. Weinberger. Unsupervised out-of- distribution detection with diffusion inpainting. International Conference on Machine Learn- orq wwt//Corpss1::25500000000
ing. 2023. URL http::tts:///a http://ap...maaa.cccccccoooooooe
J. B. MacQueen. Some methods for classification and analysis of multivariate observations. In L. M. Le Cam and J. Neyman (eds.), Proc. of the fifth Berkeley Symposium on Mathematical Statistics and Probability, volume 1, pp. 281 287- University of California Press, 1967.
Girik Pachauri and Sandeep Sharma. Anomaly detection medical wireless sensor networks using machine learning algorithms. Procedia Computer Science, 70:325-331 2015. ISSN 1877-0509. doi: https://doi.org/ 10. pttp:///...oc..2......... 10.026. URL https://wwws wwww sciencedi Proceedings of the 4th Interna- tional Conference on Eco -re-ffiiedly Computing and Communication Systems.
com/ scwssccceeeeeeeeeeeeeoee http:///...ooeee.....e/////////0
Guansong Pang, Longbing Cao, Ling Chen, and Huan Liu. Learning representations of ultrahigh- dimensional data for random distance-based outlier detection. In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, KDD '18, pp. 2041-2055 New York, NY, USA. 2018. Association for Computing Machinery. ISBN 9781450355520. doi: 10.1145/32198199322002.
URL Attps: 2tte:///c...rrg...1110.
321 5219819.3220042
Guansong Pang, Chunhua Shen, Longbing Cao, and Anton Van Den Hengel. Deep learning for anomaly detection. ACM Computing Surveys, 54(2):1-38, mar 2021. doi: 10. 145/343999. URL
Tomáš Pevný. Loda: Lightweight on-line detector of anomalies. Mach. Learn., 102(2)275-304, feb 2016. ISSN 0885-6125. doi: 10. 11110077119944--555555-144 URL httph:ttt//// /tattss://do..... 10. 1007
Sridhar Ramaswamy, Rajeev Rastogi, and Kyuseok Shim. Efficient algorithms for mining outliers from large data sets. SIGMOD Rec, 29(2):427 438, may 2000. ISSN 0163-5808. doi: 10. 145/ 335191.335437. URL/sttpps///bl..rrr
11111111//311111 335437
Danilo Rezende and Shakir Mohamed. Variational inference with normalizing flows. In Francis Bach and David Blei (eds.), Proceedings of the 32nd International Conference on Machine Learn- ing, volume 37 of Proceedings of Machine Learning Research, pp. 1530-1538, Lille, France, 07-09 Jul 2015. PMLR. URL httptttt////rrcceeiiig mlr wwwppppwwssspp.ppppppp.ppppp.pea /00peeeee/////eeeedee1.. html
12

---

Published as a conference paper at ICLR 2024
Lukas Ruff, Robert Vandermeulen, Nico Goernitz, Lucas Deecke, Shoaib Ahmed Siddiqui, Alexan- der Binder, Emmanuel Müller, and Marius Kloft. Deep one-class classification. In Jennifer Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference Machine Learn- ing. volume 80 of Proceedings of Machine Learning Research, pp. 2018. URL htt.:tttp:pppooeeeeeeeoo ngs. ml wwwppp/ww// vttf//0f0/fffffu 8a html
PMLR, 10-15 Jul
Lukas Ruff, Jacob R. Kauffmann, Robert A. Vandermeulen, Gregoire Montavon, Wojeiech Samek, Marius Kloft, Thomas G. Dietterich, and shallow anomaly detection. Proceedings of the IEEE, 109(5)755-795, may 2021. doi: 10. 1 109/ jproc. 2021 3052449
Muller. A unifying review of deep and
20roc.0221.3022444 URL wwR.hhtts:/// /tt.d/ii.ooogooooooooooo btttps::/dei..og//l.....
Mayu Sakurada and Takehisa Yairi. Anomaly detection using autoencoders with nonlinear dimen- sionality reduction. In Proceedings of the MLSDA 2014 2nd Workshop on Machine Learn- ing for Sensory Data Analysis, MLSDA' 14, pp. 4-1, New York, NY, USA, 2014. Associa- tion for Computing Machinery. ISBN 9781450331593. doi: 00.114572689744626697777 URL
ht. httpps://at www.:ioorg/....0 10w.1115//550 68974899444 2 2689747
Osman Salem, Alexey Guerassimov, Ahmed Mehaoua, Anthony Marcus, and Borko Furht. Sensor fault and patient anomaly detection and classification in medical wireless sensor networks. In 2013 IEEE International Conference on Communications (ICC), pp. 4373-4378, 2013. doi: 10. 1109/CCC:0013.6555255
Bernhard Schölkopf, Robert Williamson, Alex Smola, John Taylor, and John Platt. Support vector method for novelty detection. volume 12, pp. 582-588, 01 1999.
Tom Shenkar and Lior Wolf. Anomaly detection for tabular data with internal contrastive learning. In International Conference Learning Representations, 2022. URL
net/ www.sssshbtbtt 46bT
Mei-Ling Shyu, Shu-Ching Chen, Kanoksri Sarinnapakom, and Liwu Chang. A novel anomaly detection scheme based on principal component classifier. 01 2003.
Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International Conference on Machine Learn- ing. pp. 2256-2265. PMLR, 2015.
Yiyou Sun, Yifci Ming, Xiaojin Zhu, and Yixuan Li. Out-of-distribution detection with deep nearest neighbors. ICML., 2022.
Gian Antonio Susto, Matteo Terzi, and Alessandro Beghi. Anomaly detection approaches for semi- conductor manufacturing. Procedia Manufacturing, 11:2018-2024 2017.
Anil Osman Tur, Nicola Dall' Asen, Cigdem Beyan, and Elisa Ricci. Exploring diffusion models for unsupervised video anomaly detection. 2023 IEEE International Conference on Image Processing (ICIP). pp. 2540-2544, 2023. URL URthttss://app: remanticscholiee www.arr....ooooeeeeeeeoeeeeeeooo orgg//rpu//rps: 258079336
Julia Wolleb, Florentin Bieder, Robin Sandkühler, and Philippe C Cattin. Diffusion models for medical anomaly detection. In Medical Image Computing and Computer Assisted Intervention- MICCAI 2022: 25th International Conference, Singapore, September 18-22, 2022, Proceedings, Part VIII, pp. 35-45. Springer, 2022.
Julian Wyatt, Adam Leach, Sebastian M. Schmon, and Chris G. Willcocks. Anoddpm: Anomaly detection with denoising diffusion probabilistic models using simplex noise. In 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pp. 649-655, 2022. doi: 10. 1009//VVR256347.2222220000
Yan Xia, Xudong Cao, Fang Wen, Gang Hua, and Jian Sun. Learning discriminative reconstruc- tions for unsupervised outlier removal. 2015 IEEE International Conference on Computer Vision (ICCV), pp. 1511-1519, 2015.
Hongzuo Xu, Guansong Pang, Yijie Wang, and Yongjun Wang. Deep isolation forest for anomaly detection. IEEE Transactions on Knowledge and Data Engineering, pp. 1-14, 2023a. doi: 10. 1109/TKDDH.02233370222
13

---

Published as a conference paper at ICLR 2024
Hongzuo Xu, Yijie Wang, Juhui Wei, Songlei Jian, Yizhou Li, and Ning Liu. Fascinating supervisory signals and where to find them: Deep anomaly detection with scale learning. In Proceedings of the 40th International Conference Machine Learning, ICML'23, JMLR.org, 2023b.
T. Yairi, Y. Kawahara, R. Fujimaki, Y. Sato, and K. Machida. Telemetry-mining: machine learning approach to anomaly detection and fault diagnosis for space systems. In 2nd IEEE International Conference on Space Mission Challenges for Information Technology (SMC-IT'06), pp. 8 pp-
476. 2006. doi: 10.1 (09//MC-I..2006..77
Cheng Yan, Shiyu Zhang, Yang Liu, Guansong Pang, and Wenjun Wang. Feature prediction diffu- sion model for video anomaly detection. Proceedings of the IEEE/CVE International Confer- on Computer Vision (ICCV), pp. 5527-5537, October 2023.
Hui Zhang, Zheng Wang, Zuxuan Wu, and Yu-Gang Jiang. Diffusionad: Denoising diffusion for anomaly detection, 2023a.
Xinyi Zhang, Naiqi Li, Jiawei Li, Tao Dai, Yong Jiang, and Shu-Tao Xia. Unsupervised surface anomaly detection with diffusion probabilistic model. In Proceedings of the IEEE/CVF Interna- tional Conference on Computer Vision (ICCV), pp. 6782-6791, October 2023b.
Chong Zhou and Randy C. Paffenroth. Anomaly detection with robust deep autoencoders. In Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD 17, pp. 665-674, New York, NY, USA, 2017. Association for Computing doi: 10.1145/399993330988322 URL https:://oo.
Machinery. ISBN 978 9781450348874
org/ 10 00...1145/39999
3098052
Bo Zong, Qi Song, Martin Renqiang Min, Wei Cheng, Cristian Lumezanu, Dac ki Cho, and Haifeng Chen. Deep autoencoding gaussian mixture model for unsupervised anomaly detection. In Inter- national Conference on Learning Representations, 2018.
Yang Zou, Jongheon Jeong, Latha Pemula, Dongqing Zhang, and Onkar Dabeer. Spot-the-differenc self-supervised pre-training for anomaly detection and segmentation. In European Conference Computer Vision, pp. 392 992-408. Springer, 2022.
14

---

Published as a conference paper at ICLR 2024
A
ABLATION STUDIES
We perform several ablation studies understand DDPM and the proposed DTE method.
Unsapenicce
0.83
0.75
000/000 AUC
0.63
0.60
0.68
630
100
10
20
Figure 6: Average AUC ROC over the 57 ADBench datasets for different reconstruction timesteps of the DDPM model.
Figure 7: Average AUC ROC over the 57 AD- Bench datasets for different number of bins of the DTE categorical model.
Reconstruction timestep in DDPM When using DDPMs for anomaly detection based the reconstruction distance, the denoising model requires an input timestep to create the reconstruction. We found that this somewhat arbitrary hyperparameter choice can significantly affect performance as shown in Figure 6
For the unsupervised setting, found that value close 50% of the maximum timestep results in the highest AUC ROC score on average. For the semi-supervised. the AUC ROC decreases as increase the reconstruction timestep. Since the model trained only on normal samples, the anomalies are sufficiently distanced from the learned data manifold for minor changes to result in large reconstruction error while larger timestep decreases the precision on normal samples.
Number of bins in categorical DTE by combining multiple timesteps into bins. This turns out to be important hyperparameter it affects the final performance significantly. Figure 7 shows that a low number of bins leads to better performance. This be attributed the fact that we calculate the mean of the predicted timestep distribution rather than the mode to calculate anomaly scores and that adding more bins increases the complexity of the learning task.
As discussed in Section 3.3
we implement categorical DTE
Maximum timestep DTE We study the effect of changing the maximum timestep in the noising diffusion process. As seen in Figure g the maximum timestep affects performance un- til roughly T = 250, since for very low val- of the noisy samples might not resemble standard Gaussian noise and might not cover all potential anomalies in the dataset. We also note categorical DTE is more robust to the value of than the inverse Gamma DTE.
Standerd dediaton
0.5
120
250
350
530
750
Figure 9 shows the value of standard devia- tion versus timestep as we increase the maxi- mum timestep T'. We observe that for values of 2 250, the final timestep corresponds to standard deviation close enough to 1.0 that the data resembles samples drawn from a standard Gaussian distribution.
1500
1505
0.9
2003
1009
Figure 9: Standard deviations versus timestep for different values the maximum timestep
15

---

Published as a conference paper at ICLR 2024
0.5
0.63
Nerrge AMC RCC
ANC ROC
2.63
0.88
0.58
0.45
0.43
Unsupervbed
Unoupervbed
Semi-supendses
0.3
supervised
250
753
1050
1250
1500
1750
2000
250
530
750
1030
1253
1500
1750
2000
Maximum Emmssio
Maximum Eirestze
(a) DTE categorical
(b) DTE inverse gamma
Figure 8: Average AUC ROC over the 57 ADBench datasets for different maximum timestep T for the categorical and inverse gamma DTE models on both semi-supervised and unsupervised settings.
B INTERPRETABILITY
In certain applications, the mere identification of anomalies in the dataset is insufficient; it imperative to understand the underlying rea- sons for flagging specific data points anoma- lies. Both DDPM and DTE can provide inter- pretability by identifying corresponding "de- noised" or normal data point. In DDPM this achieved using the deterministic ODE flow, which is (rather arbitrarily) initialized at some large time step. We found the initial time step to be an important hyper-paramen parameter, which im- pacts both anomaly detection and interpretabil- ity for DDPM. In practice, 7" = .25 x T per- forms well as the initial time-step. DTE has the benefit of avoiding such hyper-parameters, where one could use the gradient flow associ- ated with the mode of the posterior to denoise given input; see Figure (d, and f).
9
9
9
9
9
9
(a) Original
(b) Denoised (c) Difference (d) Gradient
Figure 10: Interpretability in DTE (first row) and
DDPM (second row) on MNIST. Visual interpre- tation of a gray patch anomaly on an MNIST im- age using the categorical diffusion model with simple convolution network on the first row and DDPM on the second for comparison. a) original anomalous image, b) the denoised version using gradient descent c) difference between the origi- nal and the denoised image, d) visualization of the gradient on top of the original image.
Figure [igure[O]showe shows another example, this time us- ing the categorical likelihood on the MNIST dataset. We artificially introduce gray patch as an anomaly (Figure IO (a)) and perform the gradient descent procedure reducing the mean of the posterior density. We observe that this procedure indeed partially eliminates the patch (Fig- ure (b)). We also note that since it is explicitly trained to remove the noise from noisy input, DDPM performs better in removing the patch.
As detailed in Section 3.2 the non-parametric DTE yields the same anomaly score as kNN. Thus, the parametric DTE can be viewed as approximate parametric kNN algorithm. This perspective enhances DTE's interpretabilitty: the neural network's 's score represents the estimated distance of point to the manifold. Although can't 't pinpoint which training set instance most closely matches an input, interpreting the score as a distance to a certain neighbourhood offers a straightforward insight into the method's functioning.
16

---

Published as a conference paper at ICLR 2024
C NON-PARAMETRIC ESTIMATION OF TIMESTEP DISTRIBUTION
In Figure 3. we visualize the analytical posterior distribution along with the non- non -on-ammmtttri esti- mate. The difference between these distributions is shown in Figure 11 While the two distributions are quite similar, their shape is very peaked for low values of the diffusion timestep. The slight misalignment between peaks of the analytical and the non-parametric estimate gives rise to the spiky shape seen in the difference. For higher values of the diffusion timestep, the difference is very close to zero, demonstrating that the non-part parapaamttricc estimate based on k-nearest neighbours is a very close approximation to the true posterior distribution of timestep.
(a) Analytical posterior p(o
(b) Non-parametric estimate
(c) Difference
Posterior timestep distribution p(a xx). where x, is produced using diffusion with different time steps E (1,...; 7), averaged over the vertebral dataset. (a) shows the analytical distribution computed by placing Gaussian distributions of different variances at each point in the dataset, (b) shows the inverse Gamma distribution with scale parameter value depending the average distance to the k-nearest neighbours (k = 32), and (c) shows the difference between (a) and (b).
Figure 11:
D IMPLEMENTATION DETAILS
D.1 DATASETS AND PREPROCESSING
Datasets description We show the results from our methods and baselines over multiple datasets from ADBench (Han et al. 2022) described Table There are 47 tabular datasets ranging from multiple different applications. There are also five datasets composed of extracted representations of images after the last average polling layer from Resnet-18 (He et al.] 2015) model pre-trained on ImageNet. Similarly, there are five datasets composed of extracted embedding of NL. tasks from BERT (Devlin al. 2019). We also show results on VisA (Zou et al.) 2022), which is dataset composed of images of 12 different objects where the anomalies are various flaws the objects.
Training and test data configuration For ADBench, the semi-supervised setting, we use half of the normal data in the training set, and the other half is the test set with all the anomalies. For the unsupervised setting, we sample the whole dataset with replacement for the training data, while the test data is the whole dataset. This bootstrapping method allows us to test the variance over the training dataset for each method.
Preprocessing We standardize the input samples based on the mean and standard deviation calcu- lated over the training data, to ensure consistency across the input values and mitigate the impact of potential outliers or scale variations. For VisA, 90% of the normal instances are making the training data, while the anomalies and the remaining 10% are in the test set. For CIFAR-IO and MNIST, One class is set the anomaly while the others are part of the training data. 80% of the normal instances are the training data while the remaining 20% and the anomalies are the test data. For ADBench, CIFAR-10 10, MNIST-C, SVHN, and FashionMNIST are made up of one class for the normal sample, while the anomalies are the rest of the classes downsampled to make up 5% of the total data.
17

---

Published as a conference paper at ICLR 2024
On the importance of standardization for diffusion models vestigations, we discovered the critical importance of standardization. This is due to the fact that the incorporated Gaussian noise operates under the assumption that each feature is centered zero with unit standard deviation. Consequently, implementing standard scaling facilitates the comprehensive
Throughout the course of our in-
Table 1: Description of all datasets in ADBench
Dataset Name
Samples
# Features
# Anomaly
% Anomaly
Category
ALOI
49534
27
1508
3.04
Image
annthyroid
7200
6
534
7.42
Healthcare
backdoor
95329
196
2329
2.44
Network
breastw
683
9
239
34.99
Healthcare
campaign
41188
62
4640
11.27
Finance
cardio
1831
176
9.61
Healthcare
Cardiotocography
2114
466
22.04
Healthcare
celeba
202599
4547
2.24
Image
census
299285
18568
6.20
Suciology
cover
2861448
10
2747
0.96
Botany
donors
619326
10
36710
5.93
Snciology
fault
1941
27
673
34.67
Physical
fraud
284807
29
492
0.17
Finance
glass
214
7
9
4.21
Forensic
Hepatitis
80
19
16,25
Healthcare
http
567498
3
2211
0.39
Web
InternetAds
1966
1555
368
18.72
Image
lonosphere
351
32
126
35,90
Oryctognosy
landsat
6435
36
1333
20.71
Astronautics
letter
1600)
32
100
6.25
Image
Lymphography
148
18
6
4.05
Hcalthcare
magic.gama gamma
19020
10
6688
35.16
Physical
mammography
11183
6
260
2.32
Healthcare
mnist
7603
100
700
9.21
Image
musk
3062
166
97
3.17
Chemistry
optdigits
5216
64
150
2.88
Image
PageBlocks
5393
10
510
9.46
Document
pendigits
6870
16
156
2.27
Image
Pima
768
8
268
34,90
Healthcare
satellite
6435
36
2036
31.64
Astronauties
satimage-2
5803
36
71
1.22
Astronauties
shuttle
49097
9
3511
7.15
Astronautics
skin
245057
3
50859
20.75
Image
smip
95156
3
0.03
Web
SpamBase
4207
57
1679
39,91
Ducument
speech
3686
400
1.65
Linguistics
Stamps
340
9.12
Document
thyroid
3772
2.47
Healthcare
vertebrul
12.50
Biology
vowels
12
3.43
Linguistics
Waveform
3443
100
2.90
Physics
WBC
9
4.48
Healthcare
WDBC
367
30
2.72
Healthcare
Wilt
4819
5
257
5.33
Botany
wine
129
13
7.75
Chemistry
WPBC
198
33
23.74
Healthcare
yeast
1484
8
507
34.16
Biology
CIFARIO
5263
512
263
5.00
Image
FashionMNIST
6315
512
315
5.00
Image
MNIST-C
10000
512
500
5.00
Image
MVTec-AD
5354
512
1258
23,50
Image
SVHN
5208
512
260
5,00
Image
Agnews
10000
768
500
5,00
NLP
Amazon
10000
768
500
5.00
NLP
Imdb
10000
768
500
5.00
NLP
Yelp
10000
768
500
5.00
NLP
20newsgroups
11905
768
591
4.96
NLP
18

---

Published as a conference paper at ICLR 2024
coverage of the anomaly detection space by the noise. This proved to be essential component of the proposed anomaly detection method.
D.2 ALGORITIM
Algorithm Training Process for parametric DTE
Parameters: T maximum timestep, À learning rate
Input: Training data D
1: 0- 0o
Initialize weights of the model Define the schedule for forward diffusion
2: Bo, f1..... 3T-1 t linear 0.01)
3: for all < T do
4: G4
II.-1 3.)
Compute the Set standard deviation for each timestep
5: 0t vI ōt
6: end for
7:
for num num_epochs do
8:
for all in D do
9:
t~ ((, T 1) (0,1)
>> Sample timestep t uniformly Sample standard Gaussian noise >> Compute noisy sample of timestep t Equation (8) for inverse Gamma Equation (9) for categorical
10:
11:
X; + Xn + o+
12:
L loss fx())))
13:
- 8 gV
b Update model parameters
end for
15:
end for
D.3 MODEL ARCHITECTURE AND HYPERPARAMETER
We first found the hyperparameters using different training splits for the semi sume-super setting on the shuttle and thyroid datasets (network architecture, maximum timestep, batch size, number of epochs). We then tuned some of them over all the datasets using different training seeds than the ones used for the final results (number of bins and learning rate). This is the case for the diffusion methods and the normalizing flow method. For the other baselines, picked the set of hy hyperparameters from the original papers that provided the best results over the whole benchmark.
For the non- nor-maraectric DTE, the score is calculated based the approximate posterior dis- tribution in Equation (6) with for the semi-supervised setting and k = 32 for the unsupervised setting. The anomaly score is the mean of the posterior avoid having anomaly score that is restricted by the maximum variance using the mode. The be consistent, we selected the same & for the KNN baseline.
DTE
For the DTE parametric approach, we employ a multi-layer perceptron (MLP) neural network. We common architecture and set of hyperparameters across all datasets. When training on images, we used ResNet-50 architecture.
For the categorical model, we found that using the mean over each output probability bin provided the best results. That is, the anomaly score for each individual x is computed as follow:
0
score = fo(x)
2
1
where is the number of bins and fx(x)) is the output probability vector of the network using a softmax, which is an N B matrix, where the across each equates batch size. The score for each instance will be a value between 0 and B 1. The higher the score instance is.
and N is the
is, the more anomalous
Employing the mode as a measurement metric proved suboptimal given the disproportionate rep- resentation of the first bin, pattern that remained consistent among anomalous instances.
19

---

Published as a conference paper at ICLR 2024
Consequently, it was observed that while the probabilities could be diffusely distributed across the remaining bins, the mode predominantly remained in the first bin. In contrast, utilizzing the mean allowed us to effectively account for this distribution characteristic, enabling an inclusive weighting scheme across all bins. Additionally, the mean offered continuous scoring system as opposed to the integer values provided by the mode, thereby affording more nuanced understanding of the anomalous data.
Table 2: Hyperparameters for parametric DTE model
Hyperparameter
Value
Hidden layer sizes
[256, 512, 256]
Activation function
ReLU
Optimizer
Adam
Learning rate
0.0001
Dropout
0.5
Batch size
64
Number of epochs
400
Maximum timestep
300
Number of bins 7
DDPM For the DDPM model, we used a modified ResNet for tabular data (Gorishniy et al. 2021) with added time embedding before each block, inspired by the work done for TabDDPM (Kotelnikov et al. 2022). Recognizing that learning noise at each timestep presents a considerably complex task, the necessity for a more sophisticated model than a simple MLP became evident to optimize the efficacy of our method. Furthermore, the lack of research on diffusion models for tabular data has constrained our ability to apply a model of comparable strength to the U-net model typically used for images, to our benchmark datasets. This presents an interesting direction for further research, with the potential datasets. In contrast to prior work (Wyatt et al. [0222 Wolleb et al. (0022), we do not add noise to the data point before reconstructing it as we found that it leads to overall slightly better results. This is minor change, one intuition for the boost of performance could be that adding noise can modify the images toward anomalous data, thus increasing the amount of false positives.
significantly enhance the performance of machine learning models on tabular
Table 3: Hyperparameters for DDPM model
Hyperparameter
Value
Number of blocks
Main layer size
128
Hidden layer size
256
Time embedding dimensions
256
Optimizer
Adam
Learning rate
0.0001
Dropout layer 1
0.4
Dropout layer 2 Batch size
0.1
64
Number of epochs
400
Maximum timestep
1000
Reconstruction timestep
250
Normalizing Flows Baseline We compare our diffusion methods with normalizing flows base- line that uses planar flows (Rezende Mohamed 2015). Normalizing flows allow to compute the exact likelihoods of data point, which allow to easily assign anomaly scores. Once trained, the model can estimate the density of any data point in the input space. This is done by passing the data point through the inverse of the learned transformation and then computing the density of the transformed point under the simple target distribution. The density of the original point under the complex data distribution can be computed from this using the change-of-variable formula.
20

---

Published as a conference paper at ICLR 2024
Table 4: Hyperparameters for PlanarFlow model
Hyperparameter
Value
Number of transformations
10
Optimizer
Adam
Learning rate
0.002
Batch size
64
Number of epochs
200
D.4 COMPUTE
The total amount of compute required to reproduce our experiments with five seeds, including all of the baselines and the proposed DTE model amounts 473 GPU-hours for the unsupervised setting and 225 GPU-hours for the semi-supervised setting on an RTX8000 GPU with 48 gigabytes of memory for running the ADBench datasets.
Figure 12 shows the training and inference times averaged over all datasets in ADBench over five seeds for all methods discussed Section As expected, deep learning-basee methods have sig- nificantly higher training times compared classical methods but comparable inference times. In particular, the inference time for the parametric DTEs is orders of magnitude lower than all other methods. The non non-parametric variant of DTE has training phase, show the inference time in both plots.
1.0
Talong ti)
101
1.0
CBLOK
comoa
Testordhoggnng sopping
Hayp
Lor
Mco
orswM
PCA
Dreedsvo
ORoce
GOAD
Manaroor
GiNomaay
ore (Rton Para Turamefrrs
$L4D
tlnvars Gomma)
(Categorical al)
(a) Training time
(5) 1og scale
100
100
20-4
COLOR
coroo
featuratoopr
$co0
iPorect
LoGA
ACD
ocsu
PCA
Desmsuo up
GRace
GOOD
Gatemaaly
DTE (Non Paramet uYE (nvveree
$LAD
Parametrics ric)
(Categorica)
(b) Inference time
Figure 12: Mean training and inference time the 57 datasets from ADBench over five different seeds for the semi-supervised setting using normal samples only for training. Colour scheme: red (diffusion-based), green (deep learning methods), blue (classical methods).
21

---

Published as a conference paper at ICLR 2024
E CHOICE OF REPRESENTATION FOR IMAGES
In this section, we compare the effect of choice of representation on the performance of diffusion- based anomaly detection techniques. Three choices considered are 1) pixel space representation, 2) self-supervised embedding, and 3) embedding produced by a classifier. Results for three image datasets are reported in Table[5] The datasets and preprocessing are described Appendix [D.] [D.]Jand and the full results are in Appendix F As expected, using pre-trained embeddings leads to better results than pixel space for all methods considered. Tables 9 to 12 report other experiments that lead to a similar conclusion.
In particular, using self-supervised embedding for CIFAR-10, significantly improved the anomaly detection performance the pre-training was done CIFAR-10 10 itself. Note that all the other pre-training were supervised classification using ResNet-34 on ImageNet and not directly on the datasets. Overall, pre-training improves the results for all methods and all datasets with the exception of kNN and the non-parametric DTE (DTE-NP) on MNIST. This result can be attributed to the simplicity of the MNIST dataset when adapted anomaly detection tasks. As reminder, DTE-NP is equivalent to KNN, but corresponds to the variation that uses the mean distance of the k-nearest neighbours instead of the distance the kth-nearest neighbour.
Zou et al. (2022) highlighted the advantages of tailoring specialized self self-supervised learning tech- niques to specific datasets, exemplified by their method for VisA. As our methods are not explicitly designed for these datasets, our results for all diffusion-based methods reported here lag behind those of methods specialized to this dataset. In particular, VisA dataset contains images that are quite similar with the exception of highly localized anomalies.
Table Average AUC ROC and standard deviations for the different subsets of each dataset, average across 5 runs, semi-supervised setting using different pre- pre-training algorithms.
DTE-NP
DTE-C
DDPM
kNN
VisA, supervised ImageNet pre- pre-tnain
83,63(10.50)
81.07(11.01)
83.26(10.64) 80.47(12.47)
VisA, VicReg ImageNet pre- pe-training
83.36(12.44)
81.89(12.26)
83.14(13.76)
83.68(13.54
VisA, no pre-training
75.96(10.54)
64.53(19.61)
57.85(21.74)
75.40(9.85)
CIFARIO, supervised ImageNet pre- pr-itrinnig
53.91(7.16)
52.57(5.53)
52.96(7.05)
54.42(7.56)
CIFARIO, VicReg pre-training
80.92(10.81)
63.36(11.92)
54.22(10.26)
79.01(11.53)
CIFAR10, 10, no pre-training
51.53(14.81)
50.25(3.34)
50.50(7.67)
51.64(14.90)
MNIST, supervised ImageNet pre-training
78.07(12.48)
64.34(11.93)
60.62(10.26)
76.86(11.4)
MNIST, pre-training
81.94(16.46)
49.02(16.51)
51.29(18.85)
84.14(15.60)
F
FULL RESULTS
We provide the full table of results corresponding to the AUC ROC box-plots in Section 4 additional metrics including Fl score and area under the precision-recall curve (AUC PR) along with the corresponding box -looss. All results are shown averaged across five seeds along with standard deviations in brackets for all 57 datasets in ADBench. In the subsequent tables, DTE-NP refers to the non-paramttric DTE estimator, DTE-IG refers to the parametric inverse Gamma model, and DTE-C refers to the parametric categorical model. Tables to 12 show the results for three methods when using are-trained embeddings on CIFAR-10 and SVHN compared to trained directly on the images, it is up in ADBench. The difference with Table is that instead of having one class as an anomaly, here we have one class as normal while the rest of the classes are downsampled to produce the anomalies.
We report
22

---

Published as a conference paper at ICLR 2024
F.1 SEMI-SUPERVISED SETTING
DTE
(Categorical)
(inverse
(Inverse Gemmal
DTE (Non Perametrc)
DTE (Non Perametrc
DE
GANomaly
GAllmaay
PannorFow
Rannrroww
ICL
DROCK
DROC
SAGUR
DACMI
ocSMM
MCD
LOD4
LODA
Forsst
Forest
PeotueBagghn
Reature
EC00
cC30
CS.DF
CSLDF
(a) Fl scores
(b) AUC PR scores
Figure 13: F1 score and AUC PR means and standard deviations on the 57 datasets from ADBench over five different seeds for the semi-supervised setting using normal samples only for training. Colour scheme: red (diffusion-based), green (deep learning methods), blue (classical methods).
Table 6: Average AUC ROC and standard deviations for 5 runs of the VisA dataset, semi-supervised setting using embeddings supervised ResNet-34 pre-trained on ImageNet with the same training split.
DTE-NP
DTE-C
DDPM
kNN
candle
90.88(0.0)
89.26(3.37)
87.37(0.26)
90.76(0.0)
capsules
62.77(0.0)
56.04(3.51)
66.65(0.6)
62.67(0.0)
cashew
93.7(0.0)
87.17(2.95)
89.69(0.25)
93.24(0.0)
chewinggum
93.88(0.0)
94.69(1.53)
92.55(0.26)
93.52(0.0)
fryum
87.38(0.0)
81.25(3.91)
85.84(0.17)
87.68(0.0)
macaronil
70.27(0.0)
71.9(4.94)
64.33(0.69)
69.34(0.0)
macaroni2
67.65(0.0)
66.77(2.66)
51.6(0.64)
66.35(0.0)
pcbl
90.14(0.0)
85.49(1.72)
94.38(0.37)
90.67(0.0)
pcb2
88.88(0.0)
81.32(2.66)
83.24(0.4)
87.77(0.0)
pcb3
80.83(0.0)
81.72(1.98)
79.61(0.2)
81.25(0.0)
pcb4
93.77(0.0)
91.05(2.46)
88.7(0.56)
93.07(0.0)
pipe fryum
83.24(0.0)
86.19(2.38)
81.72(0.36)
82.86(0.0)
mean
83.63(10.50)
81.07(11.01)
80.47(12.47)
83.26(10.64)
23

---

Published as a conference paper at ICLR 2024
Table Average AUC ROC and standard deviations for runs of the VisA dataset, semi-supervised setting using embeddings of VicReg pre-trained on ImageNet with the same training split.
DTE-NP
DTE-C
DDPM
kNN
candle
82.97(0.0)
84.72(0.))
85.27(0.07)
85.44(0.0)
capsules
65.63(0.0)
68.24(1.08)
69.01(0.6)
65.92(0.0)
cashew
90.7(0.0)
82.75(5.28)
90.26(0.43)
90.74(0.0)
chewinggum
97.98(0.0)
97.78(0.12)
97.78(0.08)
98.0(0.0)
fryum
88.88(0.0)
79.62(2.69)
89.13(0.23)
88.82(0.0)
macaronil
70.03(0.0)
68.17(1.19)
64.09(0.117
69.52(0.0)
macaroni2
52.06(0.0)
55.81(2.06)
51.93(0.27)
52.16(0.0)
peb1
93.02(0.0)
91.07(0.5)
93.19(0.08)
93.22(0.0)
pcb2
85.7(0.0)
83.77(0.96)
83.68(0.17)
85.69(0.0)
pcb3
83.26(0.0)
81.57(0.75)
82.03(0.13)
83.05(0.0)
pcb4
98.6(0.0)
98.21(0.35)
98.35(0.03)
98.66(0.0)
pipe fryum
91.44(0.0)
90.98(1.29)
93.05(0.05)
92.96(0.0)
mean
83.36(12.44)
81.89(12.26)
83.14(13.76)
83.68(13.54)
Table 8: Average AUC ROC and standard deviations for 5 runs of the VisA dataset, semi semi-supervised setting using the images directly with the same training split.
DTE-NP
DTE-C
DDPM
kNN
candle
77.48(0.0)
83.03(4.42)
51.96(6.2)
77.38(0.0)
capsules
63.75(0.0)
72.61(7.91)
33.19(0.37)
68.02(0.0)
cashew
90.14(0.0)
79.5(27.78)
96.26(0.56)
93.32(0.0)
chewinggum
66.92(0.0)
56.99(4.66)
68.82(1.02)
65.66(0.0)
fryum
74.32(0.0)
77.28(10.99)
25.24(1.22)
74.5(0.0)
macaronil
68.67(0.0)
54.52(20.66)
74.7(1.14)
70.11
macaroni2
74.04(0.0)
54.48(8.11)
37.04(0.48)
77.02(0.0)
pebl
83.59(0.0)
51.53(16.31)
72.02(0.97)
80.53(0.0)
pcb2
87.4(0.0)
74.04(15.82)
77.56(0.55)
78.87(0.0)
pcb3
71.75(0.0)
40.86(8.75)
68.11(2.12)
66.03(0.0)
pcb4
94.5(0.0)
73.61(11.82)
28.46(2.18)
92.94(0.0)
pipe fryum
58.96(0.0)
56.04(19.1)
60.95(5.78)
60.42(0.0)
mean
75.96(10.54)
64.53(19.61)
57.85(21.74)
75.40(9.85)
Table 9: Mean AUC ROC and standard deviation over seeds for different methods trained on the images directly versus trained on embeddings generated by pre-trained ResNet-18 on ImageNet for the unsupervised setting the CIFAR-10 dataset.
DDPM
DTE-C
kNN
Images
54.72(4.55)
48.95(8.33)
57.45(1.55) 62.87(1.57)
Embeddings
66.34(0.14)
66.13())33)
Table 10: Mean AUC ROC and standard deviation over seeds for different methods trained on the images directly versus trained embeddings generated by pre-trained ResNet-13 -1 on ImageNet for the unsupervised setting on the SVHN dataset.
DDPM
DTE-C
kNN
Images
54.97(2.41)
49.07(3.06)
56.29(1.22)
Embeddings
61.48(0.24)
59.96(1.24)
61.17(0.28)
24

---

Published as a conference paper at ICLR 2024
Table 11: Mean AUC ROC and standard deviation over seeds for different methods trained on the images directly versus trained on embeddings generated by pre-trained ResNet-18 on ImageNet for the semi-supervised setting on the CIFAR-10 dataset.
DDPM
DTE-C
kNN
Images
55.96(4.69
52.66(6.22
59.10(1.80
Embeddings
67.91(0.13)
68.53(1.59)
67.53(0.0)
Table Mean AUC ROC and standard deviation over seeds different methods trained on the images directly versus trained on embeddings generated by pre-trained ResNet-18 on ImageNet for the semi-supervised setting on the SVHN dataset.
DDPM
DTE-C
kNN
Images
57.28(2.75)
48.78(4.23)
55.92(1.17)
Embeddings
61.37(0.08)
62.91(1.1)
61.69(0.0)
F.2 UNSUPERVISED SETTING
DTE (Catogerical) DTE (Nio Miio-PPae Parametns]
DTE (Categorical) (Inverse
DTE (Non Parametrc)
SLAD
ut
GANomdy
GWlomaa
VAt
VAC
RanorFlow
RonarFow
[El
DROCK
DEOK
DAGUN
MCD
MDD
LoDA
LODA
Forect
HBDS
ReaureBaggng
Reauursaggg1n
E500
2500
Coudh
CRUDT
30
AUC
(a) Fl scores
(b) AUC PR scores
Figure 14: score and AUC PR means and standard deviations on the 57 datasets from ADBench over five different seeds for the unsupervised setting with bootstrapped training instances. Colour scheme: red (diffusion baasedd,oonbaeed),,.. green (deep learning methods), blue (classical methods).
25

---

Published as a conference paper at ICLR 2024
SESII HORN AUND EEVIEM SARSE SENEU
WIILA WIN INICH
JIIG GEROON RERE WHIDER BORROD WORER CINELL
GAMEE MESIN KNUSH NORER ZOOBU BLOK WO CUA FEMOA TOMA GINI EAIN SARDD S0ODS ARAND MUEEN PUST BLOOLD SADNI MARED SANIO BAANDD GLA FOOTH STRAND NORIE MANO
SHAIIN ME SOMER
RUDES ANDS
WANTI MORTIO METHAAI
WANIH GEEEEN THAME
FOUD PRII SOMAIE DOO TO MUSE
Table 13: Average AUC ROC and standard deviations over five sceds for the semi- semi--sueeee supervise setting on ADBench.
SESIED METIR BUM MI
WOODIN MO MOTE WONEI RENCS MUDEO OF NONO CENET RESTIN MANEON
TURLIN DICK
SOALN NETRETO
TO AOCOOD AOOD OF AINNN
IN DOOGH AWOTES OF WOMA
FOR ON WIN GOOD DORTN THANE AORII AODE TH DOODD DENUR DEONO
AO
HOME AEOON MESIE EN MADENN
WONINA TO IN
TO
CON II SIIE SENT DORELA AITH TE EENE TLRNNO EIEIE BI SEWOOO
RUNEDS BUIMOT TH CIIN TEGIIAR
WAROAI NOR TE
MISS IN ILSEES ILD OI MU LIDES NOURES UR
SUIEI ANA ON THEI ANO FONN IN WETHH AOOOND THE
THE FONUTH MNTTN MUDE OI
ROMAO OI
OO MURD
WAIN
BOASS WART GOOT MORES WHINIE
MEAR TN WN ON
HRIN THE
THE
IN THEN
TORRAA TOA PRIEEO BEAN IN OUW EN
SOON TUAM IN AN
SESIDA
WE ON
MENEER OIE TH
MORDD AURE SH
WEI WE
MOUE DUNOS
305- BIS ON
OHIIN WORN
MODED
SOATE
26

---

Published as a conference paper at ICLR 2024
2010
WHOT
THAN CHICK RRIKER ANR
DARE WORE WHIR WHAN HOTT WHITD THME TOROO MILL
AETIM 2000
TOIII THANT CHIIE TIRAO ON WONH WANE THNG
ME0
Table 14: Average Fl score and standard deviations over five seeds for the semi sem-- supervisee setting on ADBench.
THE AHIN
IE HOTER
800830 SUROPAD
HE MEDIEN LORE INNON
24
SIWE TH
ALOE 30
WARE TO
TO
SOT DINGNG ME MERNCO TH
FLRE TO
MILKN
WREND DONE THE BOO
A BRIND MOTA
ALMOND MO PERNIIM DEI AEA ON RSISO BIRIESA SHITE
Doskiie
DOGH
2000 N0
AVEIAR RUSE DE HORE
HONN ALSION SIST
WOINN TINER LIGH FNR BEDIAN
ENILR BRANND WEAN WERNN HORE OOO
OC5W
SHOLLD
CAINA DOODIO WOEEE MANOO PRANCD
SHISSN
METIE HO OO
HER HNDON FRMIN
OR TH
HOOD ONA
TO
MENEENT WIIROR WORD MORI
COND
MOGER
27

---

Published as a conference paper at ICLR 2024
HOMEE WONN MORT TRAN ANN BLEE EII
BRANDS ANAN MERION aite tn
HEING ROLE
HOMEN ORILLS
MENI MONDD MHE CENER SONEL
WOMER THE BUER DONE AI
NEOWO LON HIGTO SHIN AONE
AUNE
WOME WO TH WAEST TEIN FORN HUMI BONIO AIN I.
THEE IN
ORDOON HEME HOOT ROII
PRNETOO SEEWN HURDM HONI MERUR
CONIN MOOT MOTRIA
113
Table 15: Average AUC PR and standard deviations over five seeds for the semi-sussse sun--supervieed setting on ADBench.
DI BESTE TRON ON ALAVO FERID
FRED
HODE DUMESS NEWIN MODSS
MEN SESTRO TLIR
(2IILA BO BEC METH NOTE TIE
THEN WND
ON TO OO
DIELI TIET HOME
WOR
WNON
WONH
WILLO
HINL WI LINIE LENIEE
S1R
RESTLA WINT MINEIN CONT IN CONLEN HESITI AERIIR MESILUI MIIIIN TENTT
AUTHI 2ATT
DO
LINE HURLE TO IN PIGTE AESHA AWILI
THRES
HORN
IN WHE MHRCE FONDE IN FAOII HOW
COSEESS JUSIN LAROO HO D LININ WONGH DIDIAN THE
IS MARER TORIEN DOAM
IE IN AOOOE
SOR FOUR DONN D
IN
TINESS
BROR FOR SORIA PROEON
122001 FUNEO BEVR HOWAS
WILL CHND DINT
OT MORTH THAN
COHEEN CURO DANON
HURD WIRIO
28

---

Published as a conference paper at ICLR 2024
WHIER
DR PIC
SONEC ANE SIEE TO RORGR DERI
30.30
SINEE EXTROMI
MADIR SAAI TIS ALOTE
Table 16: Average AUC ROC and standard deviations over five seeds for the unsupervised setting on ADBench.
SIMAR
BOREN BASOOD
4.R RUIDE TOAIS
2006 ALWER BUDWIN
BITTIN BOON BURIDA
TY MNGT
Tash VAG
TO
MART TO
ANIN AN BAAND TN
BULS
ALTIIN MARLER
TO THORD DR NORDN REST EE RI MARCON AODK THE SHNEN
HOTES MESS SNNS AN MOLDO
GO0D
ME TO
TO
AUDD THE
DRICE
HESS MAN FESSED THE PENIOR ROIL SURES FLATE REME OI MURGS
IN
MARIER
RIIN AADI SURORI MUREO ON ON
ON DI OI
O WUR OI
HORAY THAS 1EX S00CN
DIAG MIR HERT THE ARRMIN
THE AEOE BOST
SAIN
MAILTO AO BHOTT
MICE ON
RESILET MIDI DO TH MUSHA
MUNTE WHOS BELL TH
THE WHOSTH AN
HENUING AHE THE TH
OF COOT
12-42-4 42
CORNN TINE THE ANE
SWADD
SENINO SUN
CRUIF 030
29

---

Published as a conference paper at ICLR 2024
MUSIEM MURSS AOIEN SEANES N1 SA BON 1-
FIATH SIREU SAMER LOREE RINI MONA OAIH
THOM
BIDE SINCE RANAO
AMOSE RAS 30
ASHIS MEED HON SONEOO ARION FIEE SEEK TOOTE NODO
SINEA
Table 17: Average Fl score and standard deviations over five seeds for the unsupervised setting on ADBench.
SEGHING FLORES
BE ACHIO AEWARD
HOSE COEE
MIEE MHEM
WHIS
AUN BROND
AUDO
WHE
FORIED IN RINN CHILE
DIA AHERO
SNIL SNOW
HOREO IN HEII
IND TO
SINE
WOIND
BEI IN
TRONES
WOOO CINN
GORLS
30

---

Published as a conference paper at ICLR 2024
LEMESS REDICIO
OF 1872 BIAAP DIERD B024
TORL ANE
MAMTION SALD
SIN
1 OE
Table 18: Average AUC PR and standard deviations over five seeds for the unsupervised setting on ADBench.
SINES
SAIAIO
JAS IN HOREON
TLANDD IO
AND CEMMIN DO
T4E 1900
Reat
THE DECLO
AIRUES IE4 RENIIS
ESOW
PORTH
000 R0
50050
SROSH
18000
WHITH WHD
SLTINE KERVON JORTH
WONN ROA
LNNO AURON WHIN HRR
PCA
0009
MARIN WITH
MENORM IN
WHESIN
IN ARINOD
ML2
1000
MOC
MAN FRINO WANTH EIANN CHESH
ADHOT
ON MEE SURAIH AESSION
COTAAN SUN AN MUNNI SEMNTTI
MANION
CRUP COSD
CHION
31


Reviews:
[{"review_id": "zluJMjfJEw", "reviewer": "Reviewer_BHmN", "summary": "The manuscript proposes *diffusion time estimation* (DTE) for the task of unsupervised and semi-supervised point anomaly detection. DTE assumes a data point to be produced by diffusion process and estimates the distribution of the denoising time step required to reconstruct the data point. The mean of mode of the distribution is regarded as the anomaly score of the data point. \n\nIn addition to its effectiveness demonstrated in prior works, DTE avoids the actual denoising process redundant for anomaly detection and directly estimates the extend to which the sample appers to be anomalous. With this keen insight, the manuscript provides detailed derivation of the posterior distribution of variance of time (decided by time step) given an input image assumed to be produced by a diffusion process. Based on the derivation, the manuscript designs one non-parametric model and two parametric models (regressive and categorical respectively). \n\nThe performances of the three models are evaluated on 57 datasets from ADBench demonstrating the capabilities of DTE for the task of anomaly detection and its advantages over DDPM in quaility and efficacy.", "questions": "See *Weaknesses*.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "* The paper provides a new perspective for adopting diffusion modeling in the field of anomaly detection\n* The paper provides one parametric and two non-parametric practical models for the task of anomaly detection \n* The methods proposed in the paper (DTE) achieve significate margins in both performance and efficacy compared with DDPM", "weaknesses": "* The advantage of DTE methods hold should be demonstrated quantitatively. As the DTE methods perform worse than kNN in both quality and efficiency, quantitative results are recommended to demonstrate the distinctions in scalability of DTE methods\n* The presentation in Figure 3 needs optimization. To demonstrate the small difference between non-parametric estimate and analytical posterior, the visualization of residual part seems more straightforward.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "QiX0JX6TjT", "reviewer": "Reviewer_ahnY", "summary": "The work explores the use of diffusion models for anomaly detection (AD), and proposes an AD method based on diffusion time estimation (DTE), with three models under the DTE framework introduced. The DTE models, particularly the DNN-based parametric model, can achieve desired detection performance while substantially reducing the inference time. The models are evaluated on 57 datasets and show comparable performance compared to a set of 19 baseline/SOTA methods in both semi-supervised and unsupervised settings.", "questions": "Please see the weaknesses.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The work is well motivated and easy-to-follow.\n- The idea of using diffusion time estimation (DTE) for AD is interesting and new. It also provides a way for learning parametric DTE models that allow efficient inference time.\n- The proposed DTE models generally perform substantially better than the popular diffusion model DDPM, and show comparable performance to a large number of competing methods on 57 tabular datasets.", "weaknesses": "- The performance of the DTE models seems to be upper bounded by the simple kNN-based AD method. There are a number of kNN-based AD methods, including some deep methods like Refs [1]. It would be helpful for the empirical evidence support if these advanced kNN variants are included in the empirical comparison.\n- The models rely on distance in original feature space, and they would fail to work if the data lies in very high-dimensional space, e.g., datasets with hundreds of thousands of features or millions of features.\n- Since the method is based on generative models, it is important to discuss and compare with other generative model-based AD methods, such as GAN-based methods, to highlight the advantages of the proposed method. \n- Since the models directly work on tabular datasets, it is misleading to claim that the evaluation is performed on diverse tabular, image, and natural language datasets.\n- The work may be improved by having more discussion on recent diffusion model-based AD studies, such as [2-4].\n\n\n\n**Refs**\n- [1] Learning representations of ultrahigh-dimensional data for random distance-based outlier detection. In Proceedings of the 24th ACM SIGKDD international conference on knowledge discovery & data mining (pp. 2041-2050).\n- [2] Unsupervised Surface Anomaly Detection with Diffusion Probabilistic Model. In Proceedings of the IEEE/CVF International Conference on Computer Vision (pp. 6782-6791).\n- [3] Feature Prediction Diffusion Model for Video Anomaly Detection. In Proceedings of the IEEE/CVF International Conference on Computer Vision (pp. 5527-5537).\n- [4] Multimodal Motion Conditioned Diffusion Model for Skeleton-based Video Anomaly Detection. In Proceedings of the IEEE/CVF International Conference on Computer Vision (pp. 10318-10329).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qEzRJM9h88", "reviewer": "Reviewer_vggP", "summary": "The paper proposes a non-parametric approach based on the inverse Gamma distribution of diffusion time for noisy input, achieving accurate predictions and ranking anomalies similarly to kNN. Additionally, a parametric strategy employs a deep neural network for large datasets, demonstrating competitive performance and significantly improving inference time. Pre-trained embeddings for images are found to enhance diffusion-based methods, highlighting the potential advantage of using latent space diffusion. The evaluation on ADBench, a benchmark for anomaly detection datasets, shows promising results in comparison to prior work.", "questions": "How does varying the ratio of labeled data in the semi-supervised setting affect the performance of DTE? Can this method be extend to more challenging tasks, such as the localization of anomaly in data?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "+ The proposed approach offers a simpler alternative that avoids modeling the reverse diffusion process. Instead, it focuses on modeling the distribution over diffusion time associated with noisy input samples. The assumption is that anomalies are distant from the data manifold, leading to higher density for larger timesteps in the distribution.\n+ Both non-parametric and parametric strategies are employed for DTE based anomaly detection, and the parametric strategies achieve a tradeoff between accuracy and inference time.\n+ The evaluation is conducted on ADBench, as well as additional image datasets such as Visa, CIFAR-10, and MNIST.", "weaknesses": "-The performance in the semi-supervised setting is more competitive compared to the unsupervised setting. This indicates that DTE benefits from labeled data, allowing for a more accurate modeling of the distribution of diffusion time.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "NNMTQ2oPM2", "reviewer": "Reviewer_udy2", "summary": "In this paper, the authors investigate diffusion modeling for anomaly detection and introduce an efficient approach called Diffusion Time Estimation (DTE), showing its competitive performance and improved inference times compared to traditional methods and deep learning techniques.", "questions": "See weakness.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. This paper is well-organized and easy to follow.\n2. Investigating the diffusion model to facilitate anomaly detection is an interesting and promising research issue.\n3. The authors conduct comprehensive experiments to demonstrate the effectiveness of the proposed method.", "weaknesses": "1. The contribution of this paper appears limited. There are several existing works on anomaly detection utilizing diffusion models, such as those mentioned below:\n[1] Unsupervised Surface Anomaly Detection with Diffusion Probabilistic Model, ICCV23.\n[2] Feature Prediction Diffusion Model for Video Anomaly Detection, ICCV23.\n[3] DiffusionAD: Denoising Diffusion for Anomaly Detection, Arxiv.\n[4] Diffusion models for medical anomaly detection, MCCAI22.\nCan the authors point out the unique contributions of this paper compared with them?\n2. The motivation and rationale for introducing the diffusion model into anomaly detection are somewhat unclear. The authors should emphasize it.\n3. Authors should compare with some latest AD methods proposed in 2023. Besides, it would be more convincing to include diffusion model-based anomaly detection approaches.\n4. The reviewer has thoroughly reviewed the experimental results presented in the appendix. The authors have done a comprehensive experiment by including the results from dozens of datasets in this paper. However, the reviewer observed that the proposed method did not consistently outperform other methods on each individual dataset. It achieved the best performance only on a subset of the datasets.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (spotlight)