Disentangled Conditional Variational Autoencoder for Unsupervised Anomaly Detection
Authors: Asif Ahmed Neloy
Maxime Turgeon
Venue: Submitted to ICLR 2023 (2023)
Keywords: unsupervised anomaly detection
autoencoder
disentanglement learning
representation learning
information theory

Abstract: Recently, generative models have shown promising performance in anomaly detection tasks. Specifically, autoencoders learn representations of high-dimensional data, and their reconstruction ability can be used to assess whether a new instance is likely to be anomalous. However, the primary challenge of unsupervised anomaly detection (UAD) is in learning appropriate disentangled features and avoiding information loss, while incorporating known sources of variation to improve the reconstruction. In this paper, we propose a novel architecture of generative autoencoder by combining the frameworks of $\beta$-VAE, conditional variational autoencoder (CVAE), and the principle of total correlation (TC). We show that our architecture improves the disentanglement of latent features, optimizes TC loss more efficiently, and improves the ability to detect anomalies in an unsupervised manner with respect to high-dimensional instances, such as in imaging datasets. Through both qualitative and quantitative experiments on several benchmark datasets, we demonstrate that our proposed method excels in terms of both anomaly detection and capturing disentangled features. Our analysis underlines the importance of learning disentangled features for UAD tasks.

Under review as a conference paper at ICLR 2023
DISENTANGLED CONDITIONAL VARIATIONAL AU- TOENCODER FOR UNSUPERVISED ANOMALY DETEC- TION
Anonymous authors Paper under double-bling brind review
ABSTRACT
Recently, generative models have shown promising performance anomaly de- tection tasks. Specifically, autoencoders learn representations of high-dimensional data, and their reconstruction ability can be used to assess whether new in- stance is likely to be anomalous. However, the primary challenge of unsuper- vised anomaly detection (UAD) is learning appropriate disentangled features and avoiding information loss, while incorporating known sources of variation to improve the reconstruction. In this paper, we propose a novel architecture generative autoencoder by combining the frameworks of 3-VAE, conditional vari- ational autoencoder (CVAE), and the principle of total correlation (TC). We show that our architecture improves the disentanglement of latent features, optimizes TC loss more efficiently, and improves the ability to detect anomalies in an unsu- pervised manner with respect high-dimensional instances, such as in imaging datasets. Through both qualitative and quantitative experiments on several bench- mark datasets, we demonstrate that our proposed method excels in terms of both anomaly detection and capturing disentangled features. Our analysis underlines the importance learning disentangled features for UAD tasks.
INTRODUCTION
Unsupervised anomaly detection (UAD) has been fertile ground for methodological research for several decades. Recently, generative models, such as Variational Autoencoders (VAEs) (Kingma & Welling 2014) and Generative Adversarial Networks (GANs) (Goodfellow et al.) 2020 Arjovsky itt al. 2017). have shown exceptional performance at UAD tasks. By learning the distribution of normal data, generative models can naturally score new data as anomalous based on how well they can be reconstructed. For a recent review of deep learning for anomaly detection, see Pang et al. (221I.
In a complex task like UAD, disentanglement as a meta-prior encourages latent factors to be cap- tured by different independent variables in the low-dimensional representation. This phenomenon has been on disply in recent work that has used representation learning as a backbone for develop- ing new VAE architectures. Some the methods proposed new objective functions (Higgins al. 2011
Mathieu et al.) 2019). efficient decomposition of the evidence lower bound (ELBO) (Chen) al. 2018), partitioning of the latent space by adding regularization term the mutual infor- mation function (Zhao et al. 2017)), introducing disentanglement metrics (Kim & Mnih 2018), and penalizing total correlation (TC) loss (Gao al. 2019). Penalized TC efficiently learns disentangled features and minimizes the dependence across the dimension of the latent space. However, it often leads to loss of information, which leads lower reconstruction quality. For example, methods such as 3- VAE, Disentangling by Factorising (FactorVAB) (Kim & Mnih 2018), and Relevance Factor VAE (RFVAE) (Kim et al. 2019) encourage more factorized representations with the cost of either losing reconstruction quality losing considerable among of information about the data and drop in disentanglement performance. To draw clear boundaries between an anomalous sample and normal sample, must minimize information loss.
To address these limitations, we present Disentangled Conditional Variational Autoencoder (dC- VAE). Our approach is based on multivariate mutual information theory. Our main contribution is

---

Under review as a conference paper at ICLR 2023
generative modeling architecture which learns disentangled representations of the data while min-
imizing the loss information and thus maintaining good reconstruction capabilities. We achieve this by modeling known sources of variation, in similar fashion Conditional VAE (Pol et al. 2019).
Our paper is structured follows. We first briefty discuss related methods (Section 2). draw connec- tion between them, and present our proposed method dCVAE (Section (3). In Section we discuss our experimental design including competing methods, datasets, and model configuration. Finally, experimental results are presented in Section and Section concludes this paper.
RELATED WORK
In this section, we discuss related work on autoencoders. We focus on two types of architecture: ex- tensions of VAE enforcing disentanglement, and architectures based on mutual information theory.
2.1 3-VAE
Chen et al. 2018) is an augmentation of the original VAE with learning constraints of 3 applied to the objective balance the latent channel capacity and improve the reconstruction accuracy. As a result, B 3-VAE is capable of discovering the disentangled latent factors and generating more realistic samples while retaining the small distance between the actual and estimated distributions.
8-VAE and its extensions proposed by (Higgins et al.) 2017 Mathieu al. 2019
function of the VAE. The idea of including such
Recall the objective function of VAE proposed by Kingma & Welling 2014
LVAE(0, (x)))) log (x z) + DKL (4s x))lpe(z)).
(1)
Here, [x(x z) is the probabilistic decoder, (z x) is the recognition model. KLD is denoted by ( (g x) x)llpo (z x)) parameterized by the weights (0) and bias (ø) of inference and generative models. As the incentive of 3-VAE is to introduce the disentangling property, maximizing the probability of generating original data, and minimizing the distance between them, a constant is introduced in the objective VAE to formulate the approximate posterior distributions as below:
max Ex-x (=l()) [o =]]] such that DRL. (a=(= ()) [p)()) < 5. 0,8
(2)
Rewriting the Equation in Lagrangian form and using the KKT conditions, Higgins et al. (2017)
derive the following objective function:
(0. [xp(=))) Po 2²) - 8DLL (qo(2 x))lp(=)),
Here, is the regularization coefficient that enforces the constraints limit the capacity of the latent information z. When 3 = 1, we recover the original VAE. Increasing the value of B enforces the constraints to capture disentanglement. However, Hoffman et al. (2017) argue that with an implicit prior, optimizing the regularized ELBO equivalent to performing variational expectation maximization (EM).
2.2 FACTORVAE
Disentangling by Factorising or FactorVAE is another modification of 3-VAE proposed by Kim & Mnih (2018). FactorVAE emphasizes the trade-off between disentanglement and reconstruction quality. The authors primarily focused on the objective function of the VAE and 3-VAE. The authors propose loss function mitigate the loss of information that arise while penalizing both the mutual information and the KLD to enforce disentangled latent factors.
According to [Hffman Johnson (2016) and Makhzani & Frey (017)), the objective function of 3-VAE can be further extended into:
Exan (x) [K [[((((> x))[)(=))] T(x;)) + KT{g(=)])<<))),
2

---

Under review as a conference paper at ICLR 2023
Here, (2; 2) is the mutual information between and under the joint distribution Piata (z)g x))
Factor VAE learns the second term of (L(g(z)lpp())) and resolved the aforementioned issues by introducing total correlation penalty and density-ratio trick approximate the distribution ((2)
generated by d samples from g(2). The loss function the FactorVAE as follows:
(alart)) [logP (o(0)) 12)] - KT. (1 (x(()) [lp(z))
(5)
- ->KL(((())l((2)
2.3 THE PRINCIPLE OF TOTAL CORRELATION EXPLANATION (COREX)
Gao et al. (2019) introduced CorEx to mitigate the problem of learning disentangled and inter- pretable representations in a purely information-theerrtie way. general, for VAE, we assume generative model where x is function of a latent variable and afterward maximize the log likelihood of x. On the other hand, CorEx follows the reverse process where z a stochas- tic function of parameterized by 0, i.c., pa(z x), and seek to estimate the joint distribution po(x,20 z) = po (z x))(x)) The underlying true data distribution maximizes the following objective:
x(0;x) (Cx(x;2)
(Co(z)
informativenees
(dis)enianglemmen
(6)
TC(() - TCs 2) - (z).
Recall the definition of the total correlation (TC) in terms of entropy H (x) (Suudeny & Vejnarová 1998):
TC(x) = Σ H (x,) H(x) = DKL p(x) II p(x))
111
By non-negativity of TC, Equation naturally forms variational lower bound TC(x) to the CorEx objective, i.e., TC(x) > C(0: x) for any 0. Equation 6 can be rewritten in terms of mutual infor- mation 1(x 2) = (x) (x z) = (2) H(2 x). Further constraining the search space pe(z) x) to have the factorized form pe (z x) I[ (z; x) and the mutual information terms can be bounded by approximating the conditional distributions po (x, z) and po(z, x). Finally, can further rewrite and derive the lower bound for the objective function:
m
[(B;x)
Σ Lo (xq :2)
Σ
Lo (z, : x)
Σ H (x:)
Exo(
log log (2) decoder
(8)
DrL DKL(po(2 x) [l)).
eneoder
2.4 TOTAL CORRELATION VARIATIONAL AUTOENCODER (3-TCVAE)
Chen et al. 2018) proposed disentanglement in their learned representations by adjusting the func- tional structure the ELBO objective. The authors argued that each dimension disentangled representation should be able represent a different factor of variation in the data and be changed independently of the other dimensions. 3-TCVAE modifies the originally proposed ELBO objective by Higgins al. (2017) forcing the algorithm to lear representations without explicitly making restrictions or reduction to the latent space. Recall the ELBO objective function (Equation 3 of 3-VAE:
3

---

Under review as a conference paper at ICLR 2023
AE (0,0) = Eg po |²]] BDLL (90(2 x))p((z))
(9)
To introduce TC and disentanglement into the original 3-VAE, Chen et al. decomposed the original KLD into Index-Code MI, Total Correlation and Dimension-wise KL terms. Furthermore, in the ELBO TC-Decomposition, each training samples identified with a unique index and uniform random variable that refers to the aggregated posterior as ((2) Em g(> n)p(n) and can be denoted as:
[KL(g(> [n)lp(())) KL(((2 KL ((())) (daa)l (1(es)
(10)
Σ (g (l())))
Finally, with set of latent variables 2,. with known factors ug. the authors introduced a disentan- glement measuring metric called mutual information gap (MIG) and defined in terms of empirical mutual information In (2):0)):
1 K
In (B)(K):
max In (zy; uk) (#2(R)
(1I)
K
11 (vx)
Here, j(b) argmax, In (zy; u&) and K is the number of known factors under vk-
DISENTANGLED CONDITIONAL VARIATIONAL AUTOENCODER (DCVAE)
Our approach builds CorEx and models known sources of variation in the data, in manner similar to Conditional Variational Autoencoder (CVAE) Pol et al. (2019). In what follows, we will represent this known source of variation using the variable C. In the experiment below, C' is discrete and represents the class of each image. Modifying Equation to incorporate C, we get
[(0; = TC c) - TCo(= 2, - TCp(2 cc)
(12)
Recall that the first two terms measure the amount of correlation explained by 2, and by maximiz- ing it, we maximize the informativeness the latent representation. The third term measures the correlation between the components of 2, and by minimizing it, maximize the disentanglement between the latent dimensions.
can define the conditional differential entropy of Hxr) given and interpret mutual information as a reduction in uncertainty after conditioning:
Using Mutual Information Theory (Studený & Vejnarová 1998
I(x 2 = H(x c) H(x (c) H(x 2 (c) I( [x:2/c)= H(x c) H(x 2, H(s c) H(s x,
(13)
We can now rewrite Equation 12 using derived mutual information theory from Equation 13
((0;r,c)
Σ 1=1
(1, :2 =2
Σ Tegeool
(14)
Now, consider the KLD between px(x z, c) and an approximating distribution g z, c). In
terms of expectations with respect the joint distribution Pa(x, z c), write:
Hx c) F(loggo (2))) 2 F(logg (x 2,())-
(15)
4

---

Under review as a conference paper at ICLR 2023
Combing Equation 14 and 15 and assuming approximating distribution ra c) for po lc),
we obtain two inequalities:
I (zy :2 H (2, (c)- H--HC (x, 2, 2 H (zy (e) + Ellog (o( 22,)),
(16)
I (2, :z (z) = DEL (po (2j x, Ia (zg c))
(17)
Combining these bounds, we finally derive a lower bound for the objective function for dCVAE:
((0:2,0) 2 Σ (xy E (log (o( 2, 2,))) ==1
Σ 1=1
DEl (P(ss)n
(18)
illustrates the lower bound objective function of dCVAE where gx(x | z, c) is the generative model or decoder and pe (zg c) is the recognition model or encoder.
Equation 18
EXPERIMENTS
In the experiments below, we compare our dCVAE method to five baseline methods: VAE, CVAE, B-V,,, Factor- VAE, and RFVAE. The first two methods were selected as well-known baselines that do not explicitly enforce disentanglement; on the other hand, the latter three methods seek to achieve a disentangled representation of the data.
4.1 DATASETS
We evaluate dCVAE and other baseline models the following four datasets. Three datasets (MNIST (Dng 2012). Fashion-MNIST (Xiao et al. 2017). KMNIST (Clanuwat et al.) 2018)) are trained for UAD. The fourth dataset (EMNIST (Cohen al. (017)) is used for testing real- world dataset to assess overall performance. A more detailed description of these datasets follows:
MNIST and Fashion-MNIST (FMNIST) Firstly, we apply all models to two benchmark datasets, MNIST and Fashion-MNIST, for fair comparison with other baseline methods. We used 10 classes with 60000 and 10000 training and testing samples for both datasets with 28 x 28 x pixels channel.
KMNIST Secondly, we applied the same training process to another complex dataset, Kuzushiji-NIIST or KMNIST. KMNIST is drop-in replacement for the MNIST dataset, a Japanese cursive writing style. KMNIST contains similar 10 classes with 60000 and pixels channel.
10000 training and testing samples with 28 28
EMNIST Finally, models are tested on the Extending MNIST or EMNIST Dataset. Using all 62 classes (digit 0-9, letters uppercase A-Z and lowercase a-z) with 700000 and 80000 training and testing samples with 28 x processed from NIST Special Database Grother (1995) and contained handwritten digits and characters collected from over 500 writers.
pixels channels. The dataset was
4.2 RECONSTRUCTION ERROR AND ANOMALY SCORE
Leveraging methods for the discriminator as the anomaly score and drawing separation between normal and anomalous data is challenging for the divergent architectures of autoencoders. De- pending on the task the architecture is trained for, the discriminator varies greatly. In general, the UAD methods utilize reconstruction error (Baur al. (018). distribution-base error (Goldstein & Uchida 2016), and density- error (Kiran et al. 2018) scores to distinguish normal and anoma- lous data. Formally, for each input æ, input क considered be anomalous if reconstruction error or Anomaly score(AA) is greater than the minimum threshold value and denoted as follows:
A(o) || D(G())))l.
(19)
5

---

Under review as a conference paper at ICLR 2023
4.3 PERFORMANCE METRICS
One of the challenges of measuring the performance of disentanglement is to apply appropriate metrics based on the nature of the dataset, not of latent factors or dimensions in the latent space. Therefore, considering the different model architectures and datasets, we first measure the perfor- mance using Numerical AUC Score, reconstruction error (A), and negative ELBO score (E). These metries provide quantifiable method of accuracy, while also measuring the disentanglement among the latent factors.
We also measure performance qualitatively by visualizing the latent space and the 3D-manifold. Both allow us to visualize of the models to handle reduced latent variables and the ability to reconstruct samples.
the orthogonality between latent features and demonstrate the accuracy
4.4 MODEL CONFIGURATION
fixed set of the computational cost and reproducibility of the models. Although baseline models that chose, 9-VAF, Factor VAF, RFVAE are highly sensitive to hyper-parameters tuning, the hyper-parameteen throughout the experiment values. A minimal 50 epochs are used to train the datasets. For MNIST, FMNIST, and KMNIST the batch size is kept 64, with primary and secondary learning rates as 10 and a 10-3 respectively, However, for the EMNIST dataset, the batch size increased to 128, and learning rates as a 10-6 and 10-5
are chosen to formulate a similar platform for all models and identify
kept consistent
observe how the models perform under similar
RESULTS AND DISCUSSION
In this section, evaluate the results of dCVAE and other baseline methods on the downstream task of anomaly detection. A considerable volume of results was produced from our exhaustive evaluation. However, accounting for limitations of space here, we elected focus the results from EMNIST and KMNIST datasets in the main text. The remaining results (MNIST and FMNIST) are presented Supplementary Material.
We show the results of our evaluation in three stages: firstly, using sample reconstruction and the negative ELBO score (E) with reconstruction error AA, we evaluate and compare the disentanglement ability of dCVAE with baseline architectures. Secondly, we use the UMAP algorithm (Sainburg al., 2021) to reduce dimensions and visualize both latent representation, of the 2D-manifold to distinguish the TC by comparing information loss and effects of modeling known sources of variation. Finally, we present AUC scores and training time to summarize the overall accuracy of the experimented methods.
well as interpolation
We evaluate the quality of disentanglement by considering explicit separation of A between normal and anomalous data and minimization of E, A better disentanglement is achieved when:
(a) A higher reconstruction error A for anomalous sample and lower reconstruction error A for normal sample is obtained and
(b) E is minimized by enforcing regularization that either minimizes the negative ELBO decompo- sition DKL (P(s)) regularizes the approximate posterior go (z) x).
A clear boundary terms of learning efficient disentanglement between dCVAE and baseline meth- ods can be observed from both EMNIST (Figure [T] and KMNIST (Figure reconstruction. The first row corresponds to anomalous reconstruction and the second row shows normal sample reconstruc- tion. Both E and A score suggests that dCVAE captures more independent factors and identifies anomalous and normal samples efficiently. This observation strongly justifies one our primary claims, namely that dCVAE incorporates the disentanglement learning through enforcing TC and restrict independent latent variables to prioritize the minimization of the divergence. The other disentanglement methods presented here either only emphasize TC (indicated by the dependence between random variables) or introduce 3 (weighing the prior enforcement term). which limits the ability learn randomness in case when the hyperparameters
tuned for certain dimensions.

---

Under review as a conference paper at ICLR 2023
dcvae
VAE
FactorVAE
betavae
RFVAE
10
10
10
20
20
20
20
20
(a) & -235
(b) & -248
(c) E = -245
(d) & -246
(e) E 2244
(I) È -246
A = 0.97
A = 0.90
A 0.91
A 0.91
A = 0.92
A 0.92
10
10
10
20
20
20
20
20
20
(g) 8
=171 (h) E = 199
(i) &
180
(i) E -15
(k) & 195
(I) 3 -190
0.3.3
A= 0.48
A = 0.53
A 0
A = 0.57
A= 0.55
Reconstruction for digit zero (0) and the capital letter O. Here, & refers to Negative ELBO score and A is the reconstruction error anomaly score. Only dCVAE and FactorVAE show steady improvement for both types of reconstruction. All the other methods misclassify the samples. Morcover, can observe higher reconstruction error and ELBO scores compared to MNIST (Figure AT) and FMNIST (Figure A2)
Figure 1:
Cvge
FactorVAE
betavae
RFVAE
10
10
20
20
20
20
20
20
(a) 3 = -251
(b) E 281
(c) & = 227
(d) E 2666
(c) E 277
(f) 3 = 2270
A 0.98
A = 0.90
A 0.98
A = 0.97
A = 0.97
A 0.96
10
10
20
20
20
20
(g) -188
(h) & -211
(i) 1185
(i) =
-201
(k)
-
()) E -170
0.28
A= 0.47
A
A=
0.411
A= 0.38
Figure 2: In KMNIST dataset, without dCVAF, all other methods fail to classify both anomalous and normal samples. Reconstruction scores suggest FactorVAE, VAE almost fail to distinguish normal and anomalous observations. Since the stroke of the samples are similar in this dataset, methods that only emphasize disentanglement or empirical approximation lose more information in latent variable resulting in false anomaly detection.
The second observation is drawn using latent representation (Figure 3 and 2D-manifold embeddings (Figure 4 and 5). Through this experiment, we observe the effect modeling using a known source of variation (I.c. introducing conditional variable C' into the objective function) and minimizing information loss through multivariate mutual information theory (i.e. decomposition of TC). We can observe clear similarities between KLD loss and modeling with known score of variance in reduced latent space. Due to enforced divergence loss, the plot of VAE and 3-VAE are noticeably different from other architectures. Feature space is more compact for VAE, 3-VAE, and we can see the cluster of the different classes are not well separated. However, conditioning the generative function (encoder) of CVAE and dCVAE provides the leverage construct higher feature space and retain more accurate information in 2D-manifold (EMNIST, Figure 4 and KMNIST, Figure Furthermore, TC reduces the correlation among disentanglement degrees when specific feature

---

Under review as a conference paper at ICLR 2023
learned (shape, strokes, color, boundaries). Such classes be observed to cluster together and the other gets scattered with higher feature space (Figure B). Compared to other methods, it is evident that dCVAE maintain consistent latent space and create separate clusters more accurately. This indicates that more disentangled variables are captured, and they retain more information through conditioning the generative model by minimizing the ELBO DEL (P(rs)n.)llr(essee))
2014
10-
10-
34-
(a) EMNIST
CAA
80
KMNIST
Figure 3: Latent Representation of EMNIST and KMNIST
(a) dCVAF
(b) VAR
(c) CVAE
(d) Factor
(c) 3-VAE
(f) REVAE
Figure 4: Manifold Embeddings (EMNIST)
699999995
(a) dCVAE
(b) VAE
(c) CVAE
(d) FactorVAE
(e) 3-VE
(i) REVAE
Figure 5: Manifold Embeddings (KMNIST)
8

---

Under review as a conference paper at ICLR 2023
Finally, Table I illustrates the results of model evaluation through AUC and training time. dCVAE outperforms other methods in terms of AUC score. However, for larger divergent datasets like KMNIST and EMNIST, VAE shows lower training time compared to dCVAE. Since VAE only optimizes the negative log-likelihoon reconstruction loss and prior enforcement term, the training takes fewer latent variables regularize, resulting in less training time. Nevertheless, compared to methods that incorporate TC (c.g. FactorVAE and RFVAE) or constraint on the posterior (3-VAE), our proposed dCVAE scales all larger datasets with higher classification accuracy.
Table 1: Evaluation metrics score
MNIST
FMNIST
EMNIST
KMNIST
Model
Training
Training
Training
Training
AUC
AUC
AUC
AUC
Time (min)
Time (min)
Time (min)
Time (min)
dCVAE
88.31
88.63
78.98
102
61.02
95
VAE
88.21
37
84.12
39
67.23
92
51.13
78
CVAE
87.57
43
83.31
48
66.01
117
42.35
104
VAC
87.11
82.78
62.91
138
49.23
117
D- VAE
85.31
51
82.31
53
65.12
123
50,01
119
REVAE
85.31
55
81.11
57
55.03
130
49.51
132
The only trade-offs in our proposed method seem occur when minimizing the negative ELBO loss. In certain conditions, dCVAE reaches lower reconstruction loss (anomalous sample) yet minimizes the negative ELBO score (Figure 3 4). In general, negative ELBO loss should illustrate symmetrical change with reconstruction error. Such inconsistency could lead to significant drop the classification accuracy, thus leading to a false anomaly detection result.
CONCLUSION
In this research, we present a novel generative variational model dCVAE, to improve the unsuper- vised anomaly detection task through disentanglement learning, TC loss, and minimizing trade-offs between reconstruction loss and reconstruction quality. Introducing a conditional variable to miti- gate the loss of information effectively captures more disentangled features and produces curate reconstructions. Such architecture could be used in wider range of applications, including generating controlled image synthesis, efficient molecular design and generation, source separation for bio-signals and images, and conditional text generation. Future research direction includes inves- tigating in the gap between the posterior and the prior distribution, resolving the trade-offs between loss function and reconstruction, and inspect dCVAE using different disentanglement metrics.
REPRODUCIBILITY STATEMENT
In this research, we carefully considered reproducibility in designing and conducting all experi- ments. In our supplemental texts, we have attached our source code. The experiments are designed independently to make the results reproducible. Image reconstruction and generation, 2D-Manifold embeddings, training time, and ELBO score calculation are performed separately from other down- stream tasks like classification accuracy, reconstruction error, and latent representation. Further- more, we used both TensorFlow and PyTorch frameworks to remove package dependencies. To remove the library dependencies and installation issues, virtual environment and package require- ment files are also added. Finally, to make the results more accessible, we also provided randomly generated images with supplementary texts.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Martin Arjovsky, Soumith Chintala, and Léon Bottou. Wasserstein generative adversarial networks. In International conference on machine learning, pp. 214 223. PMLR, 2017.
Christoph Baur, Benedikt Wiestler, Shadi Albarqouni, and Nassir Navab. Deep autoencoding models for unsupervised anomaly segmentation brain mr images. In International MICCAI brainlesion workshop, pp. 161-169. Springer, 2018.
Ricky TQ Chen, Xuechen Li, Roger Grosse, and David Duvenaud. Isolating sources of disentan- glement in VAEs. In Proceedings of the 32nd International Conference Neural Information Processing Systems, pp. 2615-2625, 2018.
Tarin Clanuwat, Mikel Brizar,rizar, Asanobu Kitamoto, Alex Lamb, Kazuaki Yamamoto, and David
Ha. Deep learning for classical japanese literature. arXiv preprint arXiv:1812.01718, 2018.
Gregory Cohen, Saeed Afshar, Jonathan Tapson, and André van Schaik. Emnist: an extension of mnist to handwritten letters (2017). arXiv preprint arXiv: 1oxXiv:700..0 770200537. 2017.
Li Deng. The MNIST database of handwritten digit images for machine learning research [best of the web]. JEEE Signal Processing Magazine, 29(6):14-1443 2012.
Shuyang Gao, Rob Brekelmans, Greg Ver Steeg, and Aram Galstyan. Auto-encoding total correla- tion explanation. In The 22nd International Conference on Artificial Intelligence and Statistics, pp. 1157-1166. PMLR, 2019.
Markus Goldstein and Seiichi Uchida. comparative evaluation of unsupervised anomaly detection algorithms for multivariate data. PloS one, 14)):20152133, 2016.
lan Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. Communications of the ACM, 63(11))13,-144 2020.
Patrick J Grother. Nist special database 19. Handprinted forms and characters database, National Institute of Standards and Technology, 10, 1995.
Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. beta-vae: Learning basic visual concepts with constrained variational framework. In ICLR, 2017.
Matthew D Hoffman and Matthew J Johnson. Elbo surgery: yet another way to carve up the varia- tional evidence lower bound. In Workshop Advances Approximate Bayesian Inference, NIPS, volume 1, 2016.
Matthew D Hoffman, Carlos Riquelme, and Matthew J Johnson. The 8-vae's implicit prior. In Workshop Bayesian Deep Learning, NIPS, pp. 1-5, 2017.
Hyunjik Kim and Andriy Mnih. Disentangling by factorising. In International Conference on Machine Learning, pp. 2649-2658. PMLR, 2018.
Minyoung Kim, Yuting Wang, Pritish Sahu, and Vladimir Pavlovic, Relevance factor vae: Learning and identifying disentangled factors. arXiv preprint 4rXiv:1902.01568, 2019.
Diederik P. Kingma and Max Welling. Auto-encoding variational Bayes. In 2nd International Conference on Learning Representutions, ICLR, 2014.
B Ravi Kiran, Dilip Mathew Thomas, and Ranjith Parakkal. An overview of deep learning based methods for unsupervised and semi-supervised anomaly detection videos. Journal of Imaging, 4(2):36, 2018.
Alireza Makhzani and Brendan J Frey. Pixelgan autoencoders. Advances in Neural Information Processing Systems, 30, 2017.
10

---

Under review as a conference paper at ICLR 2023
Emile Mathieu, Tom Rainforth, Nana Siddharth, and Yee Whye Teh. Disentangling disentanglement in variational autoencoders. In International Conference on Machine Learning, pp. 1402-4412 PMLR. 2019.
Guansong Pang, Chunhua Shen, Longbing Cao, and Anton Van Den Hengel. Deep learning for anomaly detection: A review. ACM Computing Surveys (CSUR), 54(2):1-38, 2021.
Adrian Alan Pol, Victor Berger, Cecile Germain, Gianluca Cerminara, and Maurizio Pierini. Anomaly detection with conditional variational autoencoders. In 2019 18th IEEE international conference on machine learning and applications (ICMLA), pp. 1651-1657. IEEE, 2019.
Tim Sainburg, Leland McInnes, and Timothy Q. Gentner. Parametric UMAP Embeddings for Rep- resentation and Semisupervised Learning. Neural Computation, 33(11):2881-2907) 10 2021. ISSN 0899-7667. doi: 10.1162/neco. 01434.
a_01 01444
Milan Studený and Jirina Vejnarová. The multinformation function as a tool for measuring stochas- tic dependence. In Learning in graphical models, pp. 261-2977 Springer, 1998.
Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-MNIST novel image dataset for bench- marking machine learning algorithms. arXiv preprint arx1v:1708.00747, 2017.
Shengjia Zhao, Jiaming Song, and Stefano Ermon. Infovae: Information maximizing variational autoencoders. arXiv preprint arXiv: 170..0226,, 2017.

---

Under review as a conference paper at ICLR 2023
A
APPENDIX
Since we couldn't t accommodate all results in our main paper, in this section we present results produced from MNIST and FMNIST datasets. The results are categorized into three sections: Re- constructions A.I) Latent Representation (A.2), and 2D-Manifold embeddings A.3)
A.1
RECONSTRUCTION
dcvae
VAE
cvae
FactorVAE
betavae
RFVAE
0
4
4
10
10
10
10
10
20
20
20
20
20
20
20
20
(a) & = -152,
(b) E = 1811,
(c) 8 = -170
(d) 8 166.
(e) 3 -176,
(I) & -178
A = 0.88
A= 0.71
A 0.75
A 0.78
A = 0.76
A = 0.81
0
4
4
4
4
4
4
10
10
10
10
10
20
20
20
20
20
20
20
20
0
20
(g) 8 -106.
(h) & = -40,
(i) & = -136,
(i) 8 1332
(k)
-126.
-131.
0.23
0.48
=
A= 0.32
A
A 0.31
Figure Al: The reconstruction from the MNIST dataset shows similar negative ELBO and recon- struction error (AA) values for CVAE, V-VAE and RFVAE. our proposed model dCVAE performs best in terms of both reconstructing anomalous observation (first row) and normal observation (sec- ond row). We can observe trade-off in FactorVAB with respect to 3- VAE and RFVAE. FactorVAE performs better in reconstructing the anomalous observation whether the 3-VAE shows good per- formance in normal observations.
dcvae
VAE
cvae
FactorVAE
betavae
RFVAE
0
10
10
10
10
10
20
20
20
20
20
20
20
(a) 8 = -187
(b) = -198
(c) 8 -192
(d) = -221
(e) E = -231
(I) E =229
A = 0.94
A 0.79
A 0.82
A 0.90
A = 0.85
A = 0.92
10
10
10
20
20
20
20
20
20
20
20
20
20
20
0
20
(g) -1151
(h) 8 -170
(i) E =1622
(i) E -165
(k) -155
-166
0.122
A= 0.37
A 0.40
A
0
0.38 A = 0.42
Figure A2: Similar to the MNIST dataset, the FMNIST illustrates similar trade-off's among Factor- VAE, RFVAE, and 3-VAE. However, for some samples, 3-VAE mis-classifies the closely matched classes. dCVAE constrains the blurry reconstruction by enforcing conditions in the prior.
12

---

Under review as a conference paper at ICLR 2023
A.2
LATENT SPACE VISUALIZATION
drass
CVM
Classes
1.0
0.5
0.5
0.4
0.2
0.0
6.2
0.4
2.6
0.5
0.0
0.
0.6
0.6
0.8
FactoriVEE
beto-VAE
1.0
0.
0.8
0.6
0.2
0.0
0.2
0.4
0.6
0.8
0.8
5.0
0.2
0.4
0.6
0.0
Figure A3: Latent Space Representation (MNIST)
Cuat
1.0
1.0
0.6
0.6
3.7
0.0
0.2
0.4
0.6
0.5
3.8
0.2
0.6
0.8
beta VAE
1.0
1.0
0.8
0.6
0.4
0.3
0.6
0.8
3.8
0.2
0.4
0.6
0.8
Figure A4: Latent Space Representation (FMNIST)
13

---

Under review as a conference paper at ICLR 2023
A.3 LATENT MANIFOLD
0
5000000000
GO00000000
DOO000000
/
33333333333
555555557
755555555
66666660
9999999999
(a) dCVAE
(b) VAE
(c) CVAE
(d) FactorVAE
(c) B-VAB
(f) REVAE
Figure A5: Latent Embeddings (MNIST)
dCVAE
(b) VAE
(c) CVAE
(d) FactorVAE
(e) 3-V
REVAE
Figure A6: Latent Embeddings (FMNIST)
14

---

Under review as a conference paper at ICLR 2023
A.4
RANDOM GENERATION
betavae
betavae
betavae
betavae
20
20
betavae
betavae
betavae
betavae
betavae
10
10
10
20
20
betavae
belavae
betavae
belavee
belavae
10
10
20
20
20
20
20
20
20
20
20
betavae
betavae
betavae
betavae
20
20
betavae
betavae
betavae
betavae
betavae
1.0
10
10
20
20
20
20
20
cvae
cvae
cv3e
cvae
cval
10
10
10
20
20
cvae
cvae
cvae
ovae
cvae
10
10
20
20
20
20
20
20
20
20
20
20
20
cvae
cvae
cval
cvae
cvae
1.
10
10
20
20
15

---

Under review as a conference paper at ICLR 2023
5
dcvae
ocvae
dcvae
OCvae
dcvae
10
10
10
10
20
20
20
20
20
20
20
dcvae
dcvae
dcvae
dcvae
dcvae
20
20
20
20
20
dcvae
dcvae
dcvae
dcvae
2
2
5
10
10
dcvae
dcvae
dcvae
dcvee
dcvae
10
9
10
3
10
4
20
20
20
20
20
20
20
20
20
FactorVAE
FactorvAE
FactorVAE
FactorVAE
FactorVAE
10
10
10
20
20
20
20
20
20
20
FactorVAE
FactorvAE
FactorVAE
FactorVAE
FactorVAE
20
20
20
20
20
FactorVAE
FactorVAE
FactorVAE
FactorVAE
FactorVAE
10
10
10
20
20
20
20
20
20
20
16

---

Under review as a conference paper at ICLR 2023
RFVAE
RFVAE
REVAE
REVAE
RFVAE
10
10
10
20
20
20
20
20
20
20
REVAE
RFVAE
RFVAE
REVAE
REVAE
20
20
20
20
20
10
10
RFVAE
RFVAE
REVAE
RFVAE
RFVAE
10
10
10
20
20
20
20
20
20
20
20
betavae betavae_latent, hawpllltteeeeeeeeeeeeeeeee embedding
0
3
4
2
200
250
100
150
200
17

---

Under review as a conference paper at ICLR 2023
cvae www_jlannt_eemeeddie
200
250
100
150
200
latte_latetttaee eereeddng
0
0
0
0
0
1
1
1
1
1
1
1
2
2
2
2
2
2
2
2
2
2
3
3
3
3
3
3
3
3
3 4
3 4 5 6
100
4
4
4
4
4
4
4
4
150
5
5
5
5
5
5
5
5 6
6 6 6
6
6
6
6
6
200
7 7 7 7
7
7
7
7
7
7 8
8
8 8
8
8
8
8
8
8
9 9 9 9
9
9
9
9
9
9
50
100
150
200
250
18

---

Under review as a conference paper at ICLR 2023
200
250
100
150
200
RFVAE RFVAE_latent_ laabeddnttebbbddinggggggggg
50
100
150
200
50
100
150
200
250
19


Reviews:
[{"review_id": "z5wyi5nGOl", "reviewer": "Reviewer_Pbh4", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper presents disentangled conditional VAE for anomaly detection. The model and lower bound are based on a combination of total correlation and CVAE in the $\\beta$-VAE framework. The paper then conducts experiments on several grey-scale image datasets. Specifically, the paper uses reconstruction error to detect outliers, similar to some prior work. Results show a higher AUC compared to those baselines. ", "summary_of_the_review": "Overall I think the paper does not have much contribution to methodology or experiments. Therefore, I believe it needs further improvement.", "strength_and_weaknesses": "The paper has a good overview on related work, including derivations of those baseline methods and building blocks of the proposed method. However, the paper has the following major weakness. First, in the proposed method, every term is in fact the same as CorEx except that they are conditioned on $c$. I do not see any novel ideas or insights from the proposed method. Second, the experiments are only ran on simple datasets based on MNIST and its variants. The paper does not conduct experiments on more complicated datasets, especially those reported in corresponding papers of the baseline methods. Furthermore, there is only one metric (reconstruction error) that has been evaluated. However, simple metrics sometimes fail to detect anomaly (Choi et al 2021). ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "kgNLqq3qj7T", "reviewer": "Reviewer_8jzN", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a conditional VAE that uses total correlation to form their loss function. It builds off the CorEx work. They show many experimental results, some of which are promising. ", "summary_of_the_review": "Between the issues with clarity and the lack of comparison with the CorEx model in the experimental results, it is completely unclear whether there is significant improvement of the CorEx model. In general, this paper seems to rely heavily on previous work, with questionable amounts of novelty on its own. ", "strength_and_weaknesses": "\nStrengths: \n- The overall idea seems interesting\n- The results of Figure 3 and Table 1 look good\n\nMajor Weaknesses:\n- It seems like the primary contribution of this paper is to condition previous work with a class variable c. Everything from equation 12 onwards follows the same exact things presented in Section 2, except with a conditioning on c. Furthermore, nothing is described on why this particular architecture is relevant or useful for anomaly detection. \n\n- Why is CorEx not compared to in the experiments? Particularly because this work heavily depends on and builds off the model from CorEx. \n\nOther Weaknesses:\n\n- The notation is either inconsistent or extremely hard to follow. Making it difficult to review. Up until section 2.4 it seems that x is d dimensional and z is m dimensional and factorizable when conditioned on x. Although for some reason the index switches from i to j in the final sentence before equation 8. Do i and j represent different things? In section 2.4, j is used as the index and n is used to denote that there are N samples. Additionally k is used to denote the K factors v of z_j, which implies that each z_j has K factors v_k? Also what is total number of j (the sum and products are missing top values). In section 2.5, the j index is used, but now x is p dimensional and z is d dimensional? This confusion in notation makes it impossible to use the definition of TC in equation 7 with the decompositions in equation 13 to get equation 14. It also seems equation 8 is more useful in understanding the relationship between the TC for and the mutual information form of the loss function. \n\n- Equation 16 is derived using 13 and 15, not 14 and 15? \n\n- Below equation 7 you have I(x : z) instead of I(x ; z), Equation 13 and 14 are also using : instead of ;\n\n- The objective function in equation 18 has no regularization / penalty term? What does it have to do with the various \\beta or Factor models? \n\n- The equations take up a lot of space and could be organized better so that more detail about the equations is described above / below it. Some equations definitely do not need to take multiples lines and do. \n\n- There is a random l in \\hat{x} above equation 19. Also what are x and \\hat{x} paired or is the anomaly score over all x in the training set? This is not clear. \n\n- Despite the text, in Figures 1 and 2, all methods have higher A for the top row (anomalous) versus the bottom row, which according to bullet a) of better disentanglement, is desired. Also all the \\Epsilon values seems relatively close for all methods.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "XCUJEaV_hxQ", "reviewer": "Reviewer_DSrs", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this paper, the authors proposed a framework for unsupervised anomaly detection which builds on top of existing prior art on disentangled variational autoencoders (VAE's). Authors do a deep dive on existing prior art techniques such as $\\beta$ - VAE, Factor VAE and others by introducing a conditional variable to avoid loss of information and produce more accurate reconstructions. Authors present empirical results on several image datasets by demonstrating reconstruction score, accuracy and other important metrics for anomaly detection. ", "summary_of_the_review": "Unsupervised anomaly detection is an important problem, but new papers in this space need to be more novel as there is exists a significant body of work in this space (within both VAE's and GAN's). In addition, empirical evaluation should make sure the coverage is exhaustive to conclude if the results are significant. The current paper lacks most of these insights.", "strength_and_weaknesses": "**Strengths**\n\n- Paper is straightforward and relatively easy to follow.\n\n\n**Weakness**\n\n- The paper has this major flaw in its structure\na) Excessive emphasis on related work and prior art sections which is not necessary\n\n- For reasons not known, the authors have conveniently ignored most prominent unsupervised deep anomaly baselines such as \n  ### Bo Zong, Qi Song, Martin Renqiang Min, Wei Cheng, Cristian Lumezanu, Daeki Cho, and Haifeng Chen. 2018. Deep autoencoding gaussian mixture model for unsupervised anomaly detection. In International Conference on Learning Representations.\n\n ### Unsupervised Anomaly Detection with Adversarial Mirrored AutoEncoders. Somepelli et al. UAI 2021\n\n\n\n- I also believe that anomaly detection frameworks need to benchmarked with results on metrics such as AUPR and not just indicate AUC improvements.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "Mqagc5YPcNE", "reviewer": "Reviewer_HT5F", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper focuses on addressing unsupervised anomaly detection via disentangled conditional VAE.  The new architecture combines three core components: beta-VAE, CVAE and the principle of TC.  The authors claim that the new method improves the disentanglement of latent features, and the ability to detect anomalies.  Multiple experiments are used to demonstrate the performance and improvement, specifically from the perspective of capturing disentangled features. ", "summary_of_the_review": "The paper seems to be close to the original CVAE method, which is not very new.  Combining three core methods is fine but we did not understand why the new ensemble works better than before, and why one is more important.  The authors should provide an ablation study to investigate the sensitivity of each component. \n\nFor experiments, the major datasets are MNIST-based data, which is a naive and easy case. It would be more strong if the authors can apply the method for large datasets typically used in AD/OOD detection tasks, e.g., CIFAR-10/100 or ImageNet. \n\nIn addition, the authors may put more effort into improving the new methodology section with more details and explanations. The related works are good enough but should be compressed if the space is not enough. ", "strength_and_weaknesses": "- Strength\n\n1. Detail introduction about the related works, specifically the core techniques, i.e., beta-VAE, CVAE, and TC \n2. The idea is simple to follow and the paper is well-written. \n\n- Weaknesses \n\n1. The novelty is weak. \n2. Lack of ablation study on different components \n3. The methodology section is not well explained with enough details and theoretical proof\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject