Anomaly Detection Based on Unsupervised Disentangled Representation Learning in Combination with Manifold Learning
Authors: Xiaoyan Li
Iluju Kiringa
Tet Yeap
Xiaodan Zhu
Yifeng Li
Venue:  (2020)
Keywords: anomaly detection
disentangled representation learning
manifold learning

Abstract: Identifying anomalous samples from highly complex and unstructured data is a crucial but challenging task in a variety of intelligent systems. In this paper, we present a novel deep anomaly detection framework named AnoDM (standing for Anomaly detection based on unsupervised Disentangled representation learning and Manifold learning). The disentanglement learning is currently implemented by beta-VAE for automatically discovering interpretable factorized latent representations in a completely unsupervised manner. The manifold learning is realized by t-SNE for projecting the latent representations to a 2D map.  We define a new anomaly score function by combining beta-VAE's reconstruction error in the raw feature space and local density estimation in the t-SNE space. AnoDM was evaluated on both image and time-series data and achieved better results than models that use just one of the two measures and other deep learning methods.

Under review as a conference paper at ICLR 2020
ANOMALY DETECTION BASED ON UNSUPERVISED DISENTANGLED REPRESENTATION LEARNING IN COMBINATION WITH MANIFOLD LEARNING
Anonymous authors Paper under double-bling review
ABSTRACT
Identifying anomalous samples from highly complex and unstructured data is a crucial but challenging task in a variety of intelligent systems. In this paper, we present a novel deep anomaly detection framework named AnoDM (standing for Anomaly detection based on unsupervised Disentangled representation learning and Manifold learning). The disentanglement lcarning is currently implemented by 3-VAE for automatically discovering interpretable factorized latent represen- tations in a completely unsupervised manner. The manifold learning is realized by t-SNE for projecting the latent representations anomaly score function by combining 3- 3-VAE's reconstruction error in the raw feature space and local density estimation in the t-SNE space. AnoDM was eval- uated on both image and time-series data and achieved better results than models that use just one of the two measures and other deep learning methods.
2D map. We define a new
INTRODUCTION
Detecting anomalies in data flow of modern intelligent systems is an important but challenging problem. Formally speaking, anomaly detection problems can be statistically viewed identi- fying outliers having low probabilities from the modelling of data distribution p(x). Practically, since statistical modelling of the data is often difficult, it degenerates to domain description [Tax) & Duin 1999) or supervised prediction (Gornitz et al. 2013) problems in some cases. The exact explanation of an anomalous data point depends on the specific domain of focus. data centers, probably indicates an attempt of cyber intrusion. In recognition systems, it could be an adversarial attack. In biomedical information systems, it means possible onset of certain diseases. In Internet of Things (IoT) systems, it may represent hardware failure or alarming event captured by sensors. An anomalous sample is not always associated with negativity. Sometimes, it leads to novel discoveries in scientific explorations.
However, from the data analytics perspective, anomaly detection is difficult task due to the follow- ing reasons. (1) Many forms of data, e.g., images, text, and other types of sequences, often highly unstructured and complex. How can these data be well represented and high-level information be extracted by an algorithm? (2) The sample sizes of modern data sets are often extremely large and most of them are unlabelled. Unfortunately, traditional methods do not scale and perform well on these data. (3) When data of multiple modalities are naturally available for same events in a system, a robust and precise algorithm needs to be designed to integrate these information for system diag- nosis or decision making. (4) Many intelligent systems, such IoTs, require real-time detection and reaction of abnormal events to avoid costly and irrevocable damages. Thus, anomaly monitor- ing algorithms to be designed in these platforms must highly efficient. In summary, anomaly detection raises challenges in representability, scalability, multimodality, and time complexity.
Deep learning (LeCun et al. 2015) offers great potentials to overcome these challenges. (1) Rep- resentation learning mechanisms (such convolution for images, embedding for discrete symbols, and recurrence for time-series) have been developed in supervised and unsupervised deep models to consider the nature of specific types of input samples and encode them into vectors of contin- uous values as corresponding latent representations. using stochastic gradient descent that splits giant training set into mini-batches. Thus, learning be-
(2) Most deep learning models are trained

---

Under review as a conference paper at ICLR 2020
comes unrestricted and blessed by a large sample size. Particularly, stochastic variational inference (Hoffman et al.) 2013: Zhang et al. 2018 has successfully enabled scalable learning and inference for deep generative models (DGMs) a vast amount of unlablled data. (3) The development of
deep learning programming packages, such as PyTTorch (Paszke et al., 2017) and TensorFlow (Abadi ee al. 20T6), greatly eases the assembly of multiple network components (corresponding different modalities) together for multimodal representation learning (Li al. 2018b). (4) Once a deep model learned, the inference or encoding step is very efficient, thanks to the highly parallel computing architectures and techniques.
In some applications, if the domain of anomalous and normal samples is well defined, anomaly detection can be reduced to binary classification problems. However, in many situations, either the domain of anomalous samples cannot be fully understood or modelled, or the domain of the normal samples is too complicated to be modelled in one class. DGMs are more suitable than supervised methods in such cases. DGMs are concerned with the joint distribution of visible and latent variables with a hierarchy of stochastic (and deterministic) layers. With proper emphasis on disentanglement of latent representations, DGMs have the potential of dissecting hidden factors that are key to sample generation. Unsupervised disentangled representation learning (Bengio et al. 2013) renders several benefits. (1) It helps better understand our data, providing a path towards explainable AI. (2) It gives better control on the generation process of novel samples. (3) The disentanglement of latent factors may provide an opportunity to distinguish anomalies based on the landscape of latent space, which our interest in this paper. It has been shown that the likelihood of a data point p(z) estimated in DGM is not a reliable measure for detecting abnormal samples (Nalismick et al. 2019). Instead, reconstruction error is widely used as an anomaly score function (An & Cho 20155)
As a variant of variational autoencoder (VAE) (Kingma & Welling 2014), 3-VAE (Higgins et al.,
2017] is designed for unsupervised discovery of interpretable factorized latent representations from raw image data. An adjustable hyperparameter / is introduced to balance the extent of learning constraints (a limit on the capacity of the latent information channel and an emphasis on learning statistically independent latent factors) and reconstruction accuracy. It was demonstrated that 3-- VAE with appropriately tuned value of 3 (when 3 > 1) qualitatively outperforms VAE (when 3 = 1, 3-VAE is exactly VAE). Burgess et al. (2018) proposed a modification to the training regime of 3- VAE by progressively increasing the information capacity of the latent code during training. This modification facilitates the robust learning of disentangled representations in 3-VAE, without the previous trade-off in the reconstruction accuracy. Hoffman et al.] (2017) introduced a reformulation of (-VAE for 0 < 3 < 1. They argued that, within in this range, training 3-VAE is equivalent to optimizing an approximate log-marginal likelihood bound of VAE under an implicit prior.
Manifold learning is a family of nonlinear dimensionality reduction techniques. The t-distributed stochastic neighbor embedding (I-SNE) (van der Maaten & Hinton| 2008) is an unsupervised man- ifold learning method primarily used for data exploration and visualization by approximating high- dimensional data distribution using a two or ftree-dimensional map that could preserve local and certain global structures of the data. The use of t-SNE for anomaly detection has been sceptical vva der Maaten & Hinton| 2008). However, no comprehensive investigation has been made in this topic. Taking advantages of both disentangled representation learning (using 3-VAE as an imple- mentation) and low-di low-dimensional manifold learning (using t-SNE as an implementation), we propose a novel anomaly detection approach named AnoDM, standing for Anomaly detection based on unsu- pervised Disentangled representation learning and Manifold learning. We introduce a new anomaly score function by combining: (1) B- 3-VAE's reconstruction error, and (2) distances between latent representations of test points and training points in t-SNE map. AnoDM is general framework, thus any disentangled representation learning and manifold learning techniques can be applied. The choice of lower-level encoding scheme in 3-VAE depends on data type of interest. For image data, deterministic convolutional network (CNN) is used in the encoder. In case of time series (sequence) data, we design an improved version of 3- V-V by replacing CNN with temporal convolutional net- work (TCN) (Bai et al., 2018). a generic architecture for convolutional sequence prediction. in the encoder. We incorporate TCN as part of the encoder, because Bai et al. (2018) have shown that TCN outperforms canonical recurrent networks such as LSTMs (Hochreiter & Schmidhuber 1997) across a range of supervised learning tasks and recommended that CNN should be regarded as the first method to try for sequence modeling tasks. Regarding the decoding architecture, we simply choose CNN, because by choosing a simpler CNN architecture as a part of the decoder, the model can achieve a comparable even better performance but take much less running time.
2

---

Under review as a conference paper at ICLR 2020
The contributions of this paper summarized as follows. (1) We comprehensively explore the capacity of unsupervised disentangled representation learning, using 3- VAE as an implementation, for anomaly detection. (2) We thoroughly investigate the potential of manifold learning for outlier identification by taking the disentangled latent representations from 3-VAE as input to I-SNE. To the best of our knowledge, this is the first attempt to explore t-SNE for anomaly detection. (3) For sequence anomaly detection, instead of using prevailed recurrent networks (such as LSTM), as a practical contribution, we adopt an improved convolution architecture (TCN) capture the temporal dependency in the encoder in unsupervised way.
RELATED WORK
In the big data era, the development of deep learning models, especially DGMs, flourishes due to the need of modelling and amalyzing massive amount of unstructured data (such as images, time- series, graphs. text, etc.) generated in many application domains. Designing DGM-based solutions for anomaly detection becomes an important topic. Since DGMs, such as VAE (Kingma Welling 2014), and deep belief net (DBN) (Hinton et al. 2006 Li & Zhu 2018a), aim at modelling the joint distribution of visible and latent variables (that is p(æ, h)), their likelihood p(æ) by marginalizing out h. may serve as an abnormality indicator. However, unlike exponential family restricted Boltz- mann machines (cxp-RBMs) (Li & Zhu 2018b). exact likelihood is unavailable for most DGMs. Alternatively, reconstruction error serves as abnormality measure based on the intuition that out- of-distribution samples can be reconstructed badly (An & Cho 2015). Some deep hybrid methods (e.g. VAE+OCSVM (Andrews al. 2016) and DBN+OCSVM (Erfani al. (016)), successfully combine classical one-class support vector machine (OCSVM: or kernel-based support vector do- main description (SVDD)) with DGMs by using DGMs to learn latent representations of samples and using OCSVM to detect abnormal data points. However, these methods face the challenge of scalability, because the size of kernel matrices in dual form of SVDD quadratic of sample size.
Generative adversarial net (GAN) has also been applied for anomaly detection (Schlegl et al. a 2017)) Since there is no encoder in GAN, Deecke al. (2099) presented the ADGAN algorithm based on the availability of good representation of sample in latent space of its generator by assuming that the generator is able to effectively capture the distribution of the training data. [Li al. (2018a) proposed the GAN-AD method for cyber-physical systems (CPSs). It distinguishes fake data from actual data by taking into consideration of both discrimination loss calculated by the trained dis- criminator and residual loss between reconstructed and actual test data.
Furthermore, DGM-based algorithms are also devised to detect anomaly problem on sequence data (e.g. LSTM-VAE (Park et al. 2017) and GAN-AD (Li et al., (0118a). Conventionally, canonical recurrent networks (such as LSTM and GRUs (Cho et al.] (0144) are considered as the dedicated methods for sequence modeling. Some recent studies have also claimed that there was architec- ture that could consistently beat LSTM in some typical sequence modelling tasks (Jozefowicz et al.) 2015 Greff et al., 2015 Melis al.| 2017). On the other hand, some other researchers insist that CNN (LeCun et al. 1989) should be considered as more appropriate choice for sequences. Inspired by more recent CNN-based sequence modelling (such as machine translation (Gehring et al. [016]] 2017) and language modeling (Dauphin et al. 2016)), Bai et al. (2018) conducted a systematic eval- uation of generic convolutional and recurrent architectures for sequence modelling across broad range of tasks that are commonly used to benchmark recurrent networks, and concluded that con- volutional networks, rather than recurrent networks, should be respected as a "natural starting point for sequence modelling tasks".
3 METHOD
In this paper, we propose novel generic anomaly detection framework named AnoDM, which the first time combines unsupervised disentangled representation learning (implemented using 3-VAE as example) and low-dimensional manifold learning (currently using t-SNE as implementation) together to detect outliers via effectively taking the advantages of reconstruction at raw feature space and disentangled latent distribution in t-SNE map. F shows the architecture of AnoDM which includes two main phases: (1) unsupervised disentangled representation learning and (2) anomaly detector. After 3-VAE is learned using unlablled training normal samples, it then be employed
3

---

Under review as a conference paper at ICLR 2020
Training Sepur
y 9p(
(Wr (.r)lll' P..
Resomtracced training Irput
Me. + Nr
Samplesef vester Z..
Encoder
Decoder (CNN)
(CNN ICN)
Cer
Training model
Teeting
Pa
Input
C
ofmpies vector Ziv
Triized encodlr (CNN or TCN)
Bio
Trained decndes (CNN)
Zie
X/x
1n (AE space
Abanimalit store
L-N-Iodietance
3
Reconstrcctoo extor
Anomaly detector
Ri. tu pi..
Figure 1: Architecture of AnoDM implemented by 3-VAE and t-SNE for anomaly detection. First, 3-VAE learned using normal training data (upper part of the framework). Then is employed by the anomaly detector (lower part of the framework) to efficiently obtain latent encodings of training samples (or representative subset from the training set) and Meanwhile, t-SNE used to map the deterministic latent embeddings (Jurr training and Pic. for test samples) to the 2D space (we call it t-SNE space or map). such that the average distance between the 2D representation of a test sample and its & nearest neighbors from the 2D representations of training samples is calculated. Finally, the distance is combined with the reconstruction error of the test sample to define its anomaly score.
samples, and corresponding reconstructed versions using the decoder.
by the anomaly detector to efficiently obtain latent encoding and reconstructed version of a sample. Once latent embeddings of both training samples (or a representative subset from the training data) and a test sample are obtained, t-SNE used to map the latent representations of these samples fur- ther the 2-dimensional space (called t-SNE space map), such that the average distance between the 2D representation of the test sample and its l nearest neighbors from the 2D representations of training samples is calculated. Finally, this distance is combined with the reconstruction error of the test sample to define its anomaly score. The essential parts of this framework are discussed below in details. Full AnoDM approach is given in Algorithm D in Appendix B
3.1 UNSUPERVISED DISENTANGLED REPRESENTATION LEARNING
The unsupervised disentangled representation learning component in our architecture is imple- mented but not limited by 3-VAE (Higgins et al. 2017 Burgess al. 2018 Hoffman et al. 2017 Mathieu et al. 2019). A self-inclusive description of 3- V-E is provided in Appendix B The objec- tive function to be maximized for 3-VAE is defined as (Burgess et al. 2018):
- Egg ae) [log po (z] 2)] / D (go (+=x=))))l)<)) Cl.
The first term of this objective function corresponds to reconstruction error in the raw feature space. The KL divergence characterizes the discrepancy between approximate posterior and isotropic prior of latent representations. A small discrepancy between them indicates high disentanglemen of la- tent representations in independent variables. C is a hyperparameter which used improve the quality of reconstructed images. The loss function of the original 3 V-V proposed in (Higgins et al.) 2017) does not have this hyperparameter. The value of trades off reconstruction error and disen- tanglement, Unlike (Higgins et al.) 2017) and (Burgess et al., 2018). we consider 3 > 0 rather than just 3 > 1, because it is unnecessary to bound the value of 3 by 1, 3 > 0 allows us search for more appropriate disentanglement. The special case 3 = 0 could make the model learning very unstable, because the variance of inference distribution loses control. It worth highlighting that other unsuper- vised disentanglement models can be used as well in AnoDM. For example, Mathieu et al. (2019) interpreted disentanglement as decomposition instead independence by adding an additional reg- ularization term to reduce the discrepancy between the aggregate posterior and desired structured
4

---

Under review as a conference paper at ICLR 2020
prior. However, designing properly structured prior could be practically challenging. The adop- tion of 3- V-V in our framework is sufficient to prove the concept that unsupervised disentanglement helps anomaly detection.
3.2 EFFECTIVITY OF T-SNE ALGORITIMM
In addition 3-VAE's reconstruction error, we use the average distance between test sample and its k-nearest neighbors from the collection of training samples in the t-SNE map to score the outlierness of test sample. As discussed in Appendix [ t-SNE is significantly influenced by perplexity. The nature of complexity in data distributions makes it impossible to utilize a uniform criteria to define optimal perplexity for data. Moreover, Wattenberg et al.] (2016) mentioned several weaknesses of t-SNE, for examples, (1) it naturally expands dense clusters and contracts sparse ones, evening out cluster sizes, and (2) distances between clusters might not reflect global geometry. However, it is likely that k-nearest neighbors still work for local clumps, because, with proper value of perplexity, local topological information latent distributions can be preserved by the t-SNE plot. Thus, in t-SNE space, the measure of k-nearest neighbors is more suitable than full density estimation which is very sensitive with the sizes of clusters. Furthermore, as to be shown in Section 4 distancing in the 2D I-SNE map could be more robust than distancing in 9- 3-VAE's latent space where many non-determinative factors may influence the calculation of distances. Finally, worth clarifying that we do not directly learn a 2D latent representation from 3-VAE, because it will bottleneck too much the information flow for reconstruction. Instead, a lower-dimensional representation learned through t-SNE for density estimation only.
3.3 DETERMINISTIC OR STOCHASTIC LATENT REPRESENTATIONS FOR T-SNE
Fither the mean / or a sample from the approximate inference distribution (==(ær) can be passed to t-SNE to calculate the k-NN distance of test sample. There is trivial difference between perfor- mances achieved by these two methods in our framework. Generally, the u--asee method achieved slight better performance. The comparison of these two methods can be found in Table 2 in ap- pendices. Furthermore, from the latent representations' t-SNE maps (see Figure in appendices), one can interestingly see that when 3 is small (not overly large), the t-SNE maps for both methods are quite similar. As 3 becomes overly large, some normal classes can still form their own clusters (even though some similar classes, such as classes 3 and in MNIST, tend to mingle together) in μu u-based method, but in z x-based method all classes are entangled with each other. Same phenomena can be observed on the other datasets (see Figures 10 and appendices). Therefore, the pa-aaedd method is used in current design of AnoDM.
3.4 TCN ENCODER FOR UNSUPERVISED SEQUENCE MODELLING
Bai et al. 2018) distilled superior design in convolutional network into a simple architecture and referred as a temporal convolutional network (TCN) with two distinctive characteristics: (1) the convolutions in the architecture causal, and (2) the architecture can take sequence of any length and map it to an output vector of fixed length, just as with an RNN. Bai et al. (2018) also explained that TCNs capture significantly longer history than recurrent networks. Inspired by Bai et al. (2018), we replace CNN with TCN in the encoder of 3-VAE when evaluating the proposed AnoDM framework on time-series data, while still use CNN in the decoder, because our prelim- inary experiments demonstrated that keeping decoder as simpler CNN can help achieve comparable even better results, and take much less computing time. The architecture of TCN used in this pa- per is depicted in Figure 7 of appendices. In this TCN, we set kernel size to 4 and dilation factors to [1, 2, 4,8, 8, 16, 32). In Section 4 the comparison among TCN, CNN, and L.STM encoders in our framework also shows that TCN outperforms CNN and particularly LSTM to great extent for ECG signal anomaly detection.
3.5 ANOMALY SCORE FUNCTION IN ANODM
In the anomaly detector, the reconstruction error of a test sample in the original feature space and the average distance from its k-nearest-neighhor in training samples within the 2D t-SNE map
5

---

Under review as a conference paper at ICLR 2020
combined final anomaly score function:
SAVAE-INN (ie)) = aDRr (a +
(1- +1 a aisNEED (Tie)
(2)
where the first term is defined using normalized squared error (NSE):
ai x 11
(x - NSE an/))
(3)
Fi 12
where The is a test sample, and a/c is its reconstructed version by sending the stochastic latent encoding through the decoder of 3-VAE. The second term in Equation (2) is defined using 3-VAE's
deterministic latent encoding (mean from the encoder of learned 3-VAE) as input to t-SNE:
DIS (2te (1))
Σ JEN(LAA
(
1 112
(4)
K
te
where ( is the 2D representation of the i-th test sample in I-SNE map, N k) is the set of indices (f) nearest neighbors from training samples' 2D representations 4, t-SNE map. In Equation (2) E [o, 1] is the combination hyperparameter such that the two terms effectively complement each other. To allow the anomaly score function to achieve its full potential, a value should be sensitively searched, because the values of DRE and D small change of a value may dramatically alter the contributions these two terms. Alternatively, the distance score in Equation 4 can be normalized by average distance of training samples in t-SNE map, which may alleviate the magnitude difference, thus ease search of optimal value of o. The use of this normalized distance is investigated in Appendix
stay at different magnitudes, very
EXPERIMENTS
We evaluated our framework on four public image datasets, including MNIST (LeCun et al. 1998). Fashion-MNIST (iiao et al.) 2017). Small-Norb (LeCun et al. 2004) and CIFAR- Krizhevsky 2009), as well one collection of ECG heartbeat categorization time-series data named Arrhythmia Fazelli 2018). Brief descriptions of these data sets can be found Appendix G The detail of 3- VAE architecture given in Taile appendices. The number of epochs set to 20 for experiments MNIST and Mashion-MNIST datasets, and 50 for CIFAR-10, Small-Norb, and Arrhythmia datasets; batch size was set to 100 for experiments on all these datasets. When using t-SNE, the dimension of t-SNE map was set as 2 for all datasets, perplexity 30, the learning rate 200, and maximum number of iterations 1000. The value of a in the anomaly score function searched from set (0.0, 0.005, 0.01, 0.05, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 0.95, 0.99, 1.0]. We used k 1 for calculating neighbos distance in t-SNE map (we also tried set were similar).
5, the results
4.1 COMPARISON WITH CAPSNET, GANS, AND VAE
We compared AnoDM with state-of-the-arr algorithms including a supervised method CapsNet (Li al. 2019), and two types of generative models GANs (including AnoGAN and ADGAN) Deecke et al. 2019) and 3-VAE (implemented by setting a = thus using only reconstruction anomaly score). As shown in Table [ average, AnoDM achieved either comparable (on MNIST) or better (on Fashion-MNIST and CIFAR-10) performance in terms of receiver operating characteristic curve (auROC). On Fashion-MNIST, CapsNet perediition-prbbabbiity- best benchmark method, obtained an average auROC of 0.765, while AnoDM achieved 0.883. On MNIST, both AnoDM and CapsNet obtained the highest performance. However, CapsNet is a su- pervised method that takes advantages of class information, while ours is completely unsupervised that is more suitable in many practices as class information is either incomplete or unavailable. Fur- thermore, by comparing AnoDM with 3-VAE that only considers reconstruction error as anomaly score, AnoDM dramatically improved the performance all cases. In other words, t-SNE makes a prominent contribution to improve 3-VAE for anomaly detection problems. However, all generative models did not work well on Small-Nort Norb, mainly because these models used convolution to extract features from image, but convolution is only able capture translation but not other affine transfor- mations. Although CapsNet learns these transformations as a supervised method, it worth exploring unsupervised learning of affine transformations future topic.
as the

---

Under review as a conference paper at ICLR 2020
Table 1: Performance of AnoDM in comparison with other methods in terms of auROC image data. The results AnoDM were obtained by either the ju-based or z-based algorithm. reconstruction error is used as anomaly score. The results for AnoGAN and ADGAN were obtained from (Deecke et al. 2019). The results for CapsNet were obtained from (Li ct al. 2019).
column for 3-VAE, only
CapsNet
Dataset
Class
AnoGAN
ADGAN
B-VAE
AnoDM
PP-hased
RE-based
0
0.998
0.947
0.990
0.999
0.890
0.985
I
0.990
0,907
0.998
0.992
0.841
0.987
2 0,984
0,970
0.888
0.968
3
0.976
0,949
0.913
0.953
0.947
0.969
4 0.935
0.872
0.944
0.960
0.968
0.975
MNIST
5
0.970
0,966
0.912
0.955
0.966
0.976
6
0,942
0,909
0.925
0,980
0,907
0,983
7
0.987
0,934
0.964
0.950
0.899
0.977
8
0.993
0.929
0.883
0.959
0.946
0.982
0.990 9
0,871
0.958
0.965
0.794
0.928
avg.
0.977
0,925
0.937
0,913
0.975
0.620
0.454
0.500
0.844
1
0.851
0.871
0.860
0.978
2
0.818
0,486
0,459
0.783
3
0.895
0.693
0.730
0.886
0.394
0.379
0.763
Fashion-MNIST
5
0.691
0,982
0.985
0.990
6
0.801
0.480
0.501
0.713
7
0.619
0.787
0.842
0.952
8 0,912
0.885
0.876
0,980
0.656 9
0.754
0.701
0.944
uvg.
0.765
0,679
0,683
0.883
0 0.622
0.371
0.610
0.661
0.368
0.635
2 0.455
1
0.737
0.565
0.435
0,746
0.754
0.421
0.648
0.397
0.589
3 0.675
0,588
0.528
0.488
0.604
0.608
4 0.683
0,388
0.670
0.794
0.387
0.564
CIFAR-10
5
0.635
0.601
0.592
0.640
0.611
0.638
6 0.727
0.625
0,500
0.600
7
0.673
0.631
0.576
0.559
0.614
0.648
8
0,513
0.410
0.723
0.798
0.399
0.642
0.466 9
0.671
0.582
0.643
0.698
0.718
avg.
0.612
0.531
0.612
0.634
0.532
0.640
4.2 IMPACT OF BETA TO PERFORMANCE
As discussed in Higgins et al. (017)) , 3 (> 1) functions as controller encourage most efficient latent representation learning via limiting the capacity of latent information channel. Mathieu et al. (2019) however interpreted the objective of /3 tuning a proper level of overlap of encodings by working with another term that regularizes the divergence of the aggregate posterior (z(z) and the desired prior p(z). The main intuition is that purely increasing 3 induces too much overlap which actually discourages the disentanglement of data (information which is necessary for expressing de- sired structure is lost). Higgins et al. (017) demonstrated that 3- VAE with , > 1 leads to interesting variety of datasets. Surpris- ingly, our investigation demonstrates that by setting 0 < B < 1, it actually achieved the state-of-the- range of datasets, such MNIST, PaSSion-MNIST
results when learning interpretable factorized latent representations
art results for anomaly detection problems and Arrhythmia. Figure illustrates the impact of values of B3 and a in our anomaly score function. Interestingly, best performances were achieved when 3 < 1 and the performances generally degrade as 3 increases, resonating with Mathieu et al. (2019) that overly large values of 3 actually causes mismatch between (z) and p(z) (resulting in inappropriate level of overlap in the latent space). This phenomenon can be further seen in the t-SNE maps of latent embeddings in Figure 3 When / becomes extremely larger than the appropriate value, the anomalous class becomes entangled with its normal neighboring classes and the boundaries between normal classes become unclear. The- oretically, adding the divergence of the aggregate posterior (z) and the desired structured prior p(z) is an effective way to limit the level of overlap when 3 is too large. However, it is practically challenging to design an appropriate structured prior. Therefore, in our investigation, we focused on exploring the full range of I's value in B-VAE for the impact of disentanglement to anomaly
7

---

Under review as a conference paper at ICLR 2020
(a) MNIST, digit 7.
(b) Fashion-MNISTH MNIST." "Desss class.
(c) Small-Norb, "Human" class.
545
400
375
(d) CIFAR-10, "Dog" class.
(c) Arrhythmia, "F" class.
Figure 2: Performances (measured in terms of auROC) of AnoDM evaluated five datasets: MNIST, Fashion- MNIST, Small-Norb, CIFAR-10 and Arrhythmia. On cach dataset, the anomalous class is indicated in the corresponding subcaption while treating the rest classes as normal classes. CIFAR1O and Arrhythmia, because it makes learning highly unstable. As displayed in and missing values were indicated in white color at the top of corresponding heatmaps for 3 0.
that doesn' work
75
Clases
-33
-30
-00
-100
-33
-25
25
50
75
-66
-12
-36
22
60
+58
(a) = 0.01. 0.01.
(b) 8 = 0.3.
(c) 1.0.
(d) 8 2.0.
(e) 8 4.0.
10.0.
Figure 3: The impact of I's value to t-SNE map of latent representations of MNIST samples. Class is treated as anomalous class data. Each map displays 10000 data points identical in all maps including 5000 training data points and 5000 test data points.
detection. Since our framework is quite general, it can be easily extended to other unsupervised disentangled representation leaming models for anomaly detection.
8

---

Under review as a conference paper at ICLR 2020
4.3 IMPACT OF BETA TO T-SNE REPRESENTATIONS
The t-SNE plots in Figure 3 reflect the impact of 3's value on latent representations in the case of identifying anomalous digit from MNIST. In this example, the best performance (auROC=0.975) was achieved when 8 = 0.01. Clearly, as increases, all latent clusters become less dense, and more anomalous latent data points move to neighboring clusters. Furthermore, Figure also corroborates that, even though in t-SNE maps distances between clusters might not reflect global geometry and cluster sizes might not mirror the true sizes (Wattenberg et al. (016), using averaged distance from a test sample to its k nearest normal data points represented in t-SNE space to qualify outlierness still is a very effective way for distinguishing anomalous samples when | is tuned properly.
4.4 EVALUATION OF ANOMALY SCORE FUNCTION
In order to better evaluate our anomaly score function, formulated in Equation 2 a comprehensive comparison with methods only based on either distance in t-SNE map (D) or reconstruction-ers eerooo in raw feature space (DRz). To see the contribution of t-SNE, is also compared with the method that calculates nearest neighbor distance directly in latent space of 3- VAE. Figure 4 displays the ROC curves of these four approaches when assuming anomalous classes are respectively 1 ("Trouseerpantss", 3 ("Dress"). ("Sandal"), and 7 ("Sneaker") on Fashion-MNISS It is obvious that AnoDM achieves best results among them by taking advantages of both 3 VAE reconstruction and t-SNE embedding. The B- VAE reconstruction reflects whether useful information is captured by the model through recovering the input æ; the t-SNE embedding indicates the disentanglement of latent representations 2. Both measures effectively complement each other. Besides, comparing the auROCs between t-SNE-based and intent-distance based score functions, one can clearly see that the former dramatically outperforms the latter. Same conclusion can be drawn for MNIST, CIFAR-IO.
we conducted
and Arrhythmia, as displayed in Figures 12 the optimal values of are close to 1 (see Figure 2 due to magnitude difference rather than less usefulness of t-SNE, we replaced the distance score (Equation (4)) with normalized distance score in the weighted final anomaly score function (Equation 2 We found that the optimal values of shift to the lower end of the spectrum (see Figure 8). It implies that t-SNE does play a critical role our framework.
14
T6
17
and 18 in appendices. To further show that
10
Tratiee Bele
(a) Anomalous class:
(b) Anomalous class: (c) Anomalous class: 5.
(d) Anomalous class:
Figure 4; ROC curves of four methods on MaSHiOn-MNIST. These four examples illustrate the results of using four different anomaly score functions: t-SNE--Recon-emor-bbee (Dxz) and butent-distance-asse (calculating distances in latent space of 3 3-VAE). The a values for these four plots are 0.8, 0.8, 0.95, and 0.8, respectively.
(SyVAE+SNE <DRR + (1 a Di
SNE-based (D& Reconstruction-error
4.5 ANODM FOR TIME-SERIES
As mentioned in Section g our method uses a TCN encoder in 3-VAE for time-series anomaly detection. Figure didispay the comparison among TCN, CNN and LSTM encoders in the AnoDM framework on Arrhythmia. As a special case of LSTM-3-VMAR VAE, LSTM-VAE was presented (Park ct al. 2017 for state-of-the-art sequence modelling. For the five classes in Arrhythmia, iteratively one class was treated as anomalous class, while the other classes were used as normal classes. The TCN-encoder-bassee method outperforms the other two methods significantly in all five cases. Even though the CNN encoder achieved impressive results when detecting anomalous class "S", "V", "F" and "Q" respectively, it did not work quite well when class "N" was treated anomaly. One
9

---

Under review as a conference paper at ICLR 2020
possible reason might be that comparing with TCN and LSTM, the performance of CNN is more sensitive on the training sample size. Taking the above case as an example, as class ("N") accounts for over 80% of training data, when considering it as anomaly, normal training data hence become less sufficient for learning B-VAF. Nevertheless, last deterministic hidden layer before latent encoding at the bottleneck is calculated based on much longer se- quence dependency, such that less sensitive the limitation of small sample size. Conclusively, as men- tioned in (Bai et al. 2018), TCN should be regarded as natural starting point for sequence modeling tasks.
TCN-based (3-AF, each hidden unit of the
1000
As case study, Figure the original EGG signals and reconstructed signals by TCN-based 3-VAE when considering class "Q" as anomaly. Normal samples can be reconstructed very well, whereas abnormal samples suffer from larger reconstruction errors. Meanwhile, the corresponding t-SNE plot in Figure [b) displays two dis- tinctive clusters of abnormal samples. The combination of these two measures thus leads to the best performance as seen in Figure 6c
enoodar LSTi-encader
Anomalcus Classes
Figure 5: AuROCs on Arrhythmia.
bron
Sete
Trae 200000
Jicas
-50
sound Juao:. CAGL))
186
10
06 0.5 3.0
(a) Reconstructed signals by 3-VAE
(b) I-SNE plot.
(c) ROC plot.
Figure 6: Reconstructed signals, t-SNE map, and ROC curves Arthythmia with class "Q" as anomaly.
CONCLUSIONS
We propose a new methodology which successfully integrates t-SNE with disentangled representa- tion learning for anomaly detection. This approach achieved state-of-the-as performances on both image data (MNIST, Fashion-MNIST and (1FAR-10) and Arrhythmia time-series data. Specifically, best performance is accomplished when 0 < B for almost all cases involving 3-VAE. We also defined an anomaly score function by effectively taking the advantages of both low-dimensionall t-SNE embedding and 3-VAE reconstruction. Our algorithm demonstrated that t-SNE plays an es- sential role for measuring abnormality. This research initiates the research on anomaly detection using unsupervised disentangled representation learning and dimensional manifold learning. Besides. our model uses TCN network as encoding architecture for detecting anomalous time-series data and the experimental results convince us that TCN consistently outperforms CNN and LSTM. As a proof of concept, our current framework automatically inheres advantages of deep learning to address anomaly detection's 's issues representability and scalability as discussed in the begin- ning of this paper. The extension of our framework to multimodal data is straightforward. It is also possible that a neural t-SNE component could be designed and integrated into the learning of 3-VAE to achieve real-time efficiency. Other new well- well-performing manifold learning methods, such as UMAP (McInnes et al. 2018) which is faster and keeps global topologies, could be employed as replacement of I-SNE.
10

---

Under review as a conference paper at ICLR 2020
REFERENCES
Martin Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, Michael Isard, Manjunath Kudlur, Josh Levenberg, Rajat Monga, Sherry Moore, Derek G. Murray, Benoit Steiner, Paul Tucker, Vijay Vasudevan, Pete Warden, Martin Wicke, Yuan Yu, and Xiaoqiang Zheng. TensorFlow: A system for large- scale machine learning. In 12th USENIX Symposium on Operating Systems Design and Imple- mentation (OSDI 16), pp. 265-283, 2016. URL https: http://w.wuusss www..u.en.......m orq/ system/ Eiles/ http://coneeeeeeeseeeeeeeeeeeaee - pdf
Jinwon An and Sungzoon Cho. Variational autoencoder based anomaly detection using reconstruc- tion probability. Technical report, Data Mining Center, Seoul National University, Seoul, South Korea, 2015.
Jerone T. A. Andrews, Edward J. Morton, and Lewis D. Griffin. Detecting anomalous data using auto-encoders, International Journal of Machine Learning and Computing, 6(1):21-26, 2016.
Shaojie Bai, J. Zico Kolter, and Vladlen Koltun. An empirical evaluation of generic convolutional and recurrent networks for sequence modeling. ArXiv, pp. arXiv:1803.0127/V 2018.
Yoshua Bengio, Aaron Courville, and Pascal Vincent. Representation learning: A review and new perspectives. IEEE Transactions Pattern Analysis and Machine Intelligence, (88 2013.
Christopher P. Burgess, Irina Higgins, Arka Pal, Loic Matthey, Nick Watters, Guillaume Desjardins, and Alexander Lerchner. Understanding disentangling in 3-VAE, ArXiv, pp. 104.09910 April 2018.
Kyunghyun Cho, Bart van Merrienboer, Dzmitry Bahdanau, and Yoshua Bengio. On the properties of neural machine translation: Encoder-decoder approaches. ArXiv, pp. arXiv:1409.1259, 2014.
Yann N. Dauphin, Angela Fan, Michael Auli, and David Grangier. Language modeling with gated convolutional networks. ArXiv, pp. arXiv: www.:::612....
(el.611...8000 280083, 2016.
Lucas Deecke, Robert Vandermeulen, Lukas Ruff, Stephan Mandt, and Marius Kloft. Image anomaly detection with generative adversarial networks. In Machine Learning and Knowledge Discovery in Databases, pp. 3-17, Cham, 2019. Springer International Publishing.
Sarah M. Erfani, Sutharshan Rajasegarar, Shanika Karunasekera, and Christopher Leckie. High- dimensional and large-scale anomaly detection using linear one-class SVM with deep learning. Pattern Recognition, 58:121-134, 2016.
Shayan Fazeli. Eeg heartbeat categorization dataset, 2018. URL htl.httpss/// www wwwp:////w/ww..ggg...e.e com/ shayanfarzell/searbbatt Accessed: 2019-08-20.
Jonas Gehring, Michael Auli, David Grangier, and Yann N. Dauphin. A convolutional encoder model for neural machine translation. ArXiv, pp. 1rXiv:1611.02344, 2016.
Jonas Gehring, Michael Auli, David Grangier, Denis Yarats, and Yann N. Dauphin. Convolutional sequence sequence learning. In International Conference on Machine Learning, volume 70, pp. 1243-1252, 2017.
Nico Gornizz, Marius Kloft, Konrad Rieck, and Ulf Brefeld. Toward supervised anomaly detection. Journal of Artificial Intelligence Research, 43:235-262, 2013.
Klaus Greff, Rupesh Kumar Srivastava, Jan Koutník, Bas R. Steunebrink, and Jürgen Schmidhuber. LSTM: A search space odyssey. ArXiv, pp. 1xXix:1503.04065 2015.
Ryuhei Hamaguchi, Ken Sakurada, and Ryosuke Nakamura. Rare event detection using disentan- gled representation learning. ArXiv, pp. arXiv:1812.0122 2018.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. ArXiv, pp. 1512v:151..03385. 2015.

---

Under review as a conference paper at ICLR 2020
Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerehner. beta-VAE: Learning basic visual concepts with a constrained variational framework. International Conference on Learning Representations, 2(5): 6, April 2017.
Geoffrey E. Hinton and Sam Roweis. Stochastic neighbor embedding. In Advances in Neural Information Processing Systems 15, pp. 857-864, Cambridge, MA, USA, 2002. MIT Press.
Geoffrey E. Hinton, Simon Osindero, and Yee- Whye Teh. fast learning algorithm for deep belief nets. Neural Computation, 18:15 18:1527-1554, 2006.
Sepp Hochreiter and Jrgen Schmidhuber. Long short-term memory. Neural Computation, 9:1735- 80, December 1997.
Matthew D. Hoffman, David M. Blei, Chong Wang, and John Paisley. Stochastic variational infer- ence. Journal of Machine Learning Research, 14:1 14:1303-1347, 2013.
Matthew D. Hoffman, Carlos Riquelme, and Matthew J. Johnson. The 3-VAE's implicit priors. In Neural Information Processing Systems 2017 Workshop Bayesian Deep Learning, volume 2, 2017.
Rafal Jozefowicz, Wojeiech Zaremba, and Ilya Sutskever. An empirical exploration of recurrent network architectures. In International Conference on Machine Learning, pp. 2342-2350, 2015.
Mohammad Kachuee, Shayan Fazeli, and Majid Sarrafzadeh. ECG heartbeat classification: A deep transferable representation. ArXiv, pp. MrXiv:1805.00794 2018.
W. Karush. Minima of Functions of Several Variables with Inqqualities as Side Constraints. Masters thesis, Univ. of Chicago, Illinois, 1939.
Diederik P Kingma and Max Welling, Auto-encoding variational Bayes. In International Conference on Learning Representations, 2014.
Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, Depart- ment of Computer Science, University of Toronto, Toronto, Canada, 2009.
H. W. Kuhn and A. W. Tucke. Nonlinear programming. In Berkeley Symposium, pp. 41144881222 1951.
Yann LeCun, Bernhard E. Boser, John S. Denker, Donnie Henderson, Richard E. Howard, Wayne E. Hubbard, and Lawrence D. Jackel. Backpropagation applied to handwritten zip code recognition. Neural Computation, ((4):541-55) December 1989.
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(1) 86(11):2278-2322 1998.
Yann LeCun, Fu Jie Huang, and Léon Bottou. Learning methods for generic object recognition with invariance to pose and lighting. In IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2004.
Yann LeCun, Yoshua Bengio, and Geoffrey E. Hinton. Deep learning. Nature, 521:436 444, 2015.
Dan Li, Dacheng Chen, Jonathan Goh, and See-Kiong Ng. Anomaly detection with generative adversarial networks for multivariate time series. ArXiv, pp. arXiv:1809.04758, 201 8a.
Xiaoyan Li, Iluju Kiringa, Tet Yeap, Xiaodan Zhu, and Yifeng Li. Exploring deep anomaly detection methods based on capsule net. International Conference on Machine Learning 2019 Workshop Uncertainty and Robustness in Deep Learning, pp. arXiv:1907.06312. 2019.
Yifeng Li and Xiaodan Zhu. Exploring Helmholtz machine and deep belief net in the exponen- tial family perspective. In International Conference Machine Learning 2018 Workshop on Theoretical Foundations and Applications of Deep Generative Models, July 2018a.
Yifeng Li and Xiaodan Zhu. Exponential family restricted Boltzmann machines and annealed im- portance sampling. In International Joint Conference Neural Networks, pp. 39-48, July 2018b.
12

---

Under review as a conference paper at ICLR 2020
Yifeng Li, Fang- Xiang Wu, and Alioune Ngom. A review machine learning principles for multi- view biological data integration. Briefings in Bioinformatics, 19(2):325-33) 2018b.
Emile Mathieu, Tom Rainforth, N. Siddharth, and Yee Whye Teh. Disentangling disentanglement in variational autoencoders. In ICMI,, volume 97, pp. 4402-4412, Long Beach, California, USA, Jun 2019. PMLR.
Leland McInnes, John Healy, and James Melville. UMAP: Uniform manifold approximation and projection for dimension reduction. ArXiv, arXiv: 2018.
Gábor Melis, Chris Dyer, and Phil Blunsom. On the state of the art evaluation in neural language models. ArXiv, pp. urXivv:701 05w.0,558998 2017.
Eric Nalisnick, Akihiro Matsukawa, Yee Whye Teh, Dilan Gorur, and Balaji Lakshminarayanan. Do deep generative models know what they don't know? International Conference on Learning Representations, 2019.
Dachyung Park, Yuuna Hoshi, and Charles C. Kemp. A multimodal anomaly detector for robot- assisted feeding using an LSTM-based variational autoencoder. IEEE Robotics and Automation Letters, 3(3), 2017.
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang. Zachary Detito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in PyTorch. In Neural Information Processing Systems Autodiff Workshop, 2017.
Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic backpropagation and ap- proximate inference in deep generative models. In International Conference on Machine Learn- ing. pp. 1278 11111221--22211 1286, 2014.
Sara Sabour, Nicholas Frosst, and Geoffrey E. Hinton. Dynamic routing between capsules. In Neural Information Processing Systems, pp. 3856-3866, 2017.
Thomas Schlegl, Philipp Seebock, Sebastian M. Waldstein, Ursula Schmidt-Erfurth, and Georg Langs. Unsupervised anomaly detection with generative adversarial networks guide marker discovery. In International Conference on Information Processing in Medical Imaging, pp. 146- 157. Springer, 2017.
David M.J. Tax and Robert P.W. Duin. Support vector domain description. Pattern Recognition Letters, 20:1 190:1191-1999 1999.
Aäron van den Oord, Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol Vinyals, Alex Graves, Nal Kalchbrenner, Andrew W. Senior, and Koray Kavukeuoglu. WaveNet: generative model for raw audio. In Speech Synthesis Workshop. 2016.
Laurens van der Maaten and Geoffrey E. Hinton. Visualizing data using t-SNE. Journal of Machine Learning Research, 9:2579 - 2605, 2008.
Alexander Waibel, Toshiyuki Hanazawa, Geoffrey E. Hinton, Kiyohiro Shikano, and Kevin J. Lang- Readings in Speech Recognition. Morgan Kaufmann Publishers Inc., San Francisco, CA, USA, 1990.
Martin Wattenberg, Fernanda Vigas, and Ian Johnson. How to use I-NN effectively. Distill, 2016. doi: 10.23915/disttll.
URL UULL:tpp///dascll..pp//0////////
Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-MNIST: novel image dataset for bench- marking machine learning algorithms. ArXiv, pp. arXiv: 00117007777774 2017.
Cheng Zhang, Judith Butepage, Hedvig Kjellstrom, and Stephan Mandt. Advances in variational inference. IEEE Transactions on Pattern Analysis and Machine Intelligence, 41(8):2008 2026. 2018.
13

---

Under review as a conference paper at ICLR 2020
A VARIATIONAL LOWER BOUND AND VARIATIONAL AUTOENCODER
A directed generative model concerns the joint distribution (o 2) where is random vector variable representing an observation in d-dimension space, 2 is the vector of latent variables in k- dimensional space, and the set model hyperparameters. Inference po [=)x) =
p(a p[a|=)p(=] is pi
generally intractable, with few exceptions such as exponential family RBM (Li & Zhu 20186 An
easy parametric distribution 9s (= a:) introduced to approximate the intractable posterior po xz). Distribution yo( (z(z)) should be as close to pa (x) as possible by estimating inference parameters ø. divergence employed to measure with respect to Φ. After being expanded, the following formula is obtained (pe is fixed with respect to qa).
Based on mean-field variational Bayes, reverse Kullback-Leibbe
the distance between these two distributions, Xx (go (=
DKL. (4a (= (=|=)) [po (=(==) log pa (a) + DKL. (2o (()o())
(x (og po( [)]
(5)
After being rearranged, we can obtain objective with respect to both generative 0 and inference parameter Φ:
C(O, Φ) = log (o))) DxL (=(((=))))
(6)
E (t) flog po (ar 2) - log go (2|
(7)
Ex (=|a)) [lo po (x]=)] DxL (x
(8)
Since the KI. divergence in Equation (6) is always non-negative and thus L is the varitional lower bound (or evidence lower bound, ELBO) of (x)):
log (x) DEL go (2 x) (=))(()))) < log (x
Equation 88) implies that maximizing the ELBO is to maximize the reconstruction error and also
minimize the difference between the inference distribution and generative prior distribution. From the variational Bayes perspective, the aim is to obtain the optimal generative and inference parame- ters through maximizing the ELBO:
0, ② argmnax L.
(10)
0,6
The main challenge in learning of directed generative models is derive the gradient of Equation 77) with respect to the inference parameter. Therefore, the reparameterization trick introduced VAE (Kingma & Welling 2014| Rezende al. 2014) such that backpropagation can be used through expressing the random variable as deterministic variable = = To( e), where is an auxiliary independent random variable, and the transformation function Tę parameterized by converts to z. For example, when (z(x))) is a multivariate Gaussian with a diagonal covariance structure (=1(=) = (2 uu I)))), sampling from this distribution is equivalent to:
e (O, I).
(11)
m(x) (
(12)
where [f( (x))
(a( is deterministic two-head multilayer perceptrons parame- terized by 6, and @ denotes clement-wise product. The reparametrization trick enables Egng((z(a) (x))((z))]])))))))) [lo po (z, =) log {)(z(=)] Ee [logPe(x,. log (2((2x)))) such that the gradient
of Equation with respect to can be reformulated to
Ac(0, Φ)
i E Exng(zza)] [log pa (x, z) log qo (z(xa)
(13)
06
06
a Ee (x. z) log 20 (2(=))
(14)
db
0 pa (x, x) - log 4s (2|2)))
E.
(15)
db
O a(log Po pol (xt T (x t)) - log go (z(z)))
E,
(16)
14

---

Under review as a conference paper at ICLR 2020
The gradient with respect to generative parameters 0 is easier, because it does not rely the repa- rameterization trick:
ac(0, ())
i [g llg Pe (a, z) log (s [z]=)
(17)
80
00 O log po ( (xr 2) log go (2(2z)
Bz>g(=(a)
(18)
00
Olog po 2)
(19)
00
B BETA-VAE AND BEYOND FOR DISENTANGLED REPRESENTATION LEARNING
Higgins et al. (0177) proposed novel deep generative model, named 3-VAE, modification of VAE by introducing an adjustable hyperparameter 3 to learn an interpretable disentangled representation of the data generative latent factors. Specifically, / functions as controller to trade off between the extent of learning constraints and reconstruction accuracy. The constraints impose a limit on the capacity of the latent information channel and an emphasis on learning statistically independent latent factors. [Higgins et al.) (2017) demonstrated that 3- VAE with appropriately tuned /3 (3 > 1) qualitatively outperforms VAE (B 1) as well as state of the art unsupervised (InfOGAN) and semi- supervised (DC-IGN) approaches to disentangled factor learning on a variety of datasets (celebA, faces and chairs).
Higgins al. (2177) assumed that an image x is generated by the true world simulator using ground truth data generative factors: p(z v. w) Sim( w), where is set of conditionally independent factors and w set of conditionally dependent factors. Therefore, the joint distribution of the data æ and of this generative model is then to ensure that the inferred latent factors from go(2| (z)))) capture the generative factors in disentangled manner. The conditionally dependent data generative factors w can remain entangled in a separate subset of = that is not entangled with v Considering the prior p(z) is set be isotropic unit Gaussian p(z) _N( a constraint is introduced encourage the matching between 46 (z) 2)) and p(z) such that the disentangling property in the inferred g (2(2)) can be realized.
of generative latent factors is: p(==|z) p(æ/ u. w) = Sim w)) The aim
Following the same incentive as in VAE: maximizing the probability of generating real data, while minimizing the distance between the generative and approximate posterior distributions, as formu- lated below
max Exaxx (x 6
(20)
s.t. DKL. (e (go(z(x))lp(e) < <,
(21)
where X (x), 02..... ] is the training data set. Rewriting this equation as a Lagrangian under Karush 1939), the following equation F(0, (9,6,, (3 is
the KKT conditions (Kuhn & Tucke 1951 obtained:
F(0,0,
E (x (e) (log po (m|=)) - 3Dxa.
- 6)
= (x log po (x) 2)] - DKa. (go (z|=)) (p(=)) - ,35
2 (x [log po [])]]] - DK. (gol (z) (x) Ip(=)).
(22)
The objective function to be maximized in 3- 3-VAE is thus defined as:
(0 Φ) Eg [log pe (x: =)] 3D (go(z)))]lp(=))
(23)
where the Lagrangian multiplier 3 is the regularisation coefficient that constrains the capacity of the latent information channel x and puts implicit independence pressure on the learnt posterior due to the isotropic nature of the Gaussian prior z(z). When 3 1, 3 3-VAE corresponds to the original VAE formulation of Kingmna & Welling (2014). When > 1, applies stronger constraint which limits the capacity of z and encourages the model to leam the most efficient representation of the
15

---

Under review as a conference paper at ICLR 2020
data. Theoretically, a higher 3 encourages more efficient latent encoding and further encourages the disentangleemen. However, a higher / may lead to poorer reconstructions due to the loss of high frequency details when passing through a constrained latent bottleneck.
Burgess et al. (2018) proposed an improvement to 3-VAE, by progressively increasing the infor- mation capacity of the latent code during training. It facilitates the robust learning of disentangled representations in 3-VAE without the previous trade-off in reconstruction accuracy. The objective function to be maximized in 3-VAE is redefined as:
C3 Φ) (x)() [log =)]
- -C,,
(24)
where the hyperparameter 3 controls how heavily to penalise the deviation of DKL (1o(=|ar) [[(2)) and a controllable value C. By gradually increasing C' from zero to a large value, good-qualis quality reconstruction can be obtained.
Hoffmman et al. (017)) had also made a deeper research on B-VAE considering 3 < 1. They argued that optimizing this partially regularized ELBO is equivalent to performing variational expectation ((z))(())p(()))) that depends on the N - (()(=())) ) and further derived some approximations
maximization (EM) with an implicit prior r(2) (r(z)
marginal (aggregate) posterior q(z) & EN to examine this prior.
Mathieu et al. 2019) explained that most recent work for learning disentangled representations of data with deep generative models has focused on capturing purely independent factors of variation by employing regularizers explicitly encouraging independence in the representations. They argued that such approach is not generalisable, and quite restrictive for complex models where the true generative factors are not independent, very large in number, or where a set of true generative factors cannot be well defined. Overly large 9 is not universally beneficial for disentanglement, Since this in turn causes a mismatch between marginal posterior 4s (z) and the prior p(z). Thus they proposed generalization of disentanglement in VAE by explicitly separating such a decomposition as two tasks: a) the latent encoding of data should achieved an appropriate level of non-negligible overlap in aggregate encoding (=), and b) the aggregate encoding of data (=) should match the prior p(z) which demonstrates the desired dependency structure between latent variables. By improving the match of (z) and p(z). the overlap is only up an appropriate level. Mathicu al. (2019)
developed a new objective that incorporates both a) and b) by introducing an additional divergence term D (1>(2), p(z)). The objective of this improved B 3-VAE is defined as below:
La (x) = (=)() [log (=(x)))] - L (=|=)]lp(()))
D(ge(2),
(25)
By appropriately setting 33 and it allows direct control over the level of overlap and the regulariza- tion between the marginal posterior and the prior. However, a practical challenge of this method is how to define a proper structured prior when the structure of real hidden factors is poorly known. For this reason, our computational experiment in this paper is based on Burgess et al. 2018''s 3-VAF.
Instead of employing the regularization controller B to disentangle the independent data factors by putting implicit independence pressure in the representation, (Hamaguchi et al. (2018) presented a new method to learn disentangled representation by introducing two additional loss functions Lsim (a similarity loss function) and Last (an activation function). This technique aims to detect trivial events in an image resulting from environmental changes (such as illumination changes, background motions and shadows) by disentangling each image into two kinds of features, specific and common. The functionality of Lsin constrains common features to represent invariant factors between two paired images. Lus encourages activation of common features to avoid a trivial solution. After common features of paired images are separated, the means of common features from two images are fed into event detector (a classifier) for training. As mentioned in Hamaguchi et al. (2018), this method cannot achieve good disentanglement when dealing with complicated scenes, since the activation of the units in the common features are degenerated a certain value.
C T-SNE FOR MANIFOLD LEARNING
The t-distributed stochastic neighbor embedding (t-SNE), introduced by van der Maaten & Hinton 2008), is popular technique for nonlinear dimensionality reduction that is particularly well suited for visualizing similarity of high-dimensional data that lie on several different, but related, low- dimensional manifolds. It is improvement of stochastic neighbor embedding (SNE) presented
16

---

Under review as a conference paper at ICLR 2020
by Hinton & Roweis (2022) with easier optimization and better visualization via replacing the SNE cost function with a symmetrized version and using a Student-t distribution instead of a Gaussian to compute the similarity between two points in the low-dimensional space.
C.1 STOCHASTIC NEIGHBOR EMBEDDING (SNE)
SNE measures similarities (in both original high-dimensional space and mapped low-dimensional space) between points by converting Euclidean distances between data points into conditional proba- bilities under Gaussian distributions. In original high-dimensional space, for example, the similarity of a pair of data points z; and z, represented by Pyle under a Gaussian centered at z4. High Pale means and 2.j are close to each other, and vice versa. Pjjs is defined as:
expl-11a - xz 1 /2 Piji Ex exp( 1 Iz.
(26)
Ix /2/
where 0; is the variance of the Gaussian that is centered on data point 24. is influenced by the perplexity Perp(P:), which is defined as follows,
Perp( P.) = (
(27)
where P. corresponds particular o, and H (P::) is the Shannon entropy of P. measured in bits.
For the low-dimensional counterparts y; and of the high-dimensional data points x; and z., the similarity of these two points is denoted as qjji- and similarly defined as:
(28)
Exst exp(-lly: 1x
where the variance of the Gaussian is set to v² Since the focus is modeling pairwise similarities, both Paje and Qilk are set to zero.
C.2 COST FUNCTION OF SNE
Kullback-Leible divergence is utilized measure the difference between the low-dimensional data representation distribution Ozle and high-dimensional data distribution PyJó- The cost function of SNE, which is sum of Kullback-Leibler divergences over all data points, to be optimized by gra- dient descent method, is given by
C= - DKL ?
ΣΣ
Pyls log
Pili 43
(29)
Aiming to alleviate the difficult optimization problem of the cost function above and "crowding problem" (comparing nearby data points, moderate distant data points will not occupy reasonably large area in the low-dimensional map), yan der Maaten & Hinton (2008) introduced symmetric SNE and employed a Student t-distribution with one degree of freedom (which is the same as a Cauchy distribution) as the heavy-tailed distribution in the low-dimensional map. The cost function is redefined as:
C Dx (PI())
ΣΣ
Pty log
Pij Qij
(30)
where again, Pa and ga are also set to zero, and
exp(-11a; - 22,112//200
Pij = Σ kill exp( -1|2 - 22 2202)
(31)
(1 + || y (2) 21)
Qij Eke((1 (1 1l
m 1
(32)
The gradient of the Kullback-Leible divergence between P and the Student-t based joint probability distribution Q is given by
80
= 4 Σ (pij - 91) - y5 y5) + 11 - 3 -
(33)
6y,
17

---

Under review as a conference paper at ICLR 2020
D TEMPORAL CONTOTTTIN NETWORKS
Bai et al. 2018 employed a basic architecture which is essentially same as the time delay neural network proposed by Waibel et al. (1990) to ensure outputs of same length inputs and no leakage from the future into the past.
TCN ID FCN causal convolutions.
However, since a simple causal convolutions is not able to achieve long effective history size, Bai et al. (2018) employed dilated causal convolutions (same architecture as WaveNet (van den Oord ct al. 2016). please refer to Figure 7a that enable an exponentially large receptive field. The dilated convolution operation F on element of the sequence is defined as:
k-1
F(s) - ad f)()) = Σ
f(i) 2. -
(34)
F=
where a E RR is a 1-D sequence input, f : 0....., k- R is filter, d is the dilation factor, is the filter size, and d accounts for the direction of the past. When d = a dilated convolution reduces to regular convolution. By choosing larger filter sizes k and increasing the dilation factor the receptive field of TCN can be enlarged.
Bai et al. (2018) also utilized generic residual module (He et al. 2015) in place of convolutional layer. residual block defined as:
o Activation( F(x)).
The outputs of series transformations JF added the input æ of the block. To tackle with discrepancy of input and output widths in a standard residual block, an additional 1 x convolution used ensure that element wise addition receives tensors of the same shape (see Figure 7b
Bai et al. (2018) further discussed the advantages of TCN (including parallelism, flexible receptive field size, stable gradients, low memory requirement for training and variable length inputs) and its disadvantages (including possibly high memory requirement for evaluation and potential parameter change for transfer of domain).
E THE FULL ANODM ALGORITHM
Algorithm 1: AnoDM Algorithm
Result: Anomaly scores of test samples
Inputs: X,: training samples, X test samples, B > 0: hyperparameter for 3-VAE
while epoch more than training iterations do
Encoder net maps XT into and Ou.
Z = Mu o O E, [0, Z);
Decoder net reconstructs X to X using Z Update 3-VAE's parameters and Φ;
end
For Xxr and X1e, obtain frr and phe respectively using trained 3-VAE; Use t-SNE map frr and the to 2D representations and bii
for x within Xx do
DRE(ar (8) A NSF(2) 2
(4)
(111 || x
// reconstruction error
DI (2) (6) E EjeN(Le) (3)
(1) |1:/// ((i is the set of indices (0
& nearest neighbors from b.
SaVAEHSNT (1))) D'RR + a (SNE
// e 0011
13 end
18

---

Under review as a conference paper at ICLR 2020
A.A.u
Output
Hidden layer
Hidden layer
Hidden layer
Hidden layer
Hidden layer
Input
(a) A dilated causal convolution with dilation factors d [1, 1, 16, 32] and filter size
#
Idectity Map Coma)
X1
X3
Xx-1
(b) Residual block (A: = 4, d= 1).
Figure 7: Architecture of the temporal convolutional network (TCN).
F ANOMALY SCORES WITH NORMALIZED K-NN DISTANCE IN T-SNE MAPS
Since the normalised reconstruction enror in input space and the k-NN distance in t-SNE maps may have very different magnitudes (as mentioned Section 3.5 some values of a in Equation ( are close to (but not exactly equal to) 1. The reader may have the intuition that t-SNE is not useful in AnoDM. A simple way, to find out whether large a value due to magnitude difference or useless of t-SNE, is replace k-NN distance of test sample in t-SNE map (Equation (4) with k-NN distance normalized by average distance among training samples in the t-SNE map as defined below:
DI (3t DiSNE (Zu)
Ni (2
A
(35)
DISNE (Ft)
(4)
(7) x
1|2.
(36)
n
(1,
where e >> is normalization hyperparameter (we set it to 0.5 our experiment); Da is a training sample; n is the total number of training samples in t-SNE map. The heatmaps in Figure depict performances of AnoDM on Fashion-MNIST ("Dress" class considered as anomaly) and Arrhythmia ("F" class is treated anomaly) using normalized k--N distance in t-SNE maps in combination reconstruction error. By comparing Figure with Figure one can see that the optimal values of shift upper right corner area to the upper left corner area. It thus implies that, the optimal
19

---

Under review as a conference paper at ICLR 2020
value of is affected by the magnitude difference, and t-SNE indeed plays essential role in An
0.0
0.88
0.01
0.01
-080
0.05
0.05
0.1
0.1
0.2
0.84
0.4
0.4
0.80
0.5
0.6
0.6
0.7
0.8
0.9
0.76
0.9
1.0
1.5
155
3.0
3.0
0.66
40
0.68
7.0
7.0
10.0
10.0
0.60
00
0005
0.01
0.08
011
02
03
0.4
05
066
0.7
0.
0.
0.95
0.99
0.055
0.01
0.05
0.1
02
03
0.4
0.5
0.7
0.8
09
0.96
0.8
aphha
alpha
(a) Pashion-MNIST "Dress" class.
(b) Arrhythmia, "F"'class
Figure 8: Performances (measured in terms of auROC) of AnoDM evaluated on Fashion-MNIST and Arrhyth- mia when using normalized k-NN distance in t-SNE maps in combination with reconstruction error. On each dataset, the anomalous class is indicated in the corresponding subcaption while treating the rest classes normal classes.
G DESCRIPTIONS OF DATA SETS
MNIST: contains a training set of 60000 gray scale digit images of 28 x and test set of 10000 same resolution gray scale examples from approximately 500 different writers (LeCun et al. 1998).
Fashion-MNNIST: It is dataset of Zalando's article images, comprising 70000 MNIST- like labeled fashion images 28 28, with 7000 images per category (Xiao al. 2017), The training set has 60000 images and the test set has 10000 images. The samples come from 10 classes: T-shirt/top, trouser, pullover, dress, coat, sandal, shirt, sneaker, bag, and ankle boot.
Small-Norb: It contains 24300 96 grayscale images pairs of 50 toys belonging generic categories: four-legged animals, human figures, airplanes, trucks. and cars (LeCun) al. 2004). The objects were imaged by two cameras under 6 lighting conditions, 9 cleva- tions, and 18 azimuths. As in (Sabour et al. (017). the images were resized to 48 x 48; ran- dom 32x32 crops of them were obtained during training process. Central 32x 32 patches of test images were used during test.
CIFAR-10: consists of 60000 32 32 colour images in 10 classes, with 6000 images per class. There are 10000 test images which include exactly 1000 randomly- images from each class and 50000 training images (with 5000 images from each class) are randomly grouped into 5 batches (Krizhevsky 2009).
Arrhythmia: It is derived from Physionet's MIT-BIH Arrhythmia Datase which consists of ECG recordings from 47 different subjects recorded at the sampling rate of 360/l,. This dataset created by five different heart beat categories: "N", "S". "V", "F" and "Q", in accordance with Association for the Advancement of Medical Instrumentation (AAMI) EC57 standard (Kachuee al. 2018). The meanings of these categories are explained below.
1. "N" normal, left right bundle branch block, atrial escape and nodal escape.
20

---

Under review as a conference paper at ICLR 2020
2. "S": atrial premature, aberrant atrial premature, nodal premature and supra- premature.
3. "V": premature ventricular contraction and ventricular escape.
4. "F"; fusion of ventricular and normal.
5. "Q": paced, fusion of paced and normal and unclassifiable.
Table 2: AuROCs for both p-based and z z-based techniques.
4--aaeed
z-based
Dataset
Class
auROC
/
auROC
0
0.984
0.01
0.8
0,985
0.1
I
0.986
0.01
0.6
0.987
0.01
0.6
2
0.990
0.4
0.9
0.991
0.2
0.9
3
0.969
0.05
0.9
0.968
0.05
0.9
4 0.974
0.01
0.95
0.975
0.1
0.95
MNIST
5
0.975
0.01
0.95
0.976
0.01
0.95
6
0.983
0.01
0.8
0.980
0.01
0.8
7
0.975
0.01
0.9
0.977
0.01
0.9
0.928 8
0,980
0.01
0.9
0.982
0.01
0.9
0.4
0.8
0.925
0.05
0.8
avg.
0.974
0.975
0 0.844
0.05
0.2
0.840
0.05
0.0
I
0.977
0.01
0.8
0.978
0.01
0.8
2
0.783
0.01
0.0
0.783
0.05
0.0
0.886
0.01
0.8
0.884
0.01
088
0.760
0.1
0.0
0.763
0.3
Fashion-MNIST
0.990
0.2
0.95
0.990
0.95
0.713
0.05
0.709
0.952
0.01
0.8
0,950
0.01
0.8
0.980
0.05
0.8
0.980
0.05
0.8
0.940
0.3
0.8
0.944
0.3
0.7
avg.
0.883
0.882
0.635
0.4
0.0
0.634
0.752
0.6
0.99
0.754
0.99
0.589
0.558
0.608 4
10.0
0.95
0.606
10.0
0.99
0.564
0.01
0.0
0.563
0.0
CIFAR-10
5
0.638
0.4
0.95
0.627
0.7
0.95
6
0.600
0.3
0.0
0.590
0.7
0.0
7
0.648
0.1
0.95
0.644
0.01
0.95
8 0.642
0.3
0.0
0.624
1.2
0.0
0.717 9
1.0
0.95
0.718
0.01
0.95
0.639
0.632
0 0.512
0.1
0.0
0.520
7.0
0.0
1
0.656
0.01
0.0
0.647
0.01
0.0
2
0.771
0.05
1.0
0.771
0.05
1.0
Small-Norb
3
0.581
10.0
1.0
0.581
10.0
1.0
0.564
0.5
1.0
0.564
0.5
1.0
0.617
0.617
0
0.911
0.05
0.95
0.906
0.01
0.95
I
0.924
0.01
0.95
0.925
0.01
0.95
ECG (Arrhythmia)
2
0.970
0.01
0.99
0.970
0.01
0.99
3
0.905
0.01
0.95
0.910
0.01
0.95
0.991 4
0.01
0.99
0.991
0.01
0.99
avg.
0.940
0.940
21

---

Under review as a conference paper at ICLR 2020
(a) B = 0.01 (Hı-based)
(b) | = 0.8 (fu-based)
(c) 5.0 (u-
(d) / 10.0 (f- baseds
(e) 0.01 (z-hased)
(f) 3 = 0.8 (s-based)
(g)
(z-based)
(h) 3 = 10.0 (z-hased)
Figure 9: Comparison of ju-based method or z-based method on MNIST data (anomalous class is 3) for inferring latent representations that visualized in I-SNE map.
(a) B 0.01 based)
(b) 8 = 0.8 (μi- (u-based)
(c) 5.0 (çu-based)
(d) 3 10.0 (aseda
(e) 0.01 (z-based)
(f) 3 0.8 (z-based)
(g) (z-based)
(h) 9 = 10.0 (z-based)
Figure 10: Comparison of p-sae method or z-based method on Fashion-MNIST data (anomalous class is ("Trouser/pants") for inferring latent representations that visualized in t-NE space.
22

---

Under review as a conference paper at ICLR 2020
(a) B = 0.01 (f1-based)
(b) | = 0.8 (fu-based)
(c) 3 = 5.0 (f--based)
(d) / = 10.0 (u--based
(e) 8 0.01 (z b-sddddd
(1) 3 = 0.8 (z based)
(g) 8 5.0 (z-based)
(h) 3 = 10.0 (z-based)
Figure 11: Comparison of j-based method z-based method on Arrhythmia time-series data (anomalous
class is 4 ("Q")) for inferring latent representations that are visualized in t-SNE space.
190 Mite-aa bubD 1200 sell
190 Maun beai SUAD 1200 bell
(a) digit 0
(b) digit
(c) digit 2
(d) digit 3
1996 and bMMCC- 63020
seree 1909 bellC BURCH
(c) digit 1
(f) digit 5
(g) digit 6
(h) digit 7
m
ARTO
sted cali: (000
(i) digit 8
(i) digit 9
Figure 12: ROC plots on MNIST (fı-based).
23

---

Under review as a conference paper at ICLR 2020
(R0H seceest JSC 3010
1900 Moun perteae 1300 REC
(a) "T-shirt/top"
(b) ""Trouser/pants"
(c) "Pullover shirt"
(d) "Dress"
190h and balli RMCX
redd bulty 8/104
1909 bello: -0000
(e) "Coat"
(f) "Sandal"
(g) "Shirt"
(h) "Sneaker"
L800
shad +8/000
64
(i) "Bag"
"Ankle boot"
Figure 13: ROC plots on MaSHion-MNIST (H--based).
24

---

Under review as a conference paper at ICLR 2020
43
and buthh 10400
2010)
(a) "Airplane"
(b) "Automobile"
(c) "Bird"
(d) "Cat"
(490) tenteel 20mn
LETT und
und bull" 3000 aud
SUDC- Hal sell (A0N
roathee
C4
30
64 (1) "Dog"
64
(e) "Deer"
(g) "Frog"
(b) "Horse" "
cald bulls, AMCC)
buft MMCh
M4
(i) "Ship"
"Truck"
Figure 14: AuROC plots on CIFAR-10 (H--based).
(1000) 32 Fdbhhutteee
cald bults AM00
120 belO RDCA
Lifee
X2 14
(a) "Four-legged ani- mals"
(b) "Human figures"
(c) "Airplanes"
(d) "Trucks"
Beter cald LABEE-
(e) "Cars"
Figure 15: ROC plots Small-Norb (fs-based).
25

---

Under review as a conference paper at ICLR 2020
0
MBC (IN)
"N"
(b) "S"
(c) "V
(d) "F"
(e) "Q"
Figure 16: ROC plots of Arrhythmia using CNN-encoder (μ-baaeed).
Palu 01001
"N"
(b) "S"
(c) "V"
(d) "F"
(c) "Q"
Figure 17: ROC plots of Arrhythmia using LSTM-encoder (fu-based).
26

---

Under review as a conference paper at ICLR 2020
460 and bafiC (A71)
400 befiC AMMM
(a) "N"
(b) "S"
(c) "V"
(d) "F"
10
Sour CLBCC 199 Juhii C.
Fele Postiee Fax
(c) "Q"
Figure 18: ROC plots of ECG using (CN-encoder (j--based).
Table 3: Architecture the 3-VAE used in AnoDM.
Dataset
Optimizer
Architecture
Input
784 (flattened 28 28 1).
Conv2d (filters 32, kernel size 3, strides 2), Conv2d (filters 32, kernel size 3, strides 2), PC (128), FC (128). ReLU activation.
Enender
MNIST
Adam (le-3)
FC (128), FC (1568), Deconv2d (filters 32, kernel 3, strides Decony2d (filters 1. kernel size 3, strides 2). ReLU activation.
Decoder
Input
784 (flattened 28 28
Conv2d (filters 32, kernel size strides 2), Conv2d (filters 32, kemel_size strides 2), PC (128), FC (128). ReLU activation.
Encoder
Fashion-MNIST
Adam (le-3)
FC (128), (1568), Deconv2d (filters kernel 3, strides Deconv2d (filters 1, kernelsize 3, strides ReLU activation.
Decoder
Input
3072 (fattened 32
3).
Conv2d (filters 64, kernel size strides 2), Conv2d (filters 128, kernel _size z strides 2), BN, Conv2d (filters kernel strides 2), BN, FC (256), Dropout (0.9), BN, FC (256), Dropout (0.9). BN. ReLU activation.
CIFAR-10
Adam (le
Encoder
FC (256), BN, (256), BN, (2048). BN. Deconv2d (filters 128, kerne_sizee strides 2), BN, Deconv2d (filters 64, kezn 5,
Decoder
strides 2), BN, Deconv2d (filters 3, kemel size 3, strides 2). ReL.U activation.
Input
1024 (flattened 32 x 32 1).
Conv2d (filters 32, kernel size strides 2), BN, Conv2d (filters 64, kemel ssize strides 2), BN, Conv2d (filters 128, kernel.size strides 2),BN, FC (256), Dropout (0.7), BN, PC (256), Dropout (0.7). BN. ReL., activation.
Small-Norb
Adam (le (1e-3)
Encoder
FC (256), BN, FC (256), BN, FC (2048), BN, Deconv2d (filters 128. kernel size 5, strides 2), Deconv2d 2d (filters 64, kerne_size 5, strides 2),
Decoder
Deconv2d (filters 3, kernel size 3, strides 2). ReLU activation.
Input
187.
TCN (filters 64, kemel size stacks dilations 2,, 16, 32), padding "causal", use aesesskip.com -kip.connettions "True", dropout_rate 0.05)
Encoder
Arrhythmia
Adam (le (1
FC (256), FC (384), FC (2048), BN, Deconv I (filters 64,
kerne_siize 3, strides 2), Deconv2d (filters 64, kernelsiiee 3. strides 2),
Dccony2d (filters 64, kernel size 3, strides 2), Docony2d (fitters 32,
Decoder
kemne size 3, strides 2), Deconv 2d (filters 32, kernel size 3, strides 2).
Deconv21 (filters I, kernel size 3, strides 2). ReI.U activation.
27


Reviews:
[{"review_id": "r1xQXHqitr", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents a novel deep anomaly detection model. It combines two existing models: B-VAE and t-SNE. The B-VAE is trained unsupervised and learns an encoder and decoder which provide both an embedding and a reconstruction. Using t-SNE to reduce its dimensionality, the embedding is projected into a 2 dimensional space. An anomaly score function is defined that combines the reconstruction error and the distance in t-SNE space to the K nearest neighbor(s). Experiments are conducted with several image datasets (MNIST,FMNIST,CIFAR10,SmallNORB) and one timeseries dataset (Arrhythmia). For the image sets, the B-VAE model is implemented with a CNN, while for timeseries, a TCN is used. Comparisons are conducted showing the approach to be beat other SOT unsupervised methods, AnoGAN and ADGAN, by 63% and 22% respectively for MNIST and 8% and 2% for FMNIST (in terms of error reduction). For CIFAR-10 and FMNIST it is even demonstrated to beat a supervised SOT method CapsNET. Another experiment shows that t_SNE dramatically improves the performance over B-VAE alone. For the timeseries, the approach is not compared to other SOT approaches as the authors only provide an experiment showing that TCN beats CNN and LSTM for the implementation of the B-VAE. In addition the authors study the effect of the various parameters of the system, in particular the effect of the B in B-VAE and of alpha, the mixing factor between reconstruction error and kNN distance in t_SNE. 3D plots give a good idea on how to select optimal values for the various datasets. The impact of B is also shown on the t-SNE map for MNIST. Finally an ablation studies compares on MNIST the performance of the approach with t-SNE alone, reconstruction alone, and latent distance. On average over 4 digits taken as anomaly, the proposed approach dramatically outperforms the others.\n\nPROS:\n\n* The proposed approach improves over the SOT of competitive recent methods for anomaly detection on four image datasets.\n* The authors make an effort to abstract the approach into a framework where other deep learning models and dimensionality reduction techniques can be used. They illustrate this by using a TCN instead of a CNN for the timeseries example.\n* The parameter studies and ablation studies are informative and answer many of the questions i had as i read the paper.\n* The paper is relatively clearly written (at least sufficiently to easily understand the technical details).\n\n\nCONS:\n\n* The novelty of the paper is limited as it is mostly a combination of 2 existing methods.\n* The timeseries dataset is not compared to SOT methods (although the authors claim SOT in the conclusion).\n* A pseudo-code algorithm is not provided, making it unlikely someone can reproduce the method.\n* There are many typos an grammatical errors\n* The paper could have been shortened. 10 pages is too long.\n\nOverall, because of the good performance and thoughtful ablation studies, and despite the limited novelty, I think the paper makes a good contribution to anomaly detection.\n\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}, {"review_id": "HkeKsEc7tr", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes to combine beta-VAE and t-SNE for anomaly detection.\nAlthough the problem and the proposed approach is relevant, I have the following concerns.\n\n- The problem setting is not well explained.\n    In particular, it is not clear whether the setting is unsupervised or not.\n    It seems that the proposed method is for unsupervised anomaly detection.\n    However, the authors mention that beta-VAE is trained on normal data, which means that it is not unsupervised.\n    Please clarify this point.\n- The originality and the technical quality of the proposed method is not high as it is a straightforward combination of two existing method.\n    If the proposed combination has some theoretical advantage for anomaly detection, the paper becomes more interesting.\n    However, there is no theoretical analysis of the proposed method, hence the significance of the contribution is hot high in its current state.\n- Experimental results are not convincing.\n    * The authors argue that the t-SNE step is important as the proposed method is better than the naive beta-VAE.\n      However, Figure 2 shows that in most cases the score becomes better as \\alpha gets larger and shows the best score if alpha = 1.\n      From Equation (2), this means that the t-SNE step does not contribute to the anomaly detection performance.\n      This inconsistency should be carefully discussed.\n    * Also, Figure 2 shows that beta should be small and the best score is achieved when beta = 0 in most cases.\n      This means that representation learning is not meaningful and the raw representation (feature vectors) may be already effective for anomaly detection.\n      Hence the significance of the proposed method is not convincing.\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "BygqC2QXKB", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In a paper a new way to compute anomality score (for a test point) is suggested. A paper is purely experimental, based on existing techniques to dimension reduction (beta-VAE and t-SNE). Given trained beta-VAE, latent vectors, obtained for training set, are feed into t-SNE algorithm. The overall anomality score for a test point is combined from 1-NN distances on t-SNE plot and reconstruction error of beta-VNE. \n\nThere is substantive question naturally appears from the application of t-SNE to obtained latent vectors. By construction, beta-VAE tries to make latent vector to be distributed according to N(0,I). By definition, it is very hard to project such a distribution on a plane, even by non-linear methods, such as t-SNE. Yet at the second step of paper's approach, this vectors are feed into 2-dimensional t-SNE. \n\nThis aspect makes me to think that optimal alpha in paper's anomality score should be close to 1. This would imply that t-SNE step is not needed at all. So, I am curious what actual optimal alpha was in experiments? \n\nOr, how would results change if t-SNE mapping would be set to Identity tranformation (into space whose dimension is the same as latent space), but formula for anomality remains the same?", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I made a quick assessment of this paper."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper presents AnoDM (Anomaly detection based on unsupervised Disentangled representation learning and Manifold learning) that combine beta-VAE and t-SNE for anomaly detection. Experiment results on both image and time series data are shown to demonstrate the effectiveness of the proposed solution. 

The paper aims to attack a challenging problem. The proposed solution is reasonable. The authors did a job at addressing some of the concerns raised in the reviews. However, two major concerns remain: (1) the novelty in the proposed model (a combination of two existing models) is not clear; (2) the experiment results are not fully convincing. While theoretical analysis is not a must for all models, it would be useful to conduct thorough experiments to fully understand how the model works, which is missing in the current version. 

Given the two reasons above, the paper did not attract enough enthusiasm from the reviewers during the discussion. We hope the reviews can help improve the paper for a better publication in the future. 




Decision: Reject