Anomaly Detection Based on Unsupervised Disentangled Representation Learning in Combination with Manifold Learning

1 INTRODUCTION . Detecting anomalies in data flow of modern intelligent systems is an important but challenging problem . Formally speaking , anomaly detection problems can be statistically viewed as identifying outliers having low probabilities from the modelling of data distribution p ( x ) . Practically , since statistical modelling of the data is often difficult , it degenerates to domain description ( Tax & Duin , 1999 ) or supervised prediction ( Gornitz et al. , 2013 ) problems in some cases . The exact explanation of an anomalous data point depends on the specific domain of focus . In data centers , it probably indicates an attempt of cyber intrusion . In recognition systems , it could be an adversarial attack . In biomedical information systems , it means possible onset of certain diseases . In Internet of Things ( IoT ) systems , it may represent a hardware failure or alarming event captured by sensors . An anomalous sample is not always associated with negativity . Sometimes , it leads to novel discoveries in scientific explorations . However , from the data analytics perspective , anomaly detection is a difficult task due to the following reasons . ( 1 ) Many forms of data , e.g. , images , text , and other types of sequences , are often highly unstructured and complex . How can these data be well represented and high-level information be extracted by an algorithm ? ( 2 ) The sample sizes of modern data sets are often extremely large and most of them are unlabelled . Unfortunately , traditional methods do not scale and perform well on these data . ( 3 ) When data of multiple modalities are naturally available for same events in a system , a robust and precise algorithm needs to be designed to integrate these information for system diagnosis or decision making . ( 4 ) Many intelligent systems , such as IoTs , require real-time detection and reaction of abnormal events to avoid costly and irrevocable damages . Thus , anomaly monitoring algorithms to be designed in these platforms must be highly efficient . In summary , anomaly detection raises challenges in representability , scalability , multimodality , and time complexity . Deep learning ( LeCun et al. , 2015 ) offers great potentials to overcome these challenges . ( 1 ) Representation learning mechanisms ( such as convolution for images , embedding for discrete symbols , and recurrence for time-series ) have been developed in supervised and unsupervised deep models to consider the nature of specific types of input samples and encode them into vectors of continuous values as corresponding latent representations . ( 2 ) Most deep learning models are trained using stochastic gradient descent that splits a giant training set into mini-batches . Thus , learning be- comes unrestricted and blessed by a large sample size . Particularly , stochastic variational inference ( Hoffman et al. , 2013 ; Zhang et al. , 2018 ) has successfully enabled scalable learning and inference for deep generative models ( DGMs ) on a vast amount of unlablled data . ( 3 ) The development of deep learning programming packages , such as PyTorch ( Paszke et al. , 2017 ) and TensorFlow ( Abadi et al. , 2016 ) , greatly eases the assembly of multiple network components ( corresponding to different modalities ) together for multimodal representation learning ( Li et al. , 2018b ) . ( 4 ) Once a deep model is learned , the inference or encoding step is very efficient , thanks to the highly parallel computing architectures and techniques . In some applications , if the domain of anomalous and normal samples is well defined , anomaly detection can be reduced to binary classification problems . However , in many situations , either the domain of anomalous samples can not be fully understood or modelled , or the domain of the normal samples is too complicated to be modelled in one class . DGMs are more suitable than supervised methods in such cases . DGMs are concerned with the joint distribution of visible and latent variables with a hierarchy of stochastic ( and deterministic ) layers . With proper emphasis on disentanglement of latent representations , DGMs have the potential of dissecting hidden factors that are key to sample generation . Unsupervised disentangled representation learning ( Bengio et al. , 2013 ) renders several benefits . ( 1 ) It helps better understand our data , providing a path towards explainable AI . ( 2 ) It gives a better control on the generation process of novel samples . ( 3 ) The disentanglement of latent factors may provide an opportunity to distinguish anomalies based on the landscape of latent space , which is our interest in this paper . It has been shown that the likelihood of a data point p ( x ) estimated in DGM is not a reliable measure for detecting abnormal samples ( Nalisnick et al. , 2019 ) . Instead , reconstruction error is widely used as an anomaly score function ( An & Cho , 2015 ) . As a variant of variational autoencoder ( VAE ) ( Kingma & Welling , 2014 ) , β-VAE ( Higgins et al. , 2017 ) is designed for unsupervised discovery of interpretable factorized latent representations from raw image data . An adjustable hyperparameter β is introduced to balance the extent of learning constraints ( a limit on the capacity of the latent information channel and an emphasis on learning statistically independent latent factors ) and reconstruction accuracy . It was demonstrated that βVAE with appropriately tuned value of β ( when β > 1 ) qualitatively outperforms VAE ( when β = 1 , β-VAE is exactly VAE ) . Burgess et al . ( 2018 ) proposed a modification to the training regime of βVAE by progressively increasing the information capacity of the latent code during training . This modification facilitates the robust learning of disentangled representations in β-VAE , without the previous trade-off in the reconstruction accuracy . Hoffman et al . ( 2017 ) introduced a reformulation of β-VAE for 0 < β < 1 . They argued that , within in this range , training β-VAE is equivalent to optimizing an approximate log-marginal likelihood bound of VAE under an implicit prior . Manifold learning is a family of nonlinear dimensionality reduction techniques . The t-distributed stochastic neighbor embedding ( t-SNE ) ( van der Maaten & Hinton , 2008 ) is an unsupervised manifold learning method primarily used for data exploration and visualization by approximating highdimensional data distribution using a two or three-dimensional map that could preserve local and certain global structures of the data . The use of t-SNE for anomaly detection has been sceptical ( van der Maaten & Hinton , 2008 ) . However , no comprehensive investigation has been made in this topic . Taking advantages of both disentangled representation learning ( using β-VAE as an implementation ) and low-dimensional manifold learning ( using t-SNE as an implementation ) , we propose a novel anomaly detection approach named AnoDM , standing for Anomaly detection based on unsupervised Disentangled representation learning and Manifold learning . We introduce a new anomaly score function by combining : ( 1 ) β-VAE ’ s reconstruction error , and ( 2 ) distances between latent representations of test points and training points in t-SNE map . AnoDM is a general framework , thus any disentangled representation learning and manifold learning techniques can be applied . The choice of a lower-level encoding scheme in β-VAE depends on data type of interest . For image data , deterministic convolutional network ( CNN ) is used in the encoder . In case of time series ( sequence ) data , we design an improved version of β-VAE by replacing CNN with temporal convolutional network ( TCN ) ( Bai et al. , 2018 ) , a generic architecture for convolutional sequence prediction , in the encoder . We incorporate TCN as part of the encoder , because Bai et al . ( 2018 ) have shown that TCN outperforms canonical recurrent networks such as LSTMs ( Hochreiter & Schmidhuber , 1997 ) across a range of supervised learning tasks and recommended that CNN should be regarded as the first method to try for sequence modeling tasks . Regarding the decoding architecture , we simply choose CNN , because by choosing a simpler CNN architecture as a part of the decoder , the model can achieve a comparable even better performance but take much less running time . The contributions of this paper are summarized as follows . ( 1 ) We comprehensively explore the capacity of unsupervised disentangled representation learning , using β-VAE as an implementation , for anomaly detection . ( 2 ) We thoroughly investigate the potential of manifold learning for outlier identification by taking the disentangled latent representations from β-VAE as input to t-SNE . To the best of our knowledge , this is the first attempt to explore t-SNE for anomaly detection . ( 3 ) For sequence anomaly detection , instead of using prevailed recurrent networks ( such as LSTM ) , as a practical contribution , we adopt an improved convolution architecture ( TCN ) to capture the temporal dependency in the encoder in unsupervised way . 2 RELATED WORK . In the big data era , the development of deep learning models , especially DGMs , flourishes due to the need of modelling and analyzing massive amount of unstructured data ( such as images , timeseries , graphs , text , etc . ) generated in many application domains . Designing DGM-based solutions for anomaly detection becomes an important topic . Since DGMs , such as VAE ( Kingma & Welling , 2014 ) , and deep belief net ( DBN ) ( Hinton et al. , 2006 ; Li & Zhu , 2018a ) , aim at modelling the joint distribution of visible and latent variables ( that is p ( x , h ) ) , their likelihood p ( x ) by marginalizing out h may serve as an abnormality indicator . However , unlike exponential family restricted Boltzmann machines ( exp-RBMs ) ( Li & Zhu , 2018b ) , exact likelihood is unavailable for most DGMs . Alternatively , reconstruction error serves as an abnormality measure based on the intuition that outof-distribution samples can be reconstructed badly ( An & Cho , 2015 ) . Some deep hybrid methods ( e.g . VAE+OCSVM ( Andrews et al. , 2016 ) and DBN+OCSVM ( Erfani et al. , 2016 ) ) , successfully combine classical one-class support vector machine ( OCSVM ; or kernel-based support vector domain description ( SVDD ) ) with DGMs by using DGMs to learn latent representations of samples and using OCSVM to detect abnormal data points . However , these methods face the challenge of scalability , because the size of kernel matrices in dual form of SVDD is quadratic of sample size . Generative adversarial net ( GAN ) has also been applied for anomaly detection ( Schlegl et al. , 2017 ) . Since there is no encoder in GAN , Deecke et al . ( 2019 ) presented the ADGAN algorithm based on the availability of a good representation of a sample in latent space of its generator by assuming that the generator is able to effectively capture the distribution of the training data . Li et al . ( 2018a ) proposed the GAN-AD method for cyber-physical systems ( CPSs ) . It distinguishes fake data from actual data by taking into consideration of both discrimination loss calculated by the trained discriminator and residual loss between reconstructed and actual test data . Furthermore , DGM-based algorithms are also devised to detect anomaly problem on sequence data ( e.g . LSTM-VAE ( Park et al. , 2017 ) and GAN-AD ( Li et al. , 2018a ) ) . Conventionally , canonical recurrent networks ( such as LSTM and GRUs ( Cho et al. , 2014 ) ) are considered as the dedicated methods for sequence modeling . Some recent studies have also claimed that there was no architecture that could consistently beat LSTM in some typical sequence modelling tasks ( Jozefowicz et al. , 2015 ; Greff et al. , 2015 ; Melis et al. , 2017 ) . On the other hand , some other researchers insist that CNN ( LeCun et al. , 1989 ) should be considered as more appropriate choice for sequences . Inspired by more recent CNN-based sequence modelling ( such as machine translation ( Gehring et al. , 2016 ; 2017 ) and language modeling ( Dauphin et al. , 2016 ) ) , Bai et al . ( 2018 ) conducted a systematic evaluation of generic convolutional and recurrent architectures for sequence modelling across a broad range of tasks that are commonly used to benchmark recurrent networks , and concluded that convolutional networks , rather than recurrent networks , should be respected as a “ natural starting point for sequence modelling tasks ” .

Summary: This paper presents a novel deep anomaly detection model. It combines two existing models: B-VAE and t-SNE. The B-VAE is trained unsupervised and learns an encoder and decoder which provide both an embedding and a reconstruction. Using t-SNE to reduce its dimensionality, the embedding is projected into a 2 dimensional space. An anomaly score function is defined that combines the reconstruction error and the distance in t-SNE space to the K nearest neighbor(s). Experiments are conducted with several image datasets (MNIST,FMNIST,CIFAR10,SmallNORB) and one timeseries dataset (Arrhythmia). For the image sets, the B-VAE model is implemented with a CNN, while for timeseries, a TCN is used. Comparisons are conducted showing the approach to be beat other SOT unsupervised methods, AnoGAN and ADGAN, by 63% and 22% respectively for MNIST and 8% and 2% for FMNIST (in terms of error reduction). For CIFAR-10 and FMNIST it is even demonstrated to beat a supervised SOT method CapsNET. Another experiment shows that t_SNE dramatically improves the performance over B-VAE alone. For the timeseries, the approach is not compared to other SOT approaches as the authors only provide an experiment showing that TCN beats CNN and LSTM for the implementation of the B-VAE. In addition the authors study the effect of the various parameters of the system, in particular the effect of the B in B-VAE and of alpha, the mixing factor between reconstruction error and kNN distance in t_SNE. 3D plots give a good idea on how to select optimal values for the various datasets. The impact of B is also shown on the t-SNE map for MNIST. Finally an ablation studies compares on MNIST the performance of the approach with t-SNE alone, reconstruction alone, and latent distance. On average over 4 digits taken as anomaly, the proposed approach dramatically outperforms the others.