A Multi-modal one-class generative adversarial network for anomaly detection in manufacturing

Anomaly detection is of the critical issue in both fundamental machine learning research area and industrial applications . A good anomaly detection should accurately discriminate anomalies from normal samples . Although most previous anomaly detection methods achieve good performances on various tasks , they do not perform well on high-dimensional imbalanced small data-sets with multimodal distribution . Unlike existing approaches , each mode in the distribution is individually detected by a single model , and thus handle multi-modalities . In this paper , we develop a multimodal one-class model based on generative adversarial network ( GAN ) to distinguish anomalies from normal samples . The generator of the GAN takes in a noise vector with a pseudo latent code to generate instances at the low-density area of normal samples within the same data space to simulate the anomalies . The discriminator of the GAN then is trained to distinguish the generated samples from the normal samples . Since the generated instances mimic the low-density distribution of the normal samples ( outliers ) , the discriminator should directly detect real anomalies from normal samples . We test our methods on several datasets . In these experiments , our method outperforms the state-of-the-art anomaly detection methods on both the accuracy and F1 score . 1 INTRODUCTION . Anomaly detection is one of the fundamental problems in machine learning , with many critical applications in industry , e.g. , fraud detection ( Zheng et al. , 2018 ) . Compared with supervised based anomaly detection methods ( Nakazawa & Kulkarni , 2018 ; Lee et al. , 2017 ; He & Wang , 2007 ) , we focus on the anomaly detection problems that is to detect if a new data sample follows a known normal data distribution with high confidence , i.e . resides in high-density distribution . These problems are also known as novelty detection problem ( Pimentel et al. , 2014 ) . Plenty of works have been conducted in this area . There are three major directions : 1 ) density estimation methods 2 ) reconstruction loss based methods 3 ) one-class classification . These methods present strong performances on various dataset and tasks . Different from most existing anomaly detection problem , e.g. , ( Rayana & Akoglu , 2015 ; Stolfo et al. , 2000 ; McAuley & Leskovec , 2013 ; Wienke et al. , 2016 ; Poll et al. , 2007 ) , in practice , we usually suffer from several practical issues : 1 ) a limited amount of data samples 2 ) imbalanced dataset 3 ) multi-modal distribution . An example of normal and anomaly data samples are shown in Figure 1 . Thus , in this paper , to overcome these challenges , we propose to use the multimodal one-class generative adversarial network ( MMOC-GAN ) to address the challenges mentioned above . The primary idea to capture the modes of the normal samples , and generate possible outliers for these modes using the normal samples . In the GAN model , instead of taking a noise sampled from a fixed distribution , the generator takes in a noise vector with a latent code to generate samples that are complementary to the distribution of the normal samples within the same data space . We expect the generator could generate the complementary data samples for a specific mode within the same data space by adding the latent code . The discriminator is trained to discriminate the normal samples from generated samples . Since the generated samples approximate the low-density distribution of the normal samples , which simulate the outlier of normal samples , we expect the discriminator could distinguish normal samples and anomalies . The property of MMOC-GAN method makes it applicable to anomaly detection in various practical applications . The experiment on several datasets demonstrates that MMOC-GAN has superior performance over several types of anomaly detection methods . Additionally , we observe that the generator could successfully generate complementary data samples for each existing modes . The rest of the paper is organized as follows : the related work is shown in section 2 . A review of GAN is given in section 3 . We then describe the MMOC-GAN in section 4 . The experimental result is shown in section 5 . We then conclude the paper in section 6 . 2 RELATED WORK . Lots of efforts have been done in the anomaly detection area . There are primarily three directions : 1 ) density estimation methods 2 ) reconstruction loss based methods 3 ) one-class classification ( Pimentel et al. , 2014 ) . Density estimation methods are primarily based on clustering analysis , such as the Gaussian mixture model ( GMM ) ( Markou & Singh , 2003 ; Lauer , 2001 ) . However , it is hard to apply these methods directly on high-dimensional data . Therefore , various methods adopt a two-stage approach that reduces the dimension of data as the first step and then uses the density estimation method for anomaly detection as the second step ( Chandola et al. , 2009 ; Sabahi & Movaghar , 2008 ) . However , the two-stage approach has multiple drawbacks . Recently , an end-to-end deep auto-encoding Gaussian mixture model ( DAGMM ) ( Zong et al. , 2018 ) model is proposed that combines a compression network to extract latent feature and an estimation network using latent feature and reconstruction loss to estimate the sample density . Reconstruction loss based methods assume that anomaly data samples can not be reconstructed from low-dimensional space ( Shyu et al. , 2003 ; Sakurada & Yairi , 2014 ) . Many recent works presented to use the reconstruction loss by auto-encoder ( Marchi et al. , 2015 ) , variational auto-encoder ( An & Cho , 2015 ) , as well as generative adversarial network based reconstruction loss ( Schlegl et al. , 2017 ) . These works demonstrate a promising result . However , this assumption does not hold in every case . As shown in Figure 2 , most of the anomalies have similar reconstruction loss as the normal data samples . This might due to that production process of anomalies and normal samples in the dataset are very similar . One class classification method tries to build a classifier using only one class of data samples , i.e . normal data samples . This method learns a discriminative boundary surrounding the normal instances thus to detect anomalies ( Chen et al. , 2001 ; Zhang et al. , 2006 ) . One-class support vector machine ( OC-SVM ) is one of the widely adopted methods that construct a decision hyper-plane around the normal sample ( Erfani et al. , 2016 ) . However , OC-SVM usually suffers from the curse of dimensionality . One class neural network ( OCNN network ) is an end-to-end method that is developed based on OC-SVM ( Chalapathy et al. , 2018 ) . It combines a neural network to learn the latent distribution and use the objective function similar to SVM , thus , to detect anomalies . Also , some other distance-based methods have extended to this category as well . One class nearest neighbor ( OCNN neighbor ) has also been used to predict the anomalies based on the distance to its nearest neighbors ( Janssens , 2013 ; Zhao & Saligrama , 2009 ) . In some other works , the discriminator in the GAN model has also been used as an anomaly detector ( Zenati et al. , 2018 ; Sabokrou et al. , 2018 ) . One class adversarial network is recently developed based on the GAN model by using LSTM autoencoder to compress the data and the complementary GAN to learn the possible distribution of anomaly data ( Zheng et al. , 2018 ) . Most of these methods obtained relatively good performance on various datasets . However , for the datasets with multiple modes , the performance is limited . According to the characteristic of these datasets , we develop our algorithm MMOC-GAN based on the one class classification method . Based on our knowledge , this is the first one-class anomaly detection model for multi-modal distributions . 3 BACKGROUND INFORMATION . 3.1 GENERATIVE ADVERSARIAL NETWORK . Generative adversarial networks ( GANs ) ( Chen et al. , 2016 ; Goodfellow et al. , 2014 ) have recently received much attention . It is a framework for training deep generative models using mini-max optimization . A GAN framework consists of two components , a generator G and a discriminator D. In practice , these two components are usually multi-perceptron neural networks . The generator generates fake samples xG from a noise vector z sampled from a prior distribution pz , i.e . xG = G ( z ) . G is trained to learn a distribution pG that matches the real data distribution pd . In other word , we try to maximize the pd ( xG ) . The discriminator D is a binary classifier that takes in a sample x as input and output the probability that it is a real data or a generated fake data from xG = G ( z ) . Thus , D acts as a detector to estimate to the probability that a sample is from the real data distribution . The G and D are trained adversarially as competitors to each other by alternatively training G and D. G tries to fool D by making D predicts samples xG generate by G is real . This is achieved by optimizing the following objective function of G : min G Ez∼pz [ log ( 1−D ( G ( z ) ) ) ] ( 1 ) On the contrary , D tries to minimize the chance that it being fooled by maximizing the probability that it predicts the real data x is real and minimizing the probability that the generated data G ( z ) is real : max D Ex∼pd [ logD ( x ) ] + Ez∼pz [ log ( 1−D ( G ( z ) ) ) ] ( 2 ) The GAN model is thus formalized as a mini-max problem with the following objective : min G max D V ( D , G ) = Ex∼pd [ logD ( x ) ] + Ez∼pz [ log ( 1−D ( G ( z ) ) ) ] ( 3 ) The GAN model theoretically aims to minimize the Jensen-Shannon ( JS ) divergence between the data distribution Pd and the generated distribution PG . The minimization of JS divergence is achieved when pD ( G ( z ) ) = pd ( x ) / ( pd ( x ) + PG ( x ) ) that the generated samples are indistinguishable from real data samples . Therefore , the GAN model captures the distribution of the real data . 4 METHODOLOGY . 4.1 OVERVIEW . The generatorG in the GAN model uses a simple noise vector z as input and maps it to a complicated data distribution xG . This mapping requires a generator that disentangle the underlying factors of variations in the data distribution and enables multi-modal diversity . However , in practice , regular GAN is known to have the model collapse problems ( Salimans et al. , 2016 ; Berthelot et al. , 2017 ) . This problem is not desired for a multimodal dataset , as shown in Figure 1 . For these datasets , It is natural to decompose different modes into a set of essential factors of variations by injecting prior information , thus to have the generator in the GAN to produce data samples for different modes . The idea of adding prior information can be found in various works . Inspired by ( Chen et al. , 2016 ; Gurumurthy et al. , 2017 ) , our G is designed to take in both noise vector z and prior information . The discriminatorD is used to detect the anomalies from the normal samples . Different from regular GAN , our generator G tries to generate complementary data samples which lie in the low-density region of the normal samples . The discriminator tries to separate the normal samples and the generated samples , which should give itself the capability to detect anomalies from normal data samples because the generated samples simulate outliners in the low-density area . The overall framework of our method can be found in Figure 3 .

Summary: This paper presents a GAN approach adapted for multi-modal distributions of single class data. The generator is trained to generate samples in the low density areas of the data distribution. The discriminator is training to distinguish between generated and real samples and hence is able to discriminate between normal data (=real) and anomalous data (=generated, in low density areas of the normal data).