You May Need both Good-GAN and Bad-GAN for Anomaly Detection

1 INTRODUCTION . Anomaly detection ( AD ) , a.k.a . outlier detection , refers to detecting uncommon samples ( usually heterogeneous ) out of inlier distribution ( ChandolaVarun et al. , 2009 ; Pang et al. , 2021a ) . AD can be performed either in an unsupervised manner when only inliers are involved during the training process ( Schlegl et al. , 2019 ; Ngo et al. , 2019 ; Ruff et al. , 2018 ) , or in a semi-supervised manner when few labeled anomalies are available ( Ruff et al. , 2020b ; Liznerski et al. , 2021 ) . Generative adversarial nets ( GAN ) ( Goodfellow et al. , 2014 ) are considered the most effective generative models nowadays , which have also been applied for AD ( Schlegl et al. , 2017 ; 2019 ; Zenati et al. , 2018b ; Siddiquee et al. , 2019 ; Perera et al. , 2019 ) . Likewise , one-class classification and its variants are often used as the backbone of AD approaches ( Schölkopf et al. , 2001 ; Ruff et al. , 2018 ; Liznerski et al. , 2021 ; Wu et al. , 2019 ) . Recently , the adversarial training and one-class classification have been leveraged together to discriminate inlier/outlier in an end-to-end manner ( Zheng et al. , 2019 ; Ngo et al. , 2019 ) . Such efforts share a similar spirit with the work introduced in Dai et al . ( 2017 ) and here we call them Bad-GANs . Instead of generating samples to match the training data distribution as a conventional GAN ( Goodfellow et al. , 2014 ) that we refer as Good-GAN in this paper , a Bad-GAN pushes its generated samples towards the peripheral area of the training data distribution and considers them as pseudo anomalies . Rather than defining anomaly scores based on 1 ) the residual between reconstructed and original samples such as AnoGAN series ( Schlegl et al. , 2017 ; 2019 ) , or 2 ) existing one-class classification-based measures such as Deep SVDD ( Ruff et al. , 2018 ) and FCDD ( Liznerski et al. , 2021 ) , Bad-GANs directly leverage the learned discriminator to identify unseen anomalies from normal cases . Although Bad-GANs have shown great potentials , they suffer from several disadvantages . Dis1 : the generated samples can converge to limited patterns rather than being heterogeneous as real anomalies ; 1Taichi is a concept from traditional Chinese culture , which refers to the co-existence and well-balance of two components with different properties . We borrow this concept to indicate the relationship between “ Bad-GAN ” and “ Good-GAN ” in our model . Dis2 : it is hard to ensure generated samples to resemble real anomalies , while abundance of irrelevant pseudo anomalies may negatively affect the discrimination of real anomalies ; Dis3 : no application has been found to integrate labeled anomalies in prior-art Bad-GANs . In this work , we propose a new Taichi-GAN to address aforementioned disadvantages . First , based on a state-of-the-art Bad-GAN ( FenceGAN ( Ngo et al. , 2019 ) ) framework , an orthogonal loss is proposed to increase the angular diversity , which addresses Dis1 , and partially relieves Dis2 by increasing the overlap between generated pseudo anomalies and real anomalies . Second , we are the first to incorporate Good-GAN within a Bad-GAN framework , where the Good-GAN contributes to generating pseudo anomalies guided by few real anomalies , which addresses Dis2 and Dis3 . Our model is termed Taichi-GAN , as the Good-GAN and Bad-GAN are integrated in a competing yet balanced manner to generate pseudo anomalies towards better AD . We illustrate our ideas with 2D synthetic data and validate the proposed model with five datasets including clinical applications . 2 RELATED WORKS . Shallow AD models have their limitations in dealing with high-dimensional data and large-scale datasets ( Ruff et al. , 2020b ) . Here we focus on the most related deep AD approaches . Reconstruction-error-based ( REB ) Model . REB approaches have been widely applied to imagebased AD , where auto-encoder ( Sato et al. , 2018 ; Baur et al. , 2018 ; Chen & Konukoglu , 2018 ) or GAN ( Schlegl et al. , 2017 ; 2019 ; Zenati et al. , 2018b ; Siddiquee et al. , 2019 ; Akcay et al. , 2018 ; Zenati et al. , 2018a ) is typically used for reconstruction . REB methods assume that anomalies would not be well reconstructed when the model is only trained with normal samples . Therefore , the reconstruction errors can be used to evaluate the abnormality . AnoGAN ( Schlegl et al. , 2017 ) is a representative REB approach which learns a manifold of anatomical variability and detects the anomalies with reconstruction errors mainly on pixel level . F-AnoGAN ( Schlegl et al. , 2019 ) follow the same framework as AnoGAN with improved processing speed . Han et al . ( 2021 ) ensemble multiple GANs for AD . However , REB approaches are mainly designed for data synthesis rather than AD , which may limit their comprehension for heterogeneous anomalies . One-class Classification-based Measure ( OCCBM ) . OCCBM approaches ( Ruff et al. , 2018 ; Wu et al. , 2019 ; Ruff et al. , 2020b ; Chalapathy et al. , 2018 ) learn one-class embedding from normal samples to measure the derivation of an unseen case from the inlier distribution , where the measurement is then served for AD purposes . Deep OCCBM models extend the classical one-class models , e.g. , one-class SVM ( Schölkopf et al. , 2001 ) and SVDD ( Tax & Duin , 2004 ) , by extracting features via deep nets ( Ruff et al. , 2018 ; Liznerski et al. , 2021 ) . OCCBM methods have been deployed to sophisticated applications , e.g. , detecting anomalous event in practical scenes ( Wu et al. , 2019 ) . Bad-GANs . The Bad-GAN idea ( Dai et al. , 2017 ) is introduced for semi-supervised learning in multi-class classification . Similar approaches are found in the scope of the AD such as Sabokrou et al . ( 2018 ) ; Lim et al . ( 2018 ) ; Zheng et al . ( 2019 ) ; Ngo et al . ( 2019 ) . Such techniques are termed as Bad-GANs in this paper . The key idea of Bad-GANs is to learn a classifier ( i.e. , the discriminator ) based on inliers and generated pseudo outliers during training so that it can distinguish unseen outliers and inliers during inference . Bad-GANs belong to the one-class classification family and can be trained end-to-end . In addition , it does not require a manually defined one-class classification measure as OCCBM models ( Pang et al. , 2021a ) ; the Bad-GAN discriminator handles the measure . AD Models Utilizing Real Anomalies . Deep SAD ( Ruff et al. , 2020b ) and ESAD ( Huang et al. , 2020 ) extend the Deep SVDD ( Ruff et al. , 2018 ) to take advantage of labeled anomalies . Siddiquee et al . ( 2019 ) propose a REB model for AD by considering normal and anomalies as two “ styles ” via image-to-image translation techniques . Few-shot learning and reinforcement learning are adapted to utilize limited label information , respectively ( Tian et al. , 2020 ; Pang et al. , 2021b ) . 3 METHOD . 3.1 BACKGROUND : GOOD-GAN AND BAD-GAN . GANs ( Goodfellow et al. , 2014 ) train generative models using a minimax game , which generate data instances from a noise distribution z ∼ pz to match the given data distribution pdata by iteratively optimizing a generator G and a discriminator D with the following objective : min G max D V ( D , G ) = Ex∼pdata [ logD ( x ) ] + Ez∼pz [ log ( 1−D ( G ( z ) ) ] ( 1 ) Dai et al . ( 2017 ) introduce the concept of the Bad-GAN that generates samples at the periphery of the training data . The generator is optimized by minimizing Ex∼pG log pdata ( x ) I [ pdata ( x ) > ] , where I [ x ] is an indicator function . With a threshold ∈ ( 0 , 1 ) , the training process penalizes the high-density samples and leaves the low-density ones unaffected , where the data distribution pdata of Dai et al . ( 2017 ) is estimated with an additional model PixelCNN++ ( Salimans et al. , 2017 ) . The more recent FenceGAN ( Ngo et al. , 2019 ) shares the similar spirit of Dai et al . ( 2017 ) , which generates samples at the boundary of a normal distribution . FenceGAN gets rid of the additional intensity estimation model as in Dai et al . ( 2017 ) , and thus formulates an end-to-end training process . The generator loss 2 on the batch with size N is defined as : LG = β ·N∑N i=1 ( ||G ( zi ) − µ ) || ) + BCE ( D ( G ( z ) ) , α ) ( 2 ) where BCE ( A , B ) is the binary cross-entropy between A and B . The first term is the dispersion loss which regularizes generated samples to be distant from their center µ . The hyperparameter α is a discriminative anomaly score that pushes generated pseudo anomalies on inlier distribution boundary and β is the so-called dispersion hyperparameter . The discriminator loss is defined as : LD = BCE ( D ( x ) ,0 ) + γBCE ( D ( G ( z ) ) ,1 ) ( 3 ) where γ ∈ ( 0 , 1 ] is the so-called anomaly hyperparameter . The discriminator is used to compute an anomaly score ranged ( 0,1 ) for each unseen case during the inference stage . 3.2 ORTHOGONAL LOSS . As anything not normal is defined as an anomaly and anomalies are heterogeneous ( Ruff et al. , 2020b ) , we hypothesize that generated pseudo anomalies in a Bad-GAN framework should have high diversity to cover as many abnormal patterns as possible . Dispersion loss ( Ngo et al. , 2019 ) ( Ld in Fig . 1 ) regularizes generated pseudo anomalies lying at boundary of inlier distribution . Here , we propose to push pseudo anomalies further distributed to evenly cover the entire angular space . ! # ! '' ! $ As verified in empirical experiments , the generated samples from a Bad-GAN can be prone to specific patterns , even with the Euclidean dispersion regularization of FenceGAN ( Ngo et al. , 2019 ) . We introduce an orthogonal loss Lorth to expand generated samples from the angular perspective . As depicted in Fig . 1 , paired samples with smaller angles ( left in Fig . 1 ) to each other can have a 2Equations are adapted from the official Github of FenceGAN , which have a different form with its paper . larger penalty given the orthogonal regularization . The orthogonal regularization has been used for weight initialization and optimized on deep features ( Brock et al. , 2017 ; Arjovsky et al. , 2015 ; Zhao et al. , 2016 ) . Here , we define our Lorth in a decentralized way and optimized on generated pseudo anomalies in an AD framework : Lorth = 1 N ( N − 1 ) N∑ i=1 ∑ j 6=i ( ( G ( zi ) − µ ) T ( G ( zj ) − µ ) ||G ( zi ) − µ|| · ||G ( zj ) − µ|| ) 2 ( 4 ) Instead of optimizing on the commonly used Euclidean distance and individual regularization ( e.g. , the dispersion term of Ngo et al . ( 2019 ) and the losses in OCCBM models ( Ruff et al. , 2018 ; 2020a ; b ) ) , our proposed orthogonal loss leverages Cosine distance and relational regularization3 to be complementary to the dispersion term in the Bad-GAN framework .

Summary: The paper proposes a new interesting method for anomaly detection. In particular, to overcome the limitations of existing Bad-GAN, the authors introduce the orthogonal loss to regularize the generation of anomaly samples to be distributed evenly at the periphery of the training data. Furthermore, in the scenario of available anomalies, the authors combine Bad-GAN and Good-GAN together, in which Good-GAN learns to generate the anomalies while Bad-GAN reguralizes the anomaly pseudo anomalies at the boundary of inlier distribution.