You May Need both Good-GAN and Bad-GAN for Anomaly Detection

1 INTRODUCTION . Anomaly detection ( AD ) , a.k.a . outlier detection , refers to detecting uncommon samples ( usually heterogeneous ) out of inlier distribution ( ChandolaVarun et al. , 2009 ; Pang et al. , 2021a ) . AD can be performed either in an unsupervised manner when only inliers are involved during the training process ( Schlegl et al. , 2019 ; Ngo et al. , 2019 ; Ruff et al. , 2018 ) , or in a semi-supervised manner when few labeled anomalies are available ( Ruff et al. , 2020b ; Liznerski et al. , 2021 ) . Generative adversarial nets ( GAN ) ( Goodfellow et al. , 2014 ) are considered the most effective generative models nowadays , which have also been applied for AD ( Schlegl et al. , 2017 ; 2019 ; Zenati et al. , 2018b ; Siddiquee et al. , 2019 ; Perera et al. , 2019 ) . Likewise , one-class classification and its variants are often used as the backbone of AD approaches ( Schölkopf et al. , 2001 ; Ruff et al. , 2018 ; Liznerski et al. , 2021 ; Wu et al. , 2019 ) . Recently , the adversarial training and one-class classification have been leveraged together to discriminate inlier/outlier in an end-to-end manner ( Zheng et al. , 2019 ; Ngo et al. , 2019 ) . Such efforts share a similar spirit with the work introduced in Dai et al . ( 2017 ) and here we call them Bad-GANs . Instead of generating samples to match the training data distribution as a conventional GAN ( Goodfellow et al. , 2014 ) that we refer as Good-GAN in this paper , a Bad-GAN pushes its generated samples towards the peripheral area of the training data distribution and considers them as pseudo anomalies . Rather than defining anomaly scores based on 1 ) the residual between reconstructed and original samples such as AnoGAN series ( Schlegl et al. , 2017 ; 2019 ) , or 2 ) existing one-class classification-based measures such as Deep SVDD ( Ruff et al. , 2018 ) and FCDD ( Liznerski et al. , 2021 ) , Bad-GANs directly leverage the learned discriminator to identify unseen anomalies from normal cases . Although Bad-GANs have shown great potentials , they suffer from several disadvantages . Dis1 : the generated samples can converge to limited patterns rather than being heterogeneous as real anomalies ; 1Taichi is a concept from traditional Chinese culture , which refers to the co-existence and well-balance of two components with different properties . We borrow this concept to indicate the relationship between “ Bad-GAN ” and “ Good-GAN ” in our model . Dis2 : it is hard to ensure generated samples to resemble real anomalies , while abundance of irrelevant pseudo anomalies may negatively affect the discrimination of real anomalies ; Dis3 : no application has been found to integrate labeled anomalies in prior-art Bad-GANs . In this work , we propose a new Taichi-GAN to address aforementioned disadvantages . First , based on a state-of-the-art Bad-GAN ( FenceGAN ( Ngo et al. , 2019 ) ) framework , an orthogonal loss is proposed to increase the angular diversity , which addresses Dis1 , and partially relieves Dis2 by increasing the overlap between generated pseudo anomalies and real anomalies . Second , we are the first to incorporate Good-GAN within a Bad-GAN framework , where the Good-GAN contributes to generating pseudo anomalies guided by few real anomalies , which addresses Dis2 and Dis3 . Our model is termed Taichi-GAN , as the Good-GAN and Bad-GAN are integrated in a competing yet balanced manner to generate pseudo anomalies towards better AD . We illustrate our ideas with 2D synthetic data and validate the proposed model with five datasets including clinical applications . 2 RELATED WORKS . Shallow AD models have their limitations in dealing with high-dimensional data and large-scale datasets ( Ruff et al. , 2020b ) . Here we focus on the most related deep AD approaches . Reconstruction-error-based ( REB ) Model . REB approaches have been widely applied to imagebased AD , where auto-encoder ( Sato et al. , 2018 ; Baur et al. , 2018 ; Chen & Konukoglu , 2018 ) or GAN ( Schlegl et al. , 2017 ; 2019 ; Zenati et al. , 2018b ; Siddiquee et al. , 2019 ; Akcay et al. , 2018 ; Zenati et al. , 2018a ) is typically used for reconstruction . REB methods assume that anomalies would not be well reconstructed when the model is only trained with normal samples . Therefore , the reconstruction errors can be used to evaluate the abnormality . AnoGAN ( Schlegl et al. , 2017 ) is a representative REB approach which learns a manifold of anatomical variability and detects the anomalies with reconstruction errors mainly on pixel level . F-AnoGAN ( Schlegl et al. , 2019 ) follow the same framework as AnoGAN with improved processing speed . Han et al . ( 2021 ) ensemble multiple GANs for AD . However , REB approaches are mainly designed for data synthesis rather than AD , which may limit their comprehension for heterogeneous anomalies . One-class Classification-based Measure ( OCCBM ) . OCCBM approaches ( Ruff et al. , 2018 ; Wu et al. , 2019 ; Ruff et al. , 2020b ; Chalapathy et al. , 2018 ) learn one-class embedding from normal samples to measure the derivation of an unseen case from the inlier distribution , where the measurement is then served for AD purposes . Deep OCCBM models extend the classical one-class models , e.g. , one-class SVM ( Schölkopf et al. , 2001 ) and SVDD ( Tax & Duin , 2004 ) , by extracting features via deep nets ( Ruff et al. , 2018 ; Liznerski et al. , 2021 ) . OCCBM methods have been deployed to sophisticated applications , e.g. , detecting anomalous event in practical scenes ( Wu et al. , 2019 ) . Bad-GANs . The Bad-GAN idea ( Dai et al. , 2017 ) is introduced for semi-supervised learning in multi-class classification . Similar approaches are found in the scope of the AD such as Sabokrou et al . ( 2018 ) ; Lim et al . ( 2018 ) ; Zheng et al . ( 2019 ) ; Ngo et al . ( 2019 ) . Such techniques are termed as Bad-GANs in this paper . The key idea of Bad-GANs is to learn a classifier ( i.e. , the discriminator ) based on inliers and generated pseudo outliers during training so that it can distinguish unseen outliers and inliers during inference . Bad-GANs belong to the one-class classification family and can be trained end-to-end . In addition , it does not require a manually defined one-class classification measure as OCCBM models ( Pang et al. , 2021a ) ; the Bad-GAN discriminator handles the measure . AD Models Utilizing Real Anomalies . Deep SAD ( Ruff et al. , 2020b ) and ESAD ( Huang et al. , 2020 ) extend the Deep SVDD ( Ruff et al. , 2018 ) to take advantage of labeled anomalies . Siddiquee et al . ( 2019 ) propose a REB model for AD by considering normal and anomalies as two “ styles ” via image-to-image translation techniques . Few-shot learning and reinforcement learning are adapted to utilize limited label information , respectively ( Tian et al. , 2020 ; Pang et al. , 2021b ) . 3 METHOD . 3.1 BACKGROUND : GOOD-GAN AND BAD-GAN . GANs ( Goodfellow et al. , 2014 ) train generative models using a minimax game , which generate data instances from a noise distribution z ∼ pz to match the given data distribution pdata by iteratively optimizing a generator G and a discriminator D with the following objective : min G max D V ( D , G ) = Ex∼pdata [ logD ( x ) ] + Ez∼pz [ log ( 1−D ( G ( z ) ) ] ( 1 ) Dai et al . ( 2017 ) introduce the concept of the Bad-GAN that generates samples at the periphery of the training data . The generator is optimized by minimizing Ex∼pG log pdata ( x ) I [ pdata ( x ) > ] , where I [ x ] is an indicator function . With a threshold ∈ ( 0 , 1 ) , the training process penalizes the high-density samples and leaves the low-density ones unaffected , where the data distribution pdata of Dai et al . ( 2017 ) is estimated with an additional model PixelCNN++ ( Salimans et al. , 2017 ) . The more recent FenceGAN ( Ngo et al. , 2019 ) shares the similar spirit of Dai et al . ( 2017 ) , which generates samples at the boundary of a normal distribution . FenceGAN gets rid of the additional intensity estimation model as in Dai et al . ( 2017 ) , and thus formulates an end-to-end training process . The generator loss 2 on the batch with size N is defined as : LG = β ·N∑N i=1 ( ||G ( zi ) − µ ) || ) + BCE ( D ( G ( z ) ) , α ) ( 2 ) where BCE ( A , B ) is the binary cross-entropy between A and B . The first term is the dispersion loss which regularizes generated samples to be distant from their center µ . The hyperparameter α is a discriminative anomaly score that pushes generated pseudo anomalies on inlier distribution boundary and β is the so-called dispersion hyperparameter . The discriminator loss is defined as : LD = BCE ( D ( x ) ,0 ) + γBCE ( D ( G ( z ) ) ,1 ) ( 3 ) where γ ∈ ( 0 , 1 ] is the so-called anomaly hyperparameter . The discriminator is used to compute an anomaly score ranged ( 0,1 ) for each unseen case during the inference stage . 3.2 ORTHOGONAL LOSS . As anything not normal is defined as an anomaly and anomalies are heterogeneous ( Ruff et al. , 2020b ) , we hypothesize that generated pseudo anomalies in a Bad-GAN framework should have high diversity to cover as many abnormal patterns as possible . Dispersion loss ( Ngo et al. , 2019 ) ( Ld in Fig . 1 ) regularizes generated pseudo anomalies lying at boundary of inlier distribution . Here , we propose to push pseudo anomalies further distributed to evenly cover the entire angular space . ! # ! '' ! $ As verified in empirical experiments , the generated samples from a Bad-GAN can be prone to specific patterns , even with the Euclidean dispersion regularization of FenceGAN ( Ngo et al. , 2019 ) . We introduce an orthogonal loss Lorth to expand generated samples from the angular perspective . As depicted in Fig . 1 , paired samples with smaller angles ( left in Fig . 1 ) to each other can have a 2Equations are adapted from the official Github of FenceGAN , which have a different form with its paper . larger penalty given the orthogonal regularization . The orthogonal regularization has been used for weight initialization and optimized on deep features ( Brock et al. , 2017 ; Arjovsky et al. , 2015 ; Zhao et al. , 2016 ) . Here , we define our Lorth in a decentralized way and optimized on generated pseudo anomalies in an AD framework : Lorth = 1 N ( N − 1 ) N∑ i=1 ∑ j 6=i ( ( G ( zi ) − µ ) T ( G ( zj ) − µ ) ||G ( zi ) − µ|| · ||G ( zj ) − µ|| ) 2 ( 4 ) Instead of optimizing on the commonly used Euclidean distance and individual regularization ( e.g. , the dispersion term of Ngo et al . ( 2019 ) and the losses in OCCBM models ( Ruff et al. , 2018 ; 2020a ; b ) ) , our proposed orthogonal loss leverages Cosine distance and relational regularization3 to be complementary to the dispersion term in the Bad-GAN framework .

Summary: The authors of this paper propose a GAN-based method for the anomaly detection task. Their method relies on so-called Bad GANs (that uses the trained GAN discriminator to distinguish between inliers and pseudo-anomalies), a new orthogonal loss (that favorises generated samples to cover a larger angular space) and eventually a so-called Good GAN (which generator is shared with the Bad GAN; and discriminator is trained on "real" anomalies). The method is evaluated along a set of baselines in experiments that also include ablation studies.