Year: 2021

Abstract:
Generative adversarial nets (GAN) have been successfully adapted for anomaly detection, where end-to-end anomaly scoring by so-called Bad-GAN has shown promising results. A Bad-GAN generates pseudo anomalies at the low-density area of inlier distribution, and thus the inlier/outlier distinction can be approximated. However, the generated pseudo anomalies from existing Bad-GAN approaches may (1) converge to certain patterns with limited diversity, and (2) differ from the real anomalies, making the anomaly detection hard to generalize. In this work, we propose a new model called Taichi-GAN 1 to address the aforementioned issues of a conventional Bad-GAN. First, a new orthogonal loss is proposed to regularize the cosine distance of decentralized generated samples in a Bad-GAN. Second, we utilize few anomaly samples (when available) with a conventional GAN, i.e., so-called Good-GAN, to draw the generated pseudo anomalies closer to the real anomalies. Our Taichi-GAN incorporates Good-GAN and Bad-GAN in an adversarial manner; which generates pseudo anomalies that contributing to a more robust discriminator for anomaly scoring, and thus anomaly detection. Substantial improvements can be observed from our proposed model on multiple simulated and real-life anatomy detection tasks.

heading: 1 INTRODUCTION
text: Anomaly detection (AD), a.k.a. outlier detection, refers to detecting uncommon samples (usually heterogeneous) out of inlier distribution (ChandolaVarun et al., 2009; Pang et al., 2021a). AD can be performed either in an unsupervised manner when only inliers are involved during the training process (Schlegl et al., 2019; Ngo et al., 2019; Ruff et al., 2018), or in a semi-supervised manner when few labeled anomalies are available (Ruff et al., 2020b; Liznerski et al., 2021).
Generative adversarial nets (GAN) (Goodfellow et al., 2014) are considered the most effective generative models nowadays, which have also been applied for AD (Schlegl et al., 2017; 2019; Zenati et al., 2018b; Siddiquee et al., 2019; Perera et al., 2019). Likewise, one-class classification and its variants are often used as the backbone of AD approaches (Schölkopf et al., 2001; Ruff et al., 2018; Liznerski et al., 2021; Wu et al., 2019). Recently, the adversarial training and one-class classification have been leveraged together to discriminate inlier/outlier in an end-to-end manner (Zheng et al., 2019; Ngo et al., 2019). Such efforts share a similar spirit with the work introduced in Dai et al. (2017) and here we call them Bad-GANs. Instead of generating samples to match the training data distribution as a conventional GAN (Goodfellow et al., 2014) that we refer as Good-GAN in this paper, a Bad-GAN pushes its generated samples towards the peripheral area of the training data distribution and considers them as pseudo anomalies. Rather than defining anomaly scores based on 1) the residual between reconstructed and original samples such as AnoGAN series (Schlegl et al., 2017; 2019), or 2) existing one-class classification-based measures such as Deep SVDD (Ruff et al., 2018) and FCDD (Liznerski et al., 2021), Bad-GANs directly leverage the learned discriminator to identify unseen anomalies from normal cases.
Although Bad-GANs have shown great potentials, they suffer from several disadvantages. Dis1: the generated samples can converge to limited patterns rather than being heterogeneous as real anomalies;
1Taichi is a concept from traditional Chinese culture, which refers to the co-existence and well-balance of two components with different properties. We borrow this concept to indicate the relationship between “Bad-GAN” and “Good-GAN” in our model.
Dis2: it is hard to ensure generated samples to resemble real anomalies, while abundance of irrelevant pseudo anomalies may negatively affect the discrimination of real anomalies; Dis3: no application has been found to integrate labeled anomalies in prior-art Bad-GANs.
In this work, we propose a new Taichi-GAN to address aforementioned disadvantages. First, based on a state-of-the-art Bad-GAN (FenceGAN (Ngo et al., 2019)) framework, an orthogonal loss is proposed to increase the angular diversity, which addresses Dis1, and partially relieves Dis2 by increasing the overlap between generated pseudo anomalies and real anomalies. Second, we are the first to incorporate Good-GAN within a Bad-GAN framework, where the Good-GAN contributes to generating pseudo anomalies guided by few real anomalies, which addresses Dis2 and Dis3. Our model is termed Taichi-GAN, as the Good-GAN and Bad-GAN are integrated in a competing yet balanced manner to generate pseudo anomalies towards better AD. We illustrate our ideas with 2D synthetic data and validate the proposed model with five datasets including clinical applications.

heading: 2 RELATED WORKS
text: Shallow AD models have their limitations in dealing with high-dimensional data and large-scale datasets (Ruff et al., 2020b). Here we focus on the most related deep AD approaches.
Reconstruction-error-based (REB) Model. REB approaches have been widely applied to imagebased AD, where auto-encoder (Sato et al., 2018; Baur et al., 2018; Chen & Konukoglu, 2018) or GAN (Schlegl et al., 2017; 2019; Zenati et al., 2018b; Siddiquee et al., 2019; Akcay et al., 2018; Zenati et al., 2018a) is typically used for reconstruction. REB methods assume that anomalies would not be well reconstructed when the model is only trained with normal samples. Therefore, the reconstruction errors can be used to evaluate the abnormality. AnoGAN (Schlegl et al., 2017) is a representative REB approach which learns a manifold of anatomical variability and detects the anomalies with reconstruction errors mainly on pixel level. F-AnoGAN (Schlegl et al., 2019) follow the same framework as AnoGAN with improved processing speed. Han et al. (2021) ensemble multiple GANs for AD. However, REB approaches are mainly designed for data synthesis rather than AD, which may limit their comprehension for heterogeneous anomalies.
One-class Classification-based Measure (OCCBM). OCCBM approaches (Ruff et al., 2018; Wu et al., 2019; Ruff et al., 2020b; Chalapathy et al., 2018) learn one-class embedding from normal samples to measure the derivation of an unseen case from the inlier distribution, where the measurement is then served for AD purposes. Deep OCCBM models extend the classical one-class models, e.g., one-class SVM (Schölkopf et al., 2001) and SVDD (Tax & Duin, 2004), by extracting features via deep nets (Ruff et al., 2018; Liznerski et al., 2021). OCCBM methods have been deployed to sophisticated applications, e.g., detecting anomalous event in practical scenes (Wu et al., 2019).
Bad-GANs. The Bad-GAN idea (Dai et al., 2017) is introduced for semi-supervised learning in multi-class classification. Similar approaches are found in the scope of the AD such as Sabokrou et al. (2018); Lim et al. (2018); Zheng et al. (2019); Ngo et al. (2019). Such techniques are termed as Bad-GANs in this paper. The key idea of Bad-GANs is to learn a classifier (i.e., the discriminator) based on inliers and generated pseudo outliers during training so that it can distinguish unseen outliers and inliers during inference. Bad-GANs belong to the one-class classification family and can be trained end-to-end. In addition, it does not require a manually defined one-class classification measure as OCCBM models (Pang et al., 2021a); the Bad-GAN discriminator handles the measure.
AD Models Utilizing Real Anomalies. Deep SAD (Ruff et al., 2020b) and ESAD (Huang et al., 2020) extend the Deep SVDD (Ruff et al., 2018) to take advantage of labeled anomalies. Siddiquee et al. (2019) propose a REB model for AD by considering normal and anomalies as two “styles” via image-to-image translation techniques. Few-shot learning and reinforcement learning are adapted to utilize limited label information, respectively (Tian et al., 2020; Pang et al., 2021b).

heading: 3 METHOD

heading: 3.1 BACKGROUND: GOOD-GAN AND BAD-GAN
text: GANs (Goodfellow et al., 2014) train generative models using a minimax game, which generate data instances from a noise distribution z ∼ pz to match the given data distribution pdata by iteratively
optimizing a generator G and a discriminator D with the following objective:
min G max D V (D,G) = Ex∼pdata [logD(x)] + Ez∼pz [log(1−D(G(z))] (1)
Dai et al. (2017) introduce the concept of the Bad-GAN that generates samples at the periphery of the training data. The generator is optimized by minimizing Ex∼pG log pdata(x)I[pdata(x) > ], where I[x] is an indicator function. With a threshold ∈ (0, 1), the training process penalizes the high-density samples and leaves the low-density ones unaffected, where the data distribution pdata of Dai et al. (2017) is estimated with an additional model PixelCNN++ (Salimans et al., 2017).
The more recent FenceGAN (Ngo et al., 2019) shares the similar spirit of Dai et al. (2017), which generates samples at the boundary of a normal distribution. FenceGAN gets rid of the additional intensity estimation model as in Dai et al. (2017), and thus formulates an end-to-end training process. The generator loss 2 on the batch with size N is defined as:
LG = β ·N∑N
i=1(||G(zi)− µ)||) + BCE(D(G(z)), α) (2)
where BCE(A,B) is the binary cross-entropy between A and B. The first term is the dispersion loss which regularizes generated samples to be distant from their center µ. The hyperparameter α is a discriminative anomaly score that pushes generated pseudo anomalies on inlier distribution boundary and β is the so-called dispersion hyperparameter. The discriminator loss is defined as:
LD = BCE(D(x),0) + γBCE(D(G(z)),1) (3)
where γ ∈ (0, 1] is the so-called anomaly hyperparameter. The discriminator is used to compute an anomaly score ranged (0,1) for each unseen case during the inference stage.

heading: 3.2 ORTHOGONAL LOSS
text: As anything not normal is defined as an anomaly and anomalies are heterogeneous (Ruff et al., 2020b), we hypothesize that generated pseudo anomalies in a Bad-GAN framework should have high diversity to cover as many abnormal patterns as possible. Dispersion loss (Ngo et al., 2019) (Ld in Fig. 1) regularizes generated pseudo anomalies lying at boundary of inlier distribution. Here, we propose to push pseudo anomalies further distributed to evenly cover the entire angular space.
!# !"
!$
As verified in empirical experiments, the generated samples from a Bad-GAN can be prone to specific patterns, even with the Euclidean dispersion regularization of FenceGAN (Ngo et al., 2019). We introduce an orthogonal loss Lorth to expand generated samples from the angular perspective. As depicted in Fig. 1, paired samples with smaller angles (left in Fig. 1) to each other can have a
2Equations are adapted from the official Github of FenceGAN, which have a different form with its paper.
larger penalty given the orthogonal regularization. The orthogonal regularization has been used for weight initialization and optimized on deep features (Brock et al., 2017; Arjovsky et al., 2015; Zhao et al., 2016). Here, we define our Lorth in a decentralized way and optimized on generated pseudo anomalies in an AD framework:
Lorth = 1
N(N − 1) N∑ i=1 ∑ j 6=i ( (G(zi)− µ)T (G(zj)− µ) ||G(zi)− µ|| · ||G(zj)− µ|| )2 (4)
Instead of optimizing on the commonly used Euclidean distance and individual regularization (e.g., the dispersion term of Ngo et al. (2019) and the losses in OCCBM models (Ruff et al., 2018; 2020a;b)), our proposed orthogonal loss leverages Cosine distance and relational regularization3 to be complementary to the dispersion term in the Bad-GAN framework.

heading: 3.3 THE INTUITION OF ADDING GOOD-GAN TO BAD-GAN
text: As Good-GANs optimize generated samples to match the training distribution, given few real anomalies are available, we take this Good-GAN nature to guide pseudo anomaly generation in AD tasks. Simultaneously, to avoid generated samples overfitting to limited anomaly patterns, we balance the Good-GAN and the Bad-GAN (including Lorth) with a hyperparameter δ. Specifically, the Good-GAN guides the generated pseudo anomalies to resemble the reference distribution from given real anomalies, while the Bad-GAN regularizes pseudo anomalies at the boundary of inlier distribution. The Good-GAN and the Bad-GAN share the same generator, and have their own discriminators separately. The overall objective is defined as follows:
min G max Dbd max Dgd {Vbd(G,Dbd)︸ ︷︷ ︸
Bad-GAN
+ Lorth︸ ︷︷ ︸ angular diversity + δVgd(G,Dgd)︸ ︷︷ ︸ Good-GAN } (5)
where Vbd and Vgd are the objective functions of the Bad-GAN and Good-GAN, respectively. G is the shared generator. Dbd and Dgd are discriminators of Bad-GAN and Good-GAN, respectively. The generated samples serve as pseudo anomalies to train a classifier (i.e., Dbd) to discriminate unseen real anomalies from inliers during inference. Lorth is the proposed orthogonal loss to further increase the diversities of pseudo anomalies. The overall model (Eq. 5) is termed as Taichi-GAN.
As shown in Fig. 2(a), the Bad-GAN generates samples along the boundary of the inlier distribution, however, with limited diversity, even including a dispersion loss on Euclidean distance space. Adding
3Concepts of individual regularization and relational regularization have been introduced in Liu et al. (2021).
a pull-away term (Zhao et al., 2016) (an orthogonal regularization without decentralization to the sample average) is also not effective in this AD framework as in Fig. 2(b). The generated samples are more evenly distributed with the proposed orthogonal loss (Fig. 2(c)), which are more likely to cover samples that resemble real anomalies even though no real anomaly is included for training. Further, when few labeled anomalies are available, the Good-GAN inclusion pushes the generated samples to have higher density near the given anomalies, meanwhile, Lorth maintains high diversity to cover other potential anomaly patterns (Fig. 2(d)).
3.4 THE PROPOSED TAICHI-GAN
Fig. 3(a) shows our motivation with inlier and (pseudo) outlier example from HAM10000 (Tschandl et al., 2018). The upper depicts the training with only normal samples, where we target generating highly diverse samples (e.g., (1) vs. (2) in Fig. 3(a)) at the boundary of normal distribution. The bottom shows the motivation on sample generation after adding few anomalies. Besides keeping the diversity, some generated samples are close to known anomalies (e.g., (3) is close to (4)).
We borrow Bad-GAN components from FenceGAN. The generator loss is defined as follows:
LG = Lorth + β ·N∑N
i=1(||G(zi)− µ)|| + BCE(Dbd(G(z)), α)︸ ︷︷ ︸
LbadG
+ δBCE(Dgd(G(z)),0)︸ ︷︷ ︸ LgoodG (6)
The generator losses of Bad-GAN (LbadG) and Good-GAN (LgoodG), is consistent with the form of Eq. 2 and shares the same spirit of Goodfellow et al. (2014), respectively. G denotes the generator, Dbd and Dgd are the discriminators of Bad-GAN and Good-GAN, respectively. Lorth is the proposed orthogonal loss to increase the diversity from the angular perspective.
The loss of Good-GAN discriminator is defined as: (Goodfellow et al., 2014): LgoodD = BCE(Dgd(xa),0) + BCE(Dgd(G(z)),1) (7)
The loss of Bad-GAN discriminator is adapted from Ngo et al. (2019) by adding a third term:
LbadD = BCE(Dbd(xn),0) + γBCE(Dbd(G(z)),1) + ηBCE(Dbd(xa),1) (8)
where xn and xa are normal (inlier) and anomaly (outlier) cases, respectively. The third term includes real anomalies to train the discriminator of Bad-GAN. η = 0 when no anomalies are available. Note Vbd and Vgd (from Eq. 5) are composed of {LbadG, LbadD} (from Eq. 6 and Eq. 8) and {LgoodG, LgoodD} (from Eq. 6 and Eq. 7), respectively. The procedures of Taichi-GAN are illustrated in Fig. 3(b). The unsupervised training setting (no real anomalies) is performed with only the black text components (Bad-GAN + Lorth). The red text components are added when few anomalies are available.

heading: 4 EXPERIMENTS
text: In addition to 2D synthetic data in Fig. 2, we involve five datasets to evaluate our method for AD. MNIST, Fashion-MNIST, and CIFAR10 are popular in previous literature (Ruff et al., 2020b; Ngo et al., 2019; Lee et al., 2018). HAM1000 and BraTS20 are medical datasets where the anomalies are well defined clinically. The Area Under the ROC Curve (AUC) (Fawcett, 2006) is used to evaluate the AD performance. The reported AUCs are average values over three independent runs of each experiment using different random seeds. Detailed network architectures, dataset splits, dataset sizes, model hyperparameters, and baseline implementations can be found in the Appendix A, B, and E.

heading: 4.1 DATA AND SETUP
text: MNIST (LeCun et al., 1998) contains 10 classes of digits (0-9). As one-vs-rest setting (i.e., one class is inlier and the rest classes are outlier) in MNIST is proved to be easy (Ruff et al., 2018; 2020b), we
consider the N-vs-rest setup. Class 0-2 are formalized as normal set and class 3-9 are formalized as anomaly set in our main setting (Table 1). Ablation studies considering different “N” and different amount of labeled anomalies are shown in Table 2 and Table 3, respectively.
Fashion-MNIST (Xiao et al., 2017) & CIFAR10 (Krizhevsky, 2009) use one-vs-rest setting that we select class “T-shirt” and “Airplane” as the normal class and the rest as anomalies, respectively.
HAM10000 (Tschandl et al., 2018) is a clinical-based dataset consisting of ∼10000 dermatoscopic images with skin lesions of seven classes. The dominated class Melanocytic nevi ( >70%) is a common benign skin lesion which is set as inlier and the rest as outliers.
BraTS20 (UPenn, 2020) is a recent version of BraTS challenge (Bakas, 2018). Three types of anomalies are included: the GD-enhancing tumor, the peritumoral edema, and the necrotic and non-enhancing tumor core. Here we simplify the problem to image-level disease detection motivated by (Siddiquee et al., 2019) in the AD setting. We extract the 2D slices from T2-FLAIR volumes, where slices with anomaly annotations are considered as outliers and the rest as inliers. Special handling and generated pseudo anomaly examples on BraTS20 can be found in Appendix D.
Labeled Anomalies for Training. We categorize our experiments on each of the five data collections into two folds, i.e., “No anomalies available” and “Few anomalies available” (see Table 1). For the latter fold, we include the following numbers of anomalies: MNIST, one sample per class in 3-9 classes; Fashion-MNIST & CIFAR10, one sample per class in classes except “T-shirt”; HAM10000, five samples per class except “Melanocytic nevi”; BraTS20, 300 slices with tumor from 5 scans.
Methods for Comparison. When no anomalies are available, the benchmark methods include OC-SVM (Schölkopf et al., 2001) (shallow AD model), Deep SSVD (Ruff et al., 2018), F-AnoGAN (Schlegl et al., 2019) (reconstruction-error-based model), original badGAN (Dai et al., 2017), and FCDD (Liznerski et al., 2021). When few anomalies are available, representative methods using outlier exposure or semi-supervised learning are included for comparison. FCDD+OE (Liznerski et al., 2021) integrates outlier exposure (OE) (Hendrycks et al., 2019) technique to utilize available real anomalies. Deep SAD (Ruff et al., 2020b) generalizes the Deep SVDD to semi-supervised setting. Two other naive approaches are included for benchmarking: (1) training a classification model using BCE regardless of limited anomalies, and (2) involving the available anomalies in the Bad-GAN discriminator training, i.e., Bad-GAN (semi).
Variants of Taichi-GAN. We perform experiments on the following settings: 1) Bad-GAN only uses FenceGAN, 2) Bad-GAN+Lorth includes orthogonal loss regularization, 3) Bad-GAN(semi) trains the discriminator with few labeled anomalies besides the normal samples and generated samples, 4) Bad-GAN(semi)+Good-GAN involves the Good-GAN training for sample generation on top of setting 3), 5) Taichi-GAN is our ultimate approach, equivalent to Bad-GAN(semi)+Good-GAN+Lorth. Different settings serve as an approach-level ablation study.

heading: 4.2 BENCHMARK RESULTS
text: The AD performances on five datasets are reported in Table 1 across all benchmark methods and approach-level ablation settings described in Section 4.1. The proposed methods achieve the highest AUCs in both folds where No and Few anomalies available. Across all five datasets, we observe consistently that 1) the improvement from introducing orthogonal loss to Bad-GAN, 2) the improvement from leveraging few anomaly samples with a Good-GAN, and 3) Taichi-GAN including Good-GAN, Bad-GAN, and orthogonal loss achieves the best performances.
Generated pseudo anomalies are shown in Fig. 4 and appendix Fig. 6 (bottom left). With our orthogonal regularization, the generated samples are more diversified (3rd row vs. 2nd row). When few anomalies are available, by incorporating the Good-GAN, the generated samples look closer to real anomalies and maintain reasonable diversity (indicated by green boxes in the fourth row).
Benchmark Analyses. When no anomalies are available, increasing the diversity of generated samples with the proposed Lorth effectively address Dis1 (e.g., +5.6% AUC in MNIST and visual contrast of 3rd row vs. 2nd row in Fig. 4). Our proposed model achieves even better performance by integrating the Good-GAN to guide the pseudo anomaly generation in the Bad-GAN framework, and demonstrates the best effectiveness of consuming the limited labeled anomalies. By doing so, Dis2 and Dis3 are effectively addressed.

heading: 4.3 ABLATION STUDIES
text: In addition to the approach-level ablation study mentioned in Variants of Taichi-GAN, three more ablation studies have been conducted: 1) different weights δ between the Good-GAN and Bad-GAN, 2) different definitions of inlier/outlier, and 3) different availabilities of anomaly samples.
Fig. 5 shows results using different weights δ on MNIST and Fashion-MNIST. Note compared to Bad-GAN (semi), the Bad-GAN represents that the real anomalies (even when available) are not included in training Dbd, thus the comparison between Bad-GAN+Good-GAN and Bad-GAN can better highlight the contribution of Good-GAN to AD tasks. Table 2 shows results of different definitions on inlier/outlier. The results of different available anomaly numbers/styles are shown in Table 3. Based on those ablation studies, we have further findings and analyses as follows:
(1) As in Fig. 5, both too large and too small δ decrease the performance of the Taichi-GAN. Generated pseudo anomalies can over-fit to few real anomalies (with limited patterns) with too large δ. Contrastingly, too small δ decreases the efficacy of the Good-GAN (converge to Bad-GAN only).
(2) Bad-GAN(semi) includes real anomalies in training Dbd where the discrimination performance can be considerably affected by real anomalies, which does not directly show the efficacy comparison of generated pseudo anomalies across methods. By contrast, the value of the Good-GAN are more
clearly highlighted in settings with Bad-GAN as in Fig. 5 (e.g., +9.0% AUC in MNIST when adding Good-GAN to Bad-GAN, i.e., δ = 0.1 vs. δ = 0 in Bad-GAN+Good-GAN).
(3) Results from Table 2 prove that our contributions of the orthogonal loss and the integration of Good-GAN make improvements to the baseline Bad-GAN consistently, while the degree of improvement varies in different settings and datasets (also supported by Table 1).
(4) When fewer anomalies are available (e.g., settings of “[3-6]*1” and “[3-9]*1” in Table 3), the performance gain from Lorth and Good-GAN are more substantial (+ >4% AUCs).
(5) As in Table 3, with increasing number of real anomalies (e.g., “[3-9]*20”), we find 1) Bad-GAN (semi) can be less effective than BCE, and 2) the Lorth does not necessarily improve the performance. This indicates when available anomalies can already cover a reasonable number of relevant patterns, the generated samples from pure Bad-GAN could introduce noise and thus decrease the discrimination power of Dbd. The Lorth is built upon Bad-GAN by increasing the angular diversity of generated samples, and therefore also less effective in this scenario. This could be a limitation of the full-blown Taichi-GAN when sufficient diversity is already covered by the real anomalies. Yet, as a variant of Taichi-GAN, Bad-GAN(semi)+Good-GAN, remains most effective in this experiment.

heading: 5 CONCLUSION
text: In this paper, we address two key aspects in the Bad-GAN framework for AD that 1) generated pseudo anomalies from the Bad-GAN should have high diversity, 2) available anomalies (even few) can be used to guide pseudo anomaly generation. We propose a new Taichi-GAN that balances a “Good-GAN” with a “Bad-GAN” during training. Evaluated on five datasets, our model achieves the most promising performance in AD tasks compared with representative state-of-the-art methods. Instead of simply combine two types of GANs, Taichi-GAN value lies in the data efficiency as below.
Further Discussion. The key point of end-to-end anomaly scoring is to train an effective classifier (i.e., the discriminator of Bad-GAN) to distinguish unseen inliers and outliers, under the contexts that inliers are relatively sufficient while no or few real outliers are available during training. Therefore, the management of pseudo outlier generation is critical. When no outliers are available, the Bad-GAN is trained to generate peripheral samples of the inlier distribution as pseudo outliers. We validate in our experiments that higher sampling diversity achieved by the proposed orthogonal loss improves the quality of pseudo outliers towards a better classifier in inference. When few outliers are available, we demonstrate that the most effective way (compared with existing benchmarks with empirical experiments) of leveraging the available outliers is through our proposed Taichi-GAN, which enable the guidance from real outliers to the generation of pseudo outliers (e.g., the pseudo outlier boundary gets “thicker” near real outliers in Fig. 2(d)). We believe the core ideas of this work can be extended to other unsupervised or highly imbalanced classification tasks.
Ethics Statement. The concepts and information presented in this paper are based on research results that are not commercially available. Future commercial availability cannot be guaranteed.

heading: A BASELINES AND HYPER-PARAMETERS
text: For baselines of benchmark Table 1, we adapt their official public code into our datasets/settings:
OC-SVM: https://github.com/lukasruff/Deep-SVDD
Deep SSVD: https://github.com/lukasruff/Deep-SVDD-PyTorch
Original BadGAN: https://github.com/kimiyoung/ssl_bad_gan
FCDD, FCDD + OE: https://github.com/liznerski/fcdd
Deep SAD: https://github.com/lukasruff/Deep-SAD-PyTorch
FenceGAN: https://github.com/phuccuongngo99/Fence_GAN
For the Variants of Taichi-GAN mentioned in Section 4.1, we share the same network structure and hyper-parameter for the fair comparison. We did not include additional data augmentation across all compared methods. Our experiments are conducted with PyTorch 1.1.
Our hyper-parameters are highly motivated from FenceGAN (Ngo et al., 2019), as we use FenceGAN as the “Bad-GAN” in our model. We follow the hyper-parameters of FenceGAN paper in MNIST, i.e., α = 0.9, β = 30, γ = 0.5 for Eq.(6-9). Note that FenceGAN set α = 0.1 for MNIST since the normal cases are defined as 0 and anomalies are defined as 1 in their paper, which is equivalent to α = 0.9 in this paper. For the balance hyper-parameter δ between “Bad-GAN” and “Good-GAN”, we set δ = 1 for MNIST, δ = 0.1 for Fashion-MNIST, HAM10000 and BraTS20. An ablation study of different δ has conducted in Section 4.3. If not specifically noted, the hyper-parameter η is set to 0.1.

heading: B DATA SPLITS
text: We use the same data splits and the same evaluation criteria for all compared methods for fair comparison. Note that the splits and criteria in some of original official Github are different, we have made necessarily adaptions about the data loader and evaluation checkpoint. The AUC results are reported on test sets. Note that all normal samples in the training set will be used for training, while no or few anomalies are included depending on settings. The general introduction of datasets can be found in Section 4.1. The following is about data splits:
MNIST & Fashion-MNIST: Original training set of MNIST contains 60000 samples from 10 classes. We randomly split the original training set as 90% / 10% for training and validation splits. We use the original test set as our test set, which contains 10000 samples. The AUCs reported on test set and the model selected based on the performance of validation set.
CIFAR10: Original training set of MNIST contains 50000 samples from 10 classes. We randomly split the original training set as 90% / 10% for training and validation splits. We use the original test set as our test set, which contains 10000 samples.
HAM10000: The HAM10000 consists of 10015 dermatoscopic images can be used for academic machine learning purposes. We split the HAM10000 dataset into training/validation/test split is 7001/1001/2013 samples for the AD task.
BraTS20: All the data in our experiments are from 369 patients of original BraTS20 training set. The training/validation/test splits are 12534/1351/2694 slices extracted from 280/30/59 patients.

heading: C USED METRIC DISCUSSION IN THIS PAPER
text: We use the Area Under receiver operating characteristic Curve (AUC) to evaluate the model’s performance. We want to have an overall discrimination performance measurement to benchmark. AUC is known for the discrimination comparison across models and widely used in related references such as Deep SVDD and FCDD. When working on a specific application (which is not the target of this paper), for example, when there is a preference on less false positive rate or false negative rate, then precision / recall based metrics might be more preferred.

heading: D SPECIAL HANDLING OF BRATS20
text: Different from other three datasets, the intra-class variations can be larger than inter-class variations at pixel-level in BraTS20 (as the case in upper left Fig. 6), which increase the difficulty to learn a compact normal representation. Instead of feeding original image to AD detection models, we create processed different maps with F-AnoGAN (Schlegl et al., 2019). The advantages of creating the difference map include that all normal samples have the same ideal difference map (i.e., all-zero matrix), without the need to capture patterns of original images.

heading: E NETWORK STRUCTURES
text: We adapt the demonstration style of Liznerski et al. (2021) to show network structures in different datasets. Table 4, 5 show the generator and discriminator of MNIST and Fashion-MNST. Table 6, 7 show the generator and discriminator of HAM10000. Table 8, 9 show the generator and discriminator of BraTS20.

Review summaries:
The authors of this paper propose a GAN-based method for the anomaly detection task. Their method relies on so-called Bad GANs (that uses the trained GAN discriminator to distinguish between inliers and pseudo-anomalies), a new orthogonal loss (that favorises generated samples to cover a larger angular space) and eventually a so-called Good GAN (which generator is shared with the Bad GAN; and discriminator is trained on "real" anomalies). The method is evaluated along a set of baselines in experiments that also include ablation studies. 

In this paper the authors propose a method for image anomaly detection (AD) based on GANs. The use the GAN discriminator based on a method called "Bad GAN" where one trains a GAN to produce low likelihood samples (trained using the nominal data) and then use the discriminator, which has been generalized to a large class of samples, to determine if a test sample is anomalous. In the proposed method the authors include $L_{orth}$ to enforce angular diversity of generated samples, a "good" gan discriminator that is trained to correctly determine if a sample is generated, and a bad gan discriminator inducing the generator to make points near the boundary of the nominal set distribution. At test time the "bad" discriminator used to determine an anomaly score. The authors test their method experimentally on F/MNIST and some medical datasets.

The paper proposes a new interesting method for anomaly detection. In particular, to overcome the limitations of existing Bad-GAN, the authors introduce the orthogonal loss to regularize the generation of anomaly samples to be distributed evenly at the periphery of the training data. Furthermore, in the scenario of available anomalies, the authors combine Bad-GAN and Good-GAN together, in which Good-GAN learns to generate the anomalies while Bad-GAN reguralizes the anomaly pseudo anomalies at the boundary of inlier distribution.