Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: In this paper the authors propose a method for image anomaly detection (AD) based on GANs. The use the GAN discriminator based on a method called "Bad GAN" where one trains a GAN to produce low likelihood samples (trained using the nominal data) and then use the discriminator, which has been generalized to a large class of samples, to determine if a test sample is anomalous. In the proposed method the authors include $L_{orth}$ to enforce angular diversity of generated samples, a "good" gan discriminator that is trained to correctly determine if a sample is generated, and a bad gan discriminator inducing the generator to make points near the boundary of the nominal set distribution. At test time the "bad" discriminator used to determine an anomaly score. The authors test their method experimentally on F/MNIST and some medical datasets.
main_review:  * The problem addressed by the authors is important and GAN based AD does seem appropriate for certain datasets. Experimentally, a lot of the experimental improvements are quite small. 
 * It would be interesting to see how the method performs on CIFAR10 one v rest, since that seems to be a bit of a standard dataset and fairly challenging for generative model based AD. 
 * The paper lacks any theory.
 * The lack of theory is compounded by the rather small experimental setup. Lacking any theory a paper should have very strong experimental support.
 * Clarity is pretty good.
 * Good lit review.

Overall the paper is okay, I don't think the results are impressive enough, or that the proposed method is novel enough for a top tier conference.
summary_of_the_review: Overall the paper is okay, I don't think the results are impressive enough, or that the proposed method is novel enough for a top tier conference.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a new interesting method for anomaly detection. In particular, to overcome the limitations of existing Bad-GAN, the authors introduce the orthogonal loss to regularize the generation of anomaly samples to be distributed evenly at the periphery of the training data. Furthermore, in the scenario of available anomalies, the authors combine Bad-GAN and Good-GAN together, in which Good-GAN learns to generate the anomalies while Bad-GAN reguralizes the anomaly pseudo anomalies at the boundary of inlier distribution. 
main_review: **Pros**
 
1. The paper addresses how to generate pseudo anomalies to improve the anomaly detection. For me, the application of orthogonal loss into the problem itself looks simple but effective. 
 
2. The proposed TaiChi-GAN, which combines Good-GAN, Bad-GAN and Orthogonal loss,  is novel for capturing inliers and generating better pseudo anomalies. Good-GAN is sort of the regularization to Bad-GAN regarding the learning of the anomaly distribution. The design for the framework is reasonable and interesting. 
 
3. This paper provides comprehensive studies and experiments, including both qualitative analysis and quantitative results, to show the effectiveness of the proposed framework. 

**Cons**

1. However, I have concerns regarding how Good-GAN learns to generate “pseudo anomalies to resemble the reference distribution from given real anomalies”. Not sure how the authors verify this claim, e.g., when looking at Fig. 2d? 
 
2. It is good to have studies over new hyper-parameters introduced in the framework, e.g., \gamma, \eta. 
 
3. MNIST and Fashion MNIST are not too challenging datasets, so it would be good to have experiments in e.g., CIFAR, SVHN like Bad-GAN and GAB-based anomaly papers?
 
4. May consider citing missing GAN works on anomaly detection (ideally comparing them) in the paper: 
[1] DOPING: Generative Data Augmentation for Unsupervised Anomaly Detection with GAN
[2] Efficient GAN-Based Anomaly Detection
[3] GAN Ensemble for Anomaly Detection
summary_of_the_review: Overall, I vote for accepting. I like the idea of combining Good-GAN and Bad-GAN together, which is novel and interesting to me. The proposed orthogonal loss is simple and effective as being used as the complementary of the existing dispersion loss.  The results on benchmark datasets shown in the paper are encouraging and outperforms many existing works in both scenarios without or few anomalies observed.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The authors of this paper propose a GAN-based method for the anomaly detection task. Their method relies on so-called Bad GANs (that uses the trained GAN discriminator to distinguish between inliers and pseudo-anomalies), a new orthogonal loss (that favorises generated samples to cover a larger angular space) and eventually a so-called Good GAN (which generator is shared with the Bad GAN; and discriminator is trained on "real" anomalies). The method is evaluated along a set of baselines in experiments that also include ablation studies. 
main_review: Overall the paper introduces a new method that is of interest for the machine learning community.

Strengths:
- The paper is relevant to this conference and tackles an important research problem
- The related work is well presented in this paper and introduces well different anomaly detection methods
- The contributions of this work are motivated

However, my two main concerns are on the experimental methodology and on a less important manner on the writing. 

Weaknesses/ Experiments:
- Dataset generations: it is unclear how the inlier/anomaly sets were selected; on MNIST Table 1, how would the methods perform for other combinations of "normal" sets and anomalies (than the 0-2 normal and 3-9 outliers)? on Fashion MNIST what does we "randomly" select the T-shirt as normal class mean? Would you attain similar results for other configurations? Why did you change these configurations for the ablation study ? On the labeled anomalies for training how did you decide to include 1 sample per class for MNIST/ Fashion MNIST and 5 per class for HAM10000, etc... ? 
- Evaluation metrics: the authors provide results using AUCs (from an ROC curve, probably) of the anomaly detection methods, yet conventional methods in the literature also consider PRC curves using precision, recall; and eventually F1 scores. Depending on whether the focus is given on positive or negative classes (which may depend on datasets) in the anomaly detection task, both ROC and PRC curves could be discussed. This point is missing in the paper.
- Model selection: the method uses a set of hyper parameters to combine different losses (delta, Eq. 5; alpha, beta, gamma, etc. Eq. 2, 3; eta Eq. 8) and deep representations with network structures. Even though the paper relies on some previously picked models (information from Appendix A) the model selection procedure is *crucially* lacking in the paper. In Appendix B validation sets are mentioned but how are validated the models? This point is not discussed in the main paper but different hyperparameters are used for different datasets (Appendix A). 
- Statistical significance: Were the experiments randomized and averaged (different random seeds) ? 


Weaknesses/ Writing:
- The writing of the different disadvantages presented in the introduction (with the Dis notations) could be improved; especially Dis 3 seems an artefact of the Dis 2.
- The figures are not always clear; for e.g the orthogonal loss figure does not convey sufficiently the intuition of the orthogonal loss
- Evaluation metrics are not discussed enough; this is crucial in an empirical study paper. 



summary_of_the_review: The paper introduces a novel method that could be potentially interesting should the authors clarify all the concerns raised in the previous paragraph and provide extensive experiment results to address them. In the current state and given all the concerns on experimental methodology, I would recommend the rejection of the paper with regards to the standards of the ICLR conference.  

UPDATE:

Please see the response I have provided to the authors. I decided to lower my score marginally and I encourage the authors to undertake the modifications suggested by the reviewers for future venues, especially on the empirical methodology and the writing. The paper in its current form may not be ready to my mind.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The authors propose the coupling of coupling two GANs for anomaly detection: one which focuses on generating examples at the periphery of the distribution (hence the name BadGAN) and one that generates reference examples (the GoodGAN portion). These two generative models result from a joint optimization, and are evaluated on several anomaly detection benchmarks in this manuscript.

main_review: # Strengths

The paper is organized well and easy to follow. The ideas to combine GANs of different quality is interesting and novel, however seems to be slightly limited as far as applying them to more complex benchmarks is concerned (see Weaknesses).

# Weaknesses

This manuscript has a couple of issues that need addressing. For one, the experimental evaluation is somewhat limited, in that it focuses on predominantly simple distributions and benchmarks (e.g. MNIST, FashionMNIST). I would recommend that the authors streamline their experiments for multi-modal problems, i.e. those that use multiple classes from MNIST (shown in Table 2 and 3).

In particular, the community has moved toward more demanding benchmarks, for example recent work proposes the use of semantic benchmarks for AD, see Ahmed & Courville (AAAI 2020), where 9 out of 10 classes are used as normal, and only 1 out of 10 is anomalous. Aligning experiments on MNIST with this challenging setting could potentially improve the paper's contribution, and make it a bit clearer how it compares to existing work. In addition, I would however strongly recommend that the proposed method is also evaluated on more complex benchmarks, e.g. CIFAR-10.

Next, the proposed method seems to require the availability of labeled anomalies. This is a very strong assumption for AD, and as "Rethinking Assumptions in Deep Anomaly Detection" of Ruff et al. (2021) clearly shows, often very simple classification-based methods that require only very few labeled anomalies achieve very robust performance. I am afraid this will make readers wonder why they should go through the trouble of training not one but two GANs to determine anomaly.

# Minor

There appears to be a formatting issue with the citation of Varun Chandola (2009).

Two GAN-based anomaly detection works which seem to be missing from this manuscript are:
- A. Berg, M. Felsberg, and J. Ahlberg. Unsupervised adversarial learning of anomaly detection in the wild. In European Conference on Artificial Intelligence, pages 1002–1008, 2020.
- L. Deecke, R. Vandermeulen, L. Ruff, S. Mandt, and M. Kloft. Image anomaly detection with generative adversarial networks. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases, pages 3–17. Springer, 2018.

# Update

My rating has been increased slightly in response to a thorough revision, which now includes CIFAR-10 in the experiments of the manuscript. Concerns remain however with regards to performance versus competitors (such as F-AnoGAN), and because recent benchmarks like semantic AD have not been incorporated fully.
summary_of_the_review: The paper presents an interesting idea that involves multiple GANs, but limits the evaluation to relatively simple benchmarks. Moreover, the proposed method relies on the presence of labeled anomalies, which are very difficult to obtain in practice.
review_title: null

Meta-review: This paper proposes a method that combines Bad-GAN and Good-GAN, in which Good-GAN learns to generate the anomalies while Bad-GAN reguralizes the anomaly pseudo anomalies at the boundary of inlier distribution. In addition, a new orthogonal loss is proposed to  regularize the generation of anomaly samples to be distributed evenly at the periphery of the training data. The proposed method is new and shows some improvement over existing methods.

However, there are some detailed technical concerns raised by reviewers. Some of the concerns still remain unresolved after the discussion. 1) The proposed method lacks a principled way to select hyperparameters. 2) The experimental setting is a bit simple to verify the effectiveness of the proposed method in challenging real world applications. Especially, there is no theoretical guarantee of the proposed method, empirical evaluation is the only way to show the effectiveness of the proposed method. 3) The overall performance improvement is not very significant compared to existing methods. For example, the performance is very close to a method F-AnoGAN published in 2019. Addressing the concerns needs a significant amount of work. Thus, I do not recommend acceptance of this paper.
Decision: Reject