Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: In this paper the authors propose a method for improving "model compatibility" in GANs. For this reason they add to the loss of the generation procedure a term that depends on the maximum mean discrepancy between the following datasets: (1) the output of a classifier with input the real dataset, (2) the output of the same classifier with input GAN-generated samples. They authors show that in essentially all the datasets they tried, the model compatibility of the produces generator is increased after adding the aforementioned cost, while the visual quality of the data is not decreased.

Strengths:

- The low model compatibility of GANs is a very important disadvantage and hence improving this aspect of GANs is a relevant problem.

Weaknesses - Comments:
A. The increase in the model compatibility is very mild. Especially in CIFAR-10, the increase in very small.

B. In MNIST the increase is larger than CIFAR-10 but the initial model compatibility using vanilla GANs is smaller. The reason might be that for MNIST much simpler classification algorithms have been used. This maybe suggests that the proposed method affects more the model compatibility of methods that achieve lower model compatibility in before the addition of the extra cost term.

Minor Comments:
1. In equation (1) it looks strange that the summation is over A but A does not appear at all in the summand. I suggest you replace h and h' with A(D) and A(D') so that this is clear.
2. In Theorem 2, \hat{L}_G is used but for the proof the authors have replaces \hat{M} with M. There should be a comment for that. In general I believe that Theorem 2 is almost trivial and does not add value to this clearly experimental paper.
review_title: Official Blind Review #4

final_score: rating: 3: Weak Reject
review_content: This paper aims at training a GAN that can generate data matches the real data distribution well especially at the boundaries of the classifiers. A Boundary-Calibration loss (BC-loss) base on multi pretrained classifiers is introduced to match the statistics between the distributions of original data and generated data. The motivation is interesting. The story is clearly explained. However, the experiments part is weak.  

There are several typo and mistakes. The experiments only show that the proposed method got a good performance, but the analysis of the reason is not shown. The reason to name the loss as Boundary-Calibration loss (BC-loss) should be explained and the experiments should show some effect on the boundary areas. Some concerns are listed below,

1.	If the generated dataset exhibits very good property as the real dataset, it means the data is to some extent perfectly foreseen, and there is little to no privacy, is it contrary to the aim of not leaking the real dataset?
2.	It is more interesting to see the difference between the distribution of real data and the generated data, However the author only show a simple toy data distribution comparison, I would like to see more comprehensive results about the distribution differences on real dataset , e.g. the TSNE embedding? 
3.	Equation (3) is not correct.
4.	The author said that image quality of MNIST and CIFAR10 are not improved, then why the classification results are improved?  there should have some differences existed among different compared methods, it would be more convincing if you can show it out.
5.	What kind of  generator do you use for the UCI data?  How do you settle the output problem?  Since some of the data are continuous and some are discrete.

review_title: Official Blind Review #2

final_score: rating: 3: Weak Reject
review_content: In this work authors consider a problem of 'model compatibility' of GANs, i.e. usefullness of the generated samples for classification tasks. Proposed 'Boundary Calibration' GAN attempts to tackle this issue by adding non-adversarial terms to discriminator, obtained as outputs of the classifiers trained on the original data. For evaluation, it is proposed to compare accuracies obtained by classifiers trained on generated and on real data (termed 'relative acurracy'). Experiments show that the proposed methods improve such scores.

Pros:
- considered problem seems to be important GAN application which has not yet received too much attention.
- proposed method seems to improve the accuracy of classifiers trained on generated data.

Cons:
- the paper is poorly written, has multiple typos and often it is unclear what authors mean.
- the proposed evaluation does not exactly measure the potential improvements from training classifiers with generated data. It would make sense to provide information if generated data can improve classifier's scores, if added to the training data (or small part of it).
- it is unclear whether or not the proposed technique does not affect the sample quality, as no quantitative metrics are provided.

Details:
1. Quantitative scores for quality of generated samples are not provided. Authors instead provide few samples and state that it is difficult to detect the difference. Although sample quality is not the main task here, it certainly is important - otherwise we could train generators solely against the classifier 'boundary calibration' loss terms - this, however would likely lead to adversarial examples. Combining two losses often leads to trade-offs, hence showing that we can improve 'model compatibility' without the loss of sample quality is actually crucial. Metrics such as Inception score [1], FID [2] or KID [3] are desired, especially given relatively poor quality of cifar and mnist samples from all models.
2.The main metric used is the ratio between accurracies of classifiers (trained on real and generated data). It is hence difficult to tell if the classifiers that were used were trained reasonably and achieved reasonable scores.
3. In the abstract, authors claim that 'GANs often prefer generating easier synthetic data that are far from boundaries of the classifiers'. Although for some GAN settings the generators might be biased to do so, in general this claim is unfounded, as GANs optimize divergences that are agnostic to classifier boundaries.
4. It is unclear what kind of classifier-output is used as an input to MMD. Are these continuous logits, discrete class numbers, or one-hot-encoded class identities?
5. Authors use WGAN and MMDGAN with gradient penalty. It is unclear how gradient penalty is applied to MMDGAN as what should be penalized is the witness function, which is different than in WGAN-GP [4], see e.g. [3].
6. It is unclear how embeddings of class information are concatenated to discriminator inputs (p.5).
7. It is unclear to what extent feature selection is deterministic. Authors argue in Section 5.4 that the intersection of top-k features selected from two models should be large. It would be good to provide the same statistics for features selected twice on the same sample.

Overall, the paper currently does not match the quality requirements of ICLR, however it has potential for improvement if the mentioned issues are addressed.

Typos/unclear expressions:
[p1] 'may not willing' >> 'may not be willing'
[p1]'with the property similar to the original data is demanding'  >> properties, demanded/in demand
[p2] 'Although GANs are versatile as aforementioned' - strange wording
[p2] 'The pioneered work' >> 'pioneering work'
[p2] 'information of models' >> 'information from the models'
[p2] effects >> affects
[p3] related works >> related work
[p3] distribution of label >> distribution of labels
[p3] 'generated dataset adopt ' >> 'generated dataset will adopt '
[p3] 'To known about the boundary' >> ' To know the boundary'/'To include the information about the boundary'
[p3] 'the a distance'
[p3] 'the problem to distinguish whether two sets of samples' 
[p4] 'If they are close the sets might be sampled from the same distribution' (?)
[p4] tries to minimized the MMD
[p4] would not leads to
[p6, Table 2 caption] number of estimator used
[p6] at Appendix
[p6] depresses the model compatibility (?)
[p6] can providing
[p7] to known how
[p8] our work open >> our work will open


------------------------------------- Revision -------------------------------------

Although some issue seem to have been clarified, two of my main concerns, i.e. the proposed evaluation and the text quality, have not been resolved. Additionally, as pointed out by reviewer #4, the results seem somewhat incremental. For these reasons, I decided to keep the rating unchanged.
review_title: Official Blind Review #1

Meta-review: The paper presents a method for increasing the "model compatibility" of Generative Adversarial Networks by adding a term to the loss function relating to classification boundaries. The reviewers recognized the importance of the problem, but several concerns were raised about the clarity of the paper, as well as the significance of the experimental results.
Decision: Reject