Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 8: Accept
review_content: This paper presents a framework for Differentially private data generation that enables more accurate training of downstream learners without compromising on the privacy guarantees. The key insight is to introduce an ensemble of teacher discriminators in the GAN formulation instead of a single discriminator. The teachers are trained on separate subsets of the private data set. A gradient aggregator is also introduced for transmitting loss signal to the student without losing privacy by using adversarial perturbations and random projections. The authors provide theoretic guarantees of Renyi differential privacy and experiments on Kaggle Credit default dataset and MNIST. 

I have very little knowledge of this field, but the main idea idea seemed quite novel and insightful.  I have not verified the theoretical results closely, I only skimmed the derivations, but what I can understand, it seems reasonable. The experimental results seemed quite good, with improvements across the board. I would have liked to see some metric that  was not just performance of the downstream classifier, is there an intrinsic measure of the generator's performance that makes sense to report?

The presentation was quite reasonable and polished, with very few typos.  My only gripe is that they seemed to spend a little too much space in different sections re-iterating their key contributions, but not enough defining or citing sources for key definitions such as Renyi differential privacy, which would be helpful to a non-expert reader.
 

review_title: Official Blind Review #3

final_score: rating: 1: Reject
review_content: In this submission, the authors propose a method to generate synthetic datasets with privacy guarantee while preserving high data utility. However, the following concerns are important to clarify for the authors:

1) Compared to existing work, the proposed method can preserve high data utility due to the fact that the proposed method doesn't ensure differential privacy for the discriminator. Could the authors provide more details about this key observation and motivation? How about discriminator is not trustable? Can we simply assume that it is safe for discriminator to access sensitive data? How about the discriminator is attacked? I agree for some applications, we can have this assumption about discriminator, but it is very important to better understand the limitation and risk of this assumption. The real-world applications are not simply defined by us.

2) The authors claim that the proposed method is scalable? Could the author confirm this either theoretically or experimentally?

3) The experiments are conducted on a single dataset, simple MNIST dataset. It would be convincing to report experiments on more datasets such as CIFAR-10 and others.

4) For experiment comparison and analysis, the authors adopt quite large epsilon, i.e., \epsilon = 1 and \epsilon = 10. Several existing work adopt \epsilon = 0.2. Can the authors report experiment comparison with such meaningful epsilon?
review_title: Official Blind Review #1

final_score: rating: 3: Weak Reject
review_content: This paper studies the problem of differentially private data generator. Inspired by the general GAN framework and the PATE mechanism, the authors propose a new differentially private training algorithm for data generator. The problem of training data generator with privacy guarantee considered in this paper is very interesting, and the proposed algorithm looks novel. However, there are lots of unclear statements in the current paper, and I cannot tell whether the proposed algorithm is indeed better than previous methods. Following are my major concerns:
1.It is unclear what are the loss functions used in equation (1) and (2). Please define k when introducing equation (2).
2.The training framework introduced in section 3.1 is different from the traditional GAN framework, and thus my concern is that whether this framework will give us good generated samples. Because the performance of GAN has been proved in both theory and practice. The authors should at least empirically show the performance of the proposed framework in the nonprivate setting.
3.There is no introduction of the $(\epsilon,\delta)$-differential privacy before introducing the Definition 1.
4.There is no definition of Renyi differential privacy, so the statement of Theorem 2 is unclear. In addition, what is data-dependent Renyi differential privacy?
5.The privacy guarantee of Algorithm 2 is not very clear. Because there are lots of parameters in Algorithm 1 and 2 which may affect the privacy guarantee, and Theorem 3 does not state such requirements. For example, how to choose $\sigma_1,\sigma_2$? In Theorem 7, there are some constraints on different parameters, will them be satisfied by your algorithm?
6.How will the number of teacher models affect the privacy guarantee?
7.Why you choose random projection matrix with variance $1/k$, and what is the projection dimension for different algorithms?
8.In Table 1, the results of non private GAN are different from the results of non private GAN reported in PATE-GAN paper. Since the baseline results are much better in the current than the results reported in the PATE-GAN paper, it seems to me that the improvements of the proposed method comes from the stronger baseline.

Other comments:
1.$\lambda>1$ in Theorem 3.
2.Algorithm 2 should be moved to main context.
3.The last sentence in section 3.2 is not convincing.
4.Typo “differnet” in the caption of Table 1.
review_title: Official Blind Review #2

Meta-review: This paper addresses the problem of differential private data generator. The paper presents a novel approach called G_PATE which builds on the existing PATE framework. The main contribution is in using a student generator with an ensemble of teacher discriminators and in proposing a new private gradient aggregation mechanism which ensures differential privacy in the information flow from discriminator to generator.

Although the idea is interesting, there are significant concerns raised by the reviewers about the experiments and analysis done in the paper which seem to be valid and have not been addressed yet in the final revision. I believe upon making significant changes to the paper, this could be a good contribution. Thus, as of now, I am recommending a Rejection.
Decision: Reject