Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 4: Ok but not good enough - rejection
review_content: The proposed method works as follows. Given samples are partitioned into two parts; one is for classifier training and the other is for data synthesizer training. Both are trained in a differentially private manner. After training, the DP synthesizer generates samples and the DP classifier labels them so that the resulting samples can be used as training samples. By the post-processing theorems, the resulting are differentially private, which are published as synthesized samples.

The idea is interesting, simple, and unique. Also, experimental results demonstrate that the  models trained with the proposed method give s better F1 score compared to the existing methods. One limitation of this manuscript is that the reason why the proposed scheme can give better classification accuracy is not discussed. Also, the reason why the RMSE of the regression model trained with this scheme is worse than other methods is not examined, either. One quick thought is that the proposed scheme preserves the cluster structure of the samples well and therefore the classification model trained with the resulting sample has good accuracy. In contrast, the metric structure behind the samples is not preserved well and therefore the regression model does not have good RMSE. I am not sure this is correct or not, but anyway, I think further consideration on these issues will be interesting and needed for this type of experimental study to find a clue to improve data synthesization with DP guarantee.
  
Minor:
FIg 6 is in the Appendix, not in the main body. Also, many important claims (mainly in experimental results) are given with results in the Appendix. The main claim should be constructed with the contents in the main body. 

 






review_title: The idea is interesting and simple whereas no clear reason that the proposed method works well is not explained.

final_score: rating: 4: Ok but not good enough - rejection
review_content: The authors proposed QUAIL, an algorithm that uses a supervised model and a synthetic data model to generate synthetic data that is good for downstream tasks. It also shows some empirical evaluations of the algorithm.

The technical part, especially the experiments, might need some improvement. The main issue is that we cannot draw a clear conclusion from the experimental results. I would suggest taking one or two epsilon values, and look at the results in more detail to see if we can find any trend. Also, generating synthetic data under differential privacy is not an easy task, so I think it's ok to skip the regime where epsilon < 1, or try even larger epsilon to get a reasonable utility first.
However, I would like to say that I very much appreciate the authors' effort in conducting experiments on quite a few datasets, and their integrity in presenting all the results no matter positive or negative.

The paper can be better organized, for example,
- The notations in the algorithm should be clearly explained/defined before the algorithm (for example, N and X), and some intuition can be added after the algorithm description. In the algorithm description itself, I think maybe it's clearer to defined the target dimension and the rest of the dimension separately, i.e. defining one sample as (feature, target) and later on we will have (synthetic feature, synthetic target). In the "split" part, I guess eps_C should be (1-p)*eps.
- The part below Theorem 3.1 might better be put into the experiment section than the algorithm section.
- In quite a few figures, we see interesting trends like some algorithm can have worse utility as epsilon grows. So it might be important to report the standard deviation of the algorithm for readers to better understand what was going on. Also, the texts in the figures can be made larger.
- The paper called the algorithm "ensemble method". I feel like ensemble means something specific in ML, and simply using two different models together doesn't quite seem like ensemble. Maybe I'm understanding something here but it should be better explained.

review_title: Need some improvement

final_score: rating: 4: Ok but not good enough - rejection
review_content: The paper proposes QUAIL, an ensemble of a generative model and a classifier, where both are trained with differential privacy, in order to generate a differentially private dataset (from the generative model) and a label vector (from the classifier). The paper further compared QUAIL with "other" differentially private generative models based on conditional GAN.

My main concern with the paper is that there is no clear contribution. It is hard to judge whether the main claim is that the paper presents a survey of the current differentially private generative models, in which case, the survey part is very short and not in-depth. Or is that the the paper proposes QUAIL, which in many cases performs worse than other generative models. My some other concerns are detailed below:

- There is no mention of \delta used for (\epsilon,\delta) - differential privacy
- Please use something else than \delta for difference between performance measures as it can get confusing
- As the version of GAN used in the paper is conditional, shouldn't the generator be differentially private as well? i.e. how are we protecting the privacy of the labels?
- There is a mention of PATECTGAN performing better even compared to the non-noisy model trained on real data, how is this possible, please add some explanation.

review_title: Review

Meta-review: The paper surveys existing differentially private data synthesis
methods, and introduces an algorithm that learns both a generator and
a classifier in a differentially private mode.

The problem is highly timely and important. Results are promising.

Main remaining concerns after discussion between the reviewers and the
authors are:

- reason why the proposed scheme can give better classification
accuracy, should be clarified more

- unclarity on conclusions that can be drawn from the experiments. The
revised version has improved on this somewhat.

One explanation for the problems was suggested to be that the paper
tries, at the same time, to both present a new method and be a
survey. Is hard to do in a short paper, and as a result, the paper
lacks focus. At the very least, more work is needed.

The authors are encouraged to continue their work on this
important problem, and the review comments hopefully help in that.

Decision: Reject