Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: Synthetic data generation is a key application of differentially-private generative models---it allows you to build a synthetic dataset that can be used repeatedly without privacy risks for downstream ML or data science use cases. This paper presents an empirical evaluation of different DP generative models on mainly tabular datasets, e.g., studying how well they recreate the underlying data distribution and the various training costs.
strength_and_weaknesses: Strengths:
* This is very much a "in-the-weeds" paper that compares various methods on different metrics and datasets. I think some of the resulting findings will be useful for practicioners, e.g., runtime of various methods as you vary the number of features and number of rows. I also think for researchers developing new DP generative models, it'd be useful for them to evaluate their models on metrics like the ones proposed here.
* I find it interesting that there are cases when using more training data can sometimes hurt. It adds to the growing literature that tuning hyperparameters when training with DP can be quite difficult, non-intuitive, and not in line with what works best for non-private ML.
* I like seeing downstream comparison of methods from different model classes, e.g., deep generative models trained with DP-SGD versus graphical modeling approaches.

Weaknesses:
* One concern I have is that some of the "main findings" of the paper are actually just statements about the properties of the methods themselves (see introduction and RQ1/RQ2 in Section 4.5). For example, the claim that "deep generative models spend their budget per training iteration and can handle much wider datasets but become slower with more data" can be made without an empirical evaluation---the method has those strengths/weaknesses because of the very nature of how DP-SGD and neural nets work.
* While I think tabular datasets are important to study for a wide variety of real-world settings, 4/6 of the tasks in this paper are variants of sampling from a gaussian distribution.
clarity,_quality,_novelty_and_reproducibility: * I think the evaluation is decent but could be expanded to a wider set of tasks, including a wider set of tabular datasets, e.g., the entirety of the UCI ML repository. 
* The work is fairly original to my knowledge, it presents a wide empirical evaluation. 
* The writing is overall somewhat clear, but it is a bit repetitive between Sections 1, 4.5, and 5. It also spends a lot of the space introducing all of the methods compared.
summary_of_the_review: I think some of the findings can be useful to a broader community, but overall I find the evaluation and takeaways to be too limited to warrant a publication.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: In this paper, the authors rigorously study the effects of varying N (number of samples) and d (dimensionality of the dataset) for 4 separate private generative modeling approaches (belonging to two different classes of algorithms). This is done at different privacy levels (epsilon) and the evaluation is based on both generative modeling metrics (correlation/mutual information between variables) as well as classification performance. As a result of this exercise, the authors are able to report some unique behaviors of different private generative modeling approaches. 
strength_and_weaknesses: Strengths: 
1) The experimental setup used by the authors is very rigorous 
2) The question of how different generative models differ as a function of N, d and epsilon are somewhat interesting questions.

Weakness: 
1) Beyond highlighting some of the differences between the 4 generative modeling approaches, I am struggling to find the novel contribution of the paper. 
2) The title of the paper and the claim that it examines how the privacy budget is being utilized is quite misleading in my opinion. 
3) The authors only consider models for tabular data generation here but the paper could be significantly improved by considering models that generate more complex data modalities e.g. images. In the case of images for instance, one might be interested in wondering how the difference in learned convolutional filters as a function of N, d and epsilon. 
clarity,_quality,_novelty_and_reproducibility: While the paper is generally well written and easy to follow, the figures are uniformly too small with too much information compressed into them. 
summary_of_the_review: While I like the rigor of the evaluations done by the authors, I believe there aren't very many novel generalizable insights that come from the current version of the paper. As such, it reads more like a workshop paper at the moment. However, if the authors were to expand the paper to models for more complex data modalities, I believe it might make for an interesting conference paper.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: the paper benchmarks how various existing approaches for the problem of *DP synthetic tabular data generation* perform as the number of features and instances of the dataset change. in particular, the paper answers some previous under-studied research questions such as 1) how do methods scale with the number of columns. 
strength_and_weaknesses: strength: the paper is empirical in natural and aims to answer concrete practical questions. in this regard, the paper scores nicely as the experiments on evaluating performance is somewhat comprehensive in my opinion. 

weakness: i see two major drawbacks of this work, despite of its wide-covering experiments. 

1) many discoveries in this paper are rather intuitive and follows naturally from inspecting the algorithms. e.g., "PrivBayes' performance on downstream tasks degrades when a stricter privacy budget is imposed..." is the expected and much empirical verification is given in the original PrivBayes paper [1]. in this regard, there are only few truly novel empirical observations uncovered in the current work. 

2) the work uncovers two (somewhat) unexpected results in my opinion, but falls short in helping us understand how these empirical phenomena arise. e.g., on page 9, the paper suggests that "we observed that more data does not always translate to improved quality for all models and evaluation." this is quite unexpected for DP machine learning, as workflows as such tend to be bottlenecked by fitting the training data (as opposed to combating overfitting). it'd therefore be useful to design and run experiments to understand why such observations are the case. for instance, are the results due to experimental noise? are the non-monotone results w/ GANs due to bad optimization? understanding why certain counter-intuitive behavior emerges would help future research in developing better methods.


[1] Zhang, Jun, et al. "Privbayes: Private data release via bayesian networks." ACM Transactions on Database Systems (TODS) 42.4 (2017): 1-41.
clarity,_quality,_novelty_and_reproducibility: clarity: the paper has clear writing. 
quality: the paper designed careful experiments to benchmark performance, but lacks more detailed ablation studies to uncover reasons why certain methods behave in certain ways. 
novelty: some of the empirical observations are novel. but overall, the paper benchmarking existing methods and most observations are expected. 
repro: results seem reproducible, to the best of my knowledge. 
summary_of_the_review: the paper designed careful experiments to benchmark performance, but lacks more detailed ablation studies to uncover reasons why certain methods behave in certain ways.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The reviewers felt the paper had some solid experimental exploration. The critiques centered largely around low novelty and that the main findings were just confirmations of known or intuitive facts. The most interesting findings were those which were highly counterintuitive, such as more data actually hurting. However, there was too little exploration/explanation of these phenomena. The authors are suggested to revise to focus more on exploring new and counterintuitive results, and perhaps omitting the straightforward results.
justification_for_why_not_higher_score: More exploration of counterintuitive phenomenon needed.
Decision: Reject