Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper proposes to use recent advances for dataset condensation as an alternative for differentially private data generation. Specifically, they use DP-SGD to optimize a gradient matching objective that minimizes the distance between the gradients of real data on the target task and the gradients of synthetic data on the target task. The authors show this approach generates better data, in terms of downstream classification accuracy, than the classic approach which trains a discriminator and a generator. The authors then discuss potential limitations such as scalability to harder datasets and generality to other tasks.
strengths_and_weaknesses: **Strengths:**

1.This paper is very well written. Contributions and potential limitations are discussed adequately.

2.This paper improves the SOTA data utility by a large margin of differentially private data generation. The improvement is in terms of the downstream classification accuracy.

**Weakness:**

1.The technique contribution is limited. This paper mainly uses two existing techniques, DP-SGD and gradient matching.

2.This work only uses one algorithm from data condensation, i.e., gradient matching. It would be better if the authors can try more algorithms so the community can have a better understanding about data condensation for differentially private data generation. For example, distribution matching [1] that minimizes the distance between the averaged feature of real data and the averaged feature of synthetic data, which is also easy to implement with DP.

3.In Section 6 you show the generator from a previous work can improve the visual quality of your algorithm. How does the visual quality of your algorithm compare with the visual quality of data directly generated from that generator? 

4.(Minor) Line 121, minimized -> minimize.

[1]: DATASET CONDENSATION WITH DISTRIBUTION MATCHING, https://arxiv.org/pdf/2110.04181v1.pdf.

questions: Please see the Weakness.
limitations: None
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper introduces a new approach to creating a synthetic data set as a stand-in for releasing some sensitive data. Instead of the standard technique of learning some kind of general purpose generative model that tries to capture the full data distribution, the proposed methods iteratively updates a set of data points (initialised randomly) directly to maximise the utility of the synthetic data for a given downstream classification task.

More precisely, the synthetic data is used to train the downstream classification network while in parallel the original data is used to train a network of the same architecture. In each step, the synthetic data is then updated to minimise the difference between the network parameter gradients between the two networks undergoing training. Privacy is ensured by perturbing the gradients of the network trained on the original data using the Gaussian mechanism before it is used in optimising the synthetic data.

The paper performs an extensive evaluation of the method and compares it to competing methods from recent literature on image classification tasks for MNIST and FashionMNIST using a convolutional network. The classifier trained on synthetic data generated using the proposed method shows superior classification accuracy compared to when trained on synthetic data generated using competing methods. Additionally, the proposed method allows to sample a much smaller data set than the original without a significant loss in downstream accuracy, which makes downstream training much easier.

The paper additionally explores using different architectures for the downstream task, which is found to have no significant effects on accuracy as long as the architecture is sufficiently similar (i.e., some kind of convolutional network).

Finally, the paper discusses that the generated synthetic data may fall out of the original data manifold due to unconstrained optimisation, and therefore does not "look like" the original data. The discussion includes a modification of the proposed method that ensures that the synthetic data is on the manifold and compares it to the unconstrained method.
strengths_and_weaknesses: Originality: This paper presents an interesting novel perspective on generating synthetic data that deviates from established practise for generating synthetic data under DP guarantees. However, since I am not closely familiar with the privacy-agnostic(/non-private) data distillation technique, it could be that these contain similar methods except for the privacy perturbation, which could make this work a somewhat iterative. I hope some of my fellow reviewers may be more familiar with that and can form a more well-founded opinion on this. However, while the method would be iterative in that case, I believe that the evaluation for the privacy setting would still be a sufficiently original contribution.

Significance: The proposed methods demonstrates clearly superior results for the chosen downstream tasks, which could have a large impact for sensitive data release in practical applications when the downstream task the data will be used in is clear (or can be deduced with high likelihood). While I do not know how likely that setting is, I do think that it is not one impossible to find oneself in, so designing a well-working method for such a case is a valuable contribution.

Soundness/Quality: The proposed methods applies standard privacy techniques in a straightforward way, so results on privacy from earlier work transfer more or less directly. The experiments appear to have no technical flaws, however the reported results do not convey any information on error margins.

Clarity: The paper is overall clear and easy to understand.
questions: - I cannot comprehend what the sentence in lines 141-143 is trying to say, can you elaborate?

Minor notes/nitpicks:
- Def 3.1 : "There are two ways in which $\mathcal{D}$ and $\mathcal{D}'$ may differ in one training example: it could be entirely removed or replaced by a different one. This makes a (small) difference, so please specify your setting (or state explicitly that it does not matter in your case).
- Thm 3.1 : "Any function F"; technically the function $F$ is constrained to those that do not access the original data again.
limitations: The proposed method has a number of limitations that could hamper its adoption in practice. The paper discusses and performs experiments exploring the major ones of these but not always to a full degree:

- The method requires a downstream task to be known a-priori before generating the data. This may not always be the case, however it is probably likely for the data holder to find the "most likely" task their data will be used for (they have collected that data for a reason, after all). The authors give an example in Sec 5.2 of a use case for their method in practice.

- Generalisability: The synthetic data produced using this method has no guarantee to be useful for anything but the downstream task it was optimised for. The authors recognise this and perform additional experiments in Sec 5.1 where they test different downstream classification architectures. They find that utility is similar for different architectures of convolutional networks (with the synthetic data optimised using a CNN as well) but drops off for MLP networks; no other architectures are explored. The authors do not evaluate different downstream tasks that could be performed on the synthetic data, so it is remains somewhat unclear how the synthetic data generalises. However, the authors point out in the end of Sec. 6. that while other methods may have in theory better capacity to generalise, these fail to show that that materialises as a benefit in any particular task in practise.

- Out-of-distribution synthetic data: The synthetic data is optimised in an unconstrained domain and may fall out of the manifold that the original data lies on. The authors discuss and evaluate this and provide the interesting finding that there appears to be a trade-off between generating data that lies on the correct manifold versus data that maximises utility. There is no deeper inspection of why that is the case: It remains especially unclear whether this is merely an artifact of compressing the original data into a smaller synthetic set or an unavoidable trade-off. This also ties in with generalisability discussed above: A task other than the chosen downstream task may require data to be more similar to the original data (such as, e.g., image segmentation in this case).

Summarising: The method does have limitations and the paper discusses the major ones. While some questions remain and follow up work on these would be welcome, they may be deemed to lie outside the permissible length of a conference paper and the discussion present should be sufficient for the reader to be aware of the trade-offs and whether the method is suitable in their setting.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper proposes a new utility-optimized approach of privatizing high-dimensional data generation which optimizes the synthetic samples instead of the deep generative models. Experiments on MNIST and FashionMNIST are conducted to demonstrate the effectiveness.
strengths_and_weaknesses: Originality. The problem that the paper aims to address is well motivated. The proposed approach seems new to me. Relevant literatures are well discussed.

Quality. The proposed algorithm overall makes sense to me and I didn't find any technical mistakes. However, some details of the algorithm and the experiments are missing (see my questions below).

Clarity. The paper is easy to follow and well organized.

Significance. The results overall look good, yet I am not sure whether they are enough. The authors only chose $\epsilon = 1$ and 10, corresponding to high and low privacy budget. Not sure whether what happens when the budget is moderate, say $\epsilon = 5$. Also, it would be better if the performance is benchmarked on colored image datasets (e.g., CIFAR-10).
questions: I only have two minor questions.

1) In algorithm 1, what equation is used to determine the value of $\sigma$ in order to preserve $(\epsilon, \delta)$-DP? I think this may need to be explicitly discussed.

2) What are the numerical values in terms of the computational cost? I checked the supplementary but found this is missing. In particular, I am interested in the algorithm running time for different values of spc, say 10, 20, 6000 for MNIST.   
limitations: The authors adequately discussed the low visual quality issue of the proposed approach. Other aspects such as scalability and generality are also mentioned.
review_title: null

Meta-review: metareview: The paper received overall positive reviews. The rebuttal clarified all the concerns the reviewers had. At this point, I will recommend the authors to incorporate the comments from the discussion into the paper, especially the new results with CIFAR-10.
confidence: Certain
Decision: Accept