Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a data condensation algorithm that guarantees differential privacy, and demonstrated empirically that the proposed algorithm works better than existing data condensation algorithms.
strength_and_weaknesses: Strength: The idea seems interesting and the empirical results seem good.

Weakness: I'm quite confused about how the algorithm works.
clarity,_quality,_novelty_and_reproducibility: I'm confused about Algorithm 2. I think the presentation might need to be improved to clarify things. And since I'm not able to follow the algorithm, I'm not quite sure about the correctness of the privacy analysis. 
- S seems to be a set but then we have S = S - eta*gradient of l. How does that work? Is S like a concatenation of vectors? 
- What is the gradient with respect to? Is theta some learnable parameter?
- I think it would be better if you write down the gradient explicitly.
- What does it mean by "initialize S ... with random noise"?

Some other comments:
- In Table 1, why don't we compare different methods under the same privacy budget?
- I'm confused about the formulation in (1). Wouldn't it be optimal if I choose S = T? I feel like there need to be some term related to how condense / private S is, right?

--- update ---

The authors' response and revision has cleared my questions. 
summary_of_the_review: The general idea seems interesting, but I wasn't able to understand Algorithm 2 to judge the correctness of privacy.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: Paper proposes differentially private dataset condensation, and improvement over the ICML'22 work [1], and shows that the proposed method provides good utility.

[1] Dong, T., Zhao, B., & Lyu, L. (2022). Privacy for Free: How does Dataset Condensation Help Privacy?. arXiv preprint arXiv:2206.00240. 
strength_and_weaknesses: Paper proposes a differentially private version of dataset condensation and proposes two DP methods, linear and non-linear differentially private dataset condensation. Proposed methods claim to avoid the pitfall of [1], and provide formal privacy guarantees. 

Empirical evaluation shows that the proposed DP methods perform better than DP-sinkhorn and DP-MERF, and provide tighter privacy guarantees. 

Overall, I like the paper. I have not checked the proofs in detail, assuming they go through, the proposed methods are potentially useful. I think a brief discussion on the utility and use of DP data condensation methods vs general DP data generation would be useful.
clarity,_quality,_novelty_and_reproducibility: Paper was easy to read
summary_of_the_review: I like the proposed method, and assuming the proofs are solid, I have no problem accepting it to the conference.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper first discusses the issues in the controversial assumptions in a recent literature. Then it proposes two differentially private dataset condensation algorithms LDPDC and NDPDC. In the experiment, it makes the evaluations on multiple datasets and the results show that the proposed two methods achieves better privacy-utility trade-off than baselines.
strength_and_weaknesses: **Strengths**
1. The experiment results of two proposed methods show the promising improvement from the baselines.

**Weakness**
1. My major concern is that this paper misses an important and relevant literature [1], which might influence the novelty and significance of the results. The algorithm in [1] is similar to Algorithm 1 in this paper. The slight difference is that in this paper, the data batch is sampled by Poisson sampling, while [1] samples the data without replacement. Moreover, it is questionable how much improvement LDPDC and NDPDC have by comparing them with this literature.
2. The main experiment results could be more convincible if different algorithms can be compared with the (mostly) fixed DP-Budget. It is also worth showing at which $epsilon$ the performance of NDPDC would break down. They will help understand the comparison with the baselines. They are also convenient for later work to build the comparison with this work.
3. The writing of this paper can be improved. Below some clarity questions or suggestions are listed:
 - The abbreviation "DM" is not introduced.
 - The definition of $T_c$ is introduced in page 5, but appears first in page 4.
 - The number of iteration $I$ is not mentioned in the Algorithm 2.
 - Missing the description of whether the feature extractor $\phi$ is pretrained.
 - Missing the description of data augmentation in Algorithm 2.
 - What is the privacy budget setting for Table 2
 - Which column is the "ConvNet" mentioned in the second paragraph of section 4.2

[1] Lee, Kangwook, et al. "Synthesizing differentially private datasets using random mixing." 2019 IEEE International Symposium on Information Theory (ISIT). IEEE, 2019.


--------After Discussion-------
Given that the difference between [1] and LDPDC is clarified, I raised my score to 6. I recommend authors to add the discussion and solve the clarification questions in the revision.
clarity,_quality,_novelty_and_reproducibility: See the *Strength And Weaknesses*.
summary_of_the_review: I appreciate this work studies the dataset condensation with differential privacy. However, because of the missing discussion and comparison with [1], the novelty and result significance are doubt. Therefore, I recommend reject.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper resulted in some additional discussion between reviewers. One point brought up was that NDPDC is rather similar, conceptually, to DP-MERF. This implies the following:
- Both techniques in the paper seem very similar to prior works, with some slight modification.
- NDPDC is conceptually similar to DP-MERF, but seems to perform empirically better. This comparison may need to be done better, to identify what difference between the two methods really results in better utility. 

Another issue raised was that this paper seems to dodge the fundamental critiques against the paper of Dong et al. For example, the example of Assumption 3.1 was not only the details of the fact that this would be implausible in, e.g., non-stochastic and full batch gradient descent settings, but even the "correct" Assumption 3.2 seems too strong to assume (the authors of the submission mention that it may be true in some asymptotic sense for some particular setting, but is unclear whether it's something that could hold more broadly). Furthermore, the understanding is that the proof of Proposition 4.10 is incorrect even for the linear case, whereas this paper shows that it is not true for the non-linear case. In these senses, the paper seems to be "weaselly" about the issues with the paper of Dong et al., getting lost among the trees inside the broader forest. 

For these reasons, it was felt that the paper may benefit from further revision, to better clarify the contribution and differences with prior work, as well as being more direct with the shortcomings of the prior work that the authors critique.

Orthogonal to the decision, the authors may also wish to note and reference the recent (after the ICLR submission deadline) paper by Carlini, Feldman, and Nasr (https://arxiv.org/abs/2209.14987), which seems to fill the same narrow niche of critiquing this particular paper.
justification_for_why_not_higher_score: Some lack of novelty over prior work and unclear relationship with it, not obvious what aspect of the new method results in better performance, critique of previous work is somewhat indirect.
Decision: Reject