Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: This paper reveals that the noise added to the PATE voting mechanism (to attain a differential privacy guarantee) enables new forms of leakage of sensitive information. A simple adversary is enough to exploit this noise to extract high-fidelity histograms of the votes. As pointed out in this paper, a low consensus vote indeed indicates minority-group membership. The main technical contribution is the model to extract PATE histograms.

strengths_and_weaknesses: Strengths: This paper is well written. The proposed attack model is simple but interesting. I believe the authors' discovery is significant.
questions:  Weaknesses: The technical part seems too simple. I'm not sure such a simple attack model contains enough technical contribution to appear in NeurIPS. This paper might be more appropriate to be published (as a report paper) at a security conference. 
limitations:  Weaknesses: The technical part seems too simple. I'm not sure such a simple attack model contains enough technical contribution to appear in NeurIPS. This paper might be more appropriate to be published (as a report paper) at a security conference.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper first shows that the prediction histogram from several teachers will leak the membership of minority group for this test data. Then in the paper, an attack is proposed to show the possible leakage of histogram under a modern DP approach, PATE. Finally, it concludes DP doesn't protect the leakage of minority group membership.
strengths_and_weaknesses: ### Strengths
1. This paper takes effort to show what differential privacy can/cannot protect. This big topic is meaningful to the differential privacy community and the one who cares about privacy in applications.
2. The paper is written well.

### Weaknesses
1. The main conclusion of this paper is that the differential privacy doesn't protect the membership of minority group for *test data*. This conclusion seems not very surprising to me as differential privacy is to protect the privacy of *training data*.
2. The experiment result also shows that the leakage of true histogram is more severe when the privacy cost is smaller. This result is counter-intuitively and the explanation is not very convincing to me. The estimation for $q_i$ from a sequence of i.i.d. samples (0/1) from Bernoulli random variable likely wouldn't be more accurate when $q_i$ is larger, but depends on its variance $q_i(1-q_i)$. I would suggest authors to further check the difference between $Q(H)$ and $\bar{q}$ and see if this difference is smaller when the injected noise is larger.
3. The experiment doesn't include the end-to-end evaluation between privacy cost to minority group membership leakage.

--------After Rebuttal--------

Thank the authors for their reply, which solves W2 and W3 and alleviates my concern in W1.
questions: 1. I am not clear why the introduced minority group membership leakage problem is a privacy problem for training data. The motivation is a bit confused.
2. It is better to check the difference between $Q(H)$ and $\bar{q}$ to support the explanation for better attack performance with higher noise levels.
limitations: The authors list some limitations of their proposed attack algorithm. However, I am concerned whether the conclusion is meaningful to the community. PATE is a DP algorithm designed for training data and it likely will not protect the test data's information, e.g. the minority group membership discussed in this paper.
review_title: null

final_score: rating: 3: Reject: For instance, a paper with technical flaws, weak evaluation, inadequate reproducibility and incompletely addressed ethical considerations.
review_content: summary: This paper studies the confidentiality of PATE from an adversarial perspective. This paper shows that the attackers can recover the histograms of votes by querying the voting results multi-times. This paper shows an interesting result that adding larger Gaussian noise actually increases the risk to the confidentiality of the vote histogram.



strengths_and_weaknesses: Pros: This paper proposed an attack method that can extract the voted histograms by querying the voting results multi-times. This paper uses an example to show that sensitive attributes can be leaked from the voted histograms. 

Cons: The experimental settings are not clear. More experiments are needed to support the corresponding claims. 



questions: 1. Does the model update frequently? Why cannot the model send the same result to the same query? 

2. Why do you use the Euclidean distance to measure the distance between $Q\hat{H}$ and $\bar{q}$? Why do not use KL or Wasserstein metric to measure the distance?

3. In Line 247, what's the $\epsilon$ and $delta$ for the DP guarantee? In Line 250, what's the corresponding budget for $\sigma = 40$ etc?

4. In line 248, what's the DP budget for each query? And what's the total budget for $10^4$ queries?

5. In line 282, $(8, 10^{-5})$ is the DP budget for one query? 

6 In Figure 4, what's the corresponding DP budget for $\sigma=40$?

7. The results shown in Figures 6 and 7 are counter-intuitive. What's the corresponding DP budget for each query? To verify the correctness, it's better to add a large enough noise (i.e. the standard deviation of the Gaussian noise is close to inf.). Another baseline is that we add no noise to the results. Can we reach the same conclusion?

8. Is it possible to perform some experiments that the attacker can steal the private attributes from the recovered histogram with some DP guarantee?

limitations: See the Weaknesses and Questions part.
review_title: null

final_score: rating: 10: Award quality: Technically flawless paper with groundbreaking impact, with exceptionally strong evaluation, reproducibility, and resources, and no unaddressed ethical considerations.
review_content: summary: This work provides an in-depth investigation of a particularly relevant failure mode of Differential Privacy (DP) when training ML models using the PATE framework.  In a nutshell, PATE takes the prediction of multiple independently trained models and outputs a prediction that is DP by first adding noise to the histogram and then taking the prediction with most votes.   
This paper describes a privacy attack based in querying the database multiple times with the same question.  By building up a distribution of these (noisy) responses, they show it is possible to back out the underlying histogram that the noise was meant to protect.  
strengths_and_weaknesses: Strengths:
- The paper is quite enjoyable to read -- it is well-written, easy to follow, logical, and concise. 
- It will likely spur much-needed discussions about the practical usefulness of DP in real applications. 

Weaknesses: I don't see any significant weakness. 

Minor suggestions: 
- Instead of using $M$ for number of samples and $m$ for number of classes, use different letters.
- (Line 263-176) I would prefer if in the appendix there was an explanation of precisely how the privacy cost is computed so that the reader doesn't need to refer to Papernot et al. as much.

Tiny typo: (line 34) *a* attacker


questions: Honestly, I read everything carefully (including the appendix) and I found it quite clear, thank you! 
limitations: The authors address the limitations of their work.  

Not only I don't have any qualms with this work, but I believe that it is an important contribution that could have a major positive societal impact (if the work is taken seriously by the privacy community). 
I think this paper should definitely be given a spotlight talk.
review_title: null

Meta-review: metareview: Although the fact that DP does not protect against population statistics is a widely known fact, the paper weaves this together with PATE (which relies on DP statistics) to demonstrate the danger of mis interpreting the protection guarantees provided by DP. This is a point worth discussing among the privacy and security community.
confidence: Less certain
Decision: Accept