Title: Near-Zero-Cost Differentially Private Deep Learning with Teacher Ensembles
Authors: Lichao Sun
Yingbo Zhou
Jia Li
Richard Socher
Philip S. Yu
Caiming Xiong
Venue: ICLR 2020

Abstract: Ensuring the privacy of sensitive data used to train modern machine learning models is of paramount importance in many areas of practice. One approach to study these concerns is through the lens of differential privacy. In this framework, privacy guarantees are generally obtained by perturbing models in such a way that specifics of data used to train the model are made ambiguous. A particular instance of this approach is through a ``teacher-student'' model, wherein the teacher, who owns the sensitive data, provides the student with useful, but noisy, information, hopefully allowing the student model to perform well on a given task without access to particular features of the sensitive data. Because stronger privacy guarantees generally involve more significant noising on the part of the teacher, deploying existing frameworks fundamentally involves a trade-off between utility and privacy guarantee. One of the most important techniques used in previous work involves an ensemble of teacher models, which return information to a student based on a noisy voting procedure.  In this work, we propose a novel voting mechanism, which we call an Immutable Noisy ArgMax, that, under certain conditions, can bear very large random noising from the teacher without affecting the useful information transferred to the student. Our mechanisms improve over the state-of-the-art methods on all measures, and scale to larger tasks with both higher utility and stronger privacy ($\epsilon \approx 0$).


Review discussion:
The paper proposes an improvement on the PATE framework for achieving near-zero privacy cost, and showed privacy analyses and experimental evaluations of the proposed method.
The proposed method can be technically flawed. Adding a consent to the max will not guarantee privacy unless you account for the privacy cost for testing whether the distance of f(D) is larger than 2. This is because the distance of f(D) is data-dependent and revealing it violates privacy. Since the whole privacy analysis of PATE is based on the privacy guarantee of the Noisy ArgMax, the epsilon calculated here is voided.
Thank you for reviewing and comments. The following are the reply to some of your comments.
Our work closely follows PATE uses the sensitive data D and partition it into n sub-dataset.
Then individual teachers are trained on each data partition. After a student query sample x, then all teachers can return a voting result with the noise perturbation for privacy concerns. It is clear to see that due to the partition policy and uses the partition sub-dataset to train teachers, PATE is a data-dependent approach. Mathematically, PATE has a sensitive data D as input, and need a voting vector output V, the data transformer function f is f_{D, x}(D) -> V, where f is dependent on both sensitive data and student query, then it can transform the sensitive data D to voting vector V. In the whole process of PATE, for each student sample x, the data transformer function f_{D, x} is different, then PATE uses the composition theorem to estimate the total privacy loss of multiple mechanisms.
We follow each step as in PATE, and the difference here is that we use a different function f which adds a constant to the max. This can potentially make the sensitivity among different mechanisms different.
However, for each mechanism (i.e. given corresponding query x and dataset D), the sensitivity is fixed and will not be changed. Just like PATE need to know the partition information to bound and estimate the sensitivity of each mechanism, we also need to estimate the sensitivity of each of our mechanisms based on the dataset and the output of the data transformer f_{D, x}.
Finally, also use the composition theorem to estimate the total privacy loss of multiple mechanisms.
As such, we think our proposed approach is technically correct.
Please let us know if you have any further questions and concerns. Thanks a lot again.
This paper studies the teacher ensembles setting for differentially private learning. In this setting, each teacher holds part of the training set and trains a local model. The student uses unlabeled examples to query teacher model.
Then the student trains a model from scratch using the examples labeled by teachers.
In order to make the labeling process differentially private, previous work uses noisy argmax mechanism. Each class of label is assigned with a count number. The student first queries the same example to multiple teachers. To guarantee differential privacy, the counts are perturbed by noise before releasing.
Then, because of the post-processing property of differential privacy, the argmax operator on such noisy counts are still differentially private.
This paper proposes to add a constant c to the largest count before perturbing and releasing the counts. The authors argue this would improve the accuracy of the noisy argmax operator and yield the same privacy loss as previous approach.
However, adding a constant c would increase the sensitivity and therefore degenerates the privacy guarantee. The added noise cannot guarantee the privacy if all others are the same as previous work. To see this clearer, for example, if c=0, then one sample point can at most change the count by 1. If c>0, then one sample point can change the count by 1+c.
Because of this, the proposed method cannot guarantee the amount of differential privacy as the paper claimed.
Thank you for reviewing and comments. The following are the reply to some of your comments.
In fact, we mainly follow each step as in PATE. In PATE, for each student sample x, it will create an individual mechanism, and then multiple student data samples lead to multiple individual mechanisms. The difference here is that we propose to use each function f_{D, x} which adds a constant to the max, where D is the sensitive dataset. This can potentially make the sensitivity among different mechanisms different.
Then, based on different sensitivities of different mechanisms, we add the noise based on each sensitivity of each individual function.
Since both PATE and our work are data-dependent privacy analysis for each mechanism. In this case, what you mentioned "If c>0, then one sample point can change the count by 1+c" would not be for all mechanisms. Most mechanisms (with different sample points) can still hold the fact that at most change the count by 1 with adding a constant c, instead of c + 1.
Lastly, we estimate the total privacy budget by using the composition theorem.
As such, we think our proposed approach is technically correct. Please let us know if you have any further questions and concerns. Thanks a lot again.
To improve the privacy-utility tradeoff, this manuscript proposes a voting mechanism used in a teacher-student model, where there is an ensemble of teachers, from which the student can get gradient information for utility improvement. The main idea of the proposed approach is to add a constant C to the maximum count collected from the ensemble, and then noise is furthermore added to the new counts. I can understand that by adding the large constant C, the identity of the maximum count could be preserved with high probability, leading to a better utility on the student side.
However, equivalently, this could also be understood as that the noise is not added uniformly across all the counts, but instead a relatively smaller noise is added to the maximum count.
Hence it is not clear to me whether the final composition will still be differentially private?
Thank you for reviewing and comments. The following are the reply to some of your comments.
Yes, our composition theorem is also differential private (DP). In more detail, our work closely follows PATE uses the sensitive data D and partition it into n sub-dataset.
Then individual teachers are then trained on each data partition. After a student query sample x, all teachers can return a voting result with the noise perturbation for privacy concerns. Same as in PATE, for each query sample from the student, there is a particular mechanism, and we use the composition theorem to estimate the total privacy loss of multiple mechanisms. It still satisfies the DP requirements.
As such, we think our proposed approach is technically correct. Please let us know if you have any further questions and concerns. Thanks a lot again.

Meta-review:
role: PC
data: comment: This paper presents a differentially private mechanism, called Noisy ArgMax, for privately aggregating predictions from several teacher models. There is a consensus in the discussion that the technique of adding a large constant to the largest vote breaks differential privacy. Given this technical flaw, the paper cannot be accepted.

Decision: Reject