Title: Semi-supervised Knowledge Transfer for Deep Learning from Private Training Data
Authors: Nicolas Papernot
Martín Abadi
Úlfar Erlingsson
Ian Goodfellow
Kunal Talwar
Venue: ICLR 2017

Abstract: Some machine learning applications involve training data that is sensitive, such
as the medical histories of patients in a clinical trial. A model may
inadvertently and implicitly store some of its training data; careful analysis
of the model may therefore reveal sensitive information.

To address this problem, we demonstrate a generally applicable approach to
providing strong privacy guarantees for training data: Private Aggregation of Teacher Ensembles (PATE). The approach combines, in
a black-box fashion, multiple models trained with disjoint datasets, such as
records from different subsets of users. Because they rely directly on sensitive
data, these models are not published, but instead used as ''teachers'' for a ''student'' model. 
The student learns to predict an output chosen by noisy voting
among all of the teachers, and cannot directly access an individual teacher or
the underlying data or parameters. The student's privacy properties can be
understood both intuitively (since no single teacher and thus no single dataset
dictates the student's training) and formally, in terms of differential privacy.
 These properties hold even if an adversary can not only query the student but
also inspect its internal workings.

Compared with previous work, the approach imposes only weak assumptions on how
teachers are trained: it applies to any model, including non-convex models like
DNNs. We achieve state-of-the-art privacy/utility trade-offs on MNIST and SVHN
thanks to an improved privacy analysis and semi-supervised learning.


TL;DR: Semi-supervised learning of a privacy-preserving student model with GANs by knowledge transfer from an ensemble of teachers trained on partitions of private data.

Review discussion:
Thanks for the nice paper with inspiring intuitions. The theory analysis is cool, however, the probabilistic bound has quite a number of  empirical parameters, which makes it difficult to understand whether the absolute 100% security is  guaranteed.
Thus we have to check the experiments to understand how secure the method is, unfortunately it is not straightforward to evaluate either.
More specifically, I have two questions related to the experiments.
1) It is not clear from the experiments that how much better the proposed method is compared with the previous works, e.g., [Fredikson et al, 2015] and [Hamm et al, 2016].
2)  I am quite interested in the comparison between GANs vs distillation for semisupervised learning. From the paper my impression is that GAN-based approach is more secure while distillation-based approach may have higher performance. If it is correct, please give more details and analyze why.
[Note: This response was crafted with the original version of the pre-review question in mind. We later noticed that the question had been modified and extended our reply in the comment below to take into account the modifications made to the pre-review question.]
1) The Fredrikson et al. 2015 paper on model inversion (the one that we reference) addresses a very different question than ours, so we cannot provide an apples-to-apples comparison.
Indeed, it is difficult to see how differential privacy can apply to the problem domain of their work (identifying individuals' faces). This difficulty is highlighted by the problem's incompatibility with our learning approach. Teachers trained on disjoint sets of faces could never agree, or generalize, in a manner that would allow the student to learn.
Please let us know if we misunderstood your question on Fredrikson et al.’s paper, though.
In particular, perhaps your question was referring to a different 2015 paper by Wu, Fredrikson, et al. https://arxiv.org/abs/1512.06388 ? This work applies output perturbation and considers strongly convex models, for which the sensitivity is small and one can therefore hope to get good differential privacy guarantees. In a non-convex setting such as ours, the sensitivity of the optimization function is very large, and adding noise proportional to it would yield a model whose accuracy is no better than a random predictor.
Compared to Hamm et al.’s work, our most significant contribution is that we broaden the scope within which the method may be applied, in part because our privacy analysis, based on the moments accountant, makes fewer assumptions and can provide privacy guarantees for non-convex models such as deep neural networks. We do not have a direct comparison for the performance of our method and that of Hamm et al. because they focus on logistic regression while we consider more sophisticated models, with different datasets. We believe that the moments account would also lead to improved results on logistic regression, however.
2) In our setting, with distillation, the student would learn only from data that is labeled by the teachers (making no use of purely unlabeled data).
In contrast, with semi-supervised learning techniques, the student can learn both from labeled data and also from completely unlabeled data—which may be more abundant, and more easily accessible.
One way to do semi-supervised learning is using GANs, as in our work. As indicated in Appendix C.1, we have observed that distillation does not perform sufficiently well when training with very few queries—as is required to ensure tight privacy bounds, particularly when each answer includes the entire probability vector output by the ensemble of teachers.
Thus, we believe that, at equal levels of privacy, semi-supervised learning with GANs will provide higher utility than distillation. We hope that the further details in the appendix provide a satisfactory explanation; please let us know if you have further questions, or if you would recommend that we move some of those details into the body of the paper.
The newer comments from the reviewer relate to the parameters of our work, and their evaluation.
Our analysis follows the established framework of differential privacy. It defines a probabilistic bound, with parameters (epsilon and delta) on changes in the output of our algorithm when it is ran on differing inputs. Smaller values of the parameters yield stronger guarantees. As is often the case with security, achieving 100% privacy (represented by epsilon = delta = 0) is unfortunately impossible while releasing a useful model.
However, we follow standard practices and guidelines, setting epsilon in the single-digit range (as is often done, for instance, by Google's Rappor system and in Apple's use of differential privacy) and setting delta so that its order of magnitude is inverse to the number of training samples. The other "empirical" parameters of our method (e.g., number of teachers and level of noise) do not have to be considered separately: they can be accurately judged by their impact on privacy (as represented by epsilon and delta), and on utility (the accuracy on test data). We believe that our results provide a validation for these "empirical" parameters, in the sense that we can take epsilon and delta as explained above while maintaining competitive accuracy.
In other words, to evaluate how “secure the method is,” using the standard metrics of the security and privacy community, it suffices to look at the privacy parameters epsilon and delta; it is not necessary to consider details of the algorithm, the internal parameters, or the experimental setup.
We would welcome input on whether adding an explanation along these lines to the paper would be helpful.
Also, if we are speaking to a different notion of security than what the reviewer had in mind, we would be happy to continue this discussion.
You mentioned future work on tasks in which "the discrete output categories are not as distinctly defined by the salient input space features."
Is this primarily a matter of empirical experimentation, or are there also some technical/theoretical challenges that would need to be addressed? If so, please elaborate.
Thank you for the insightful question. The short answer is that it's
both empirical and theoretical work. Here's the long answer:
Suppose we have input images x and class labels y. Purely supervised
learning consists of learning p(y|x). A common strategy for
semi-supervised learning is to learn a generative model p(x,y). The
conditional distribution p(y|x) in this generative model often solves
the semi-supervised learning task with less overfitting than would
occur with direct supervised learning. This common strategy does not
yet have a clear theoretical justification. The limitations of
theoretical understanding of this approach are described in the Deep
Learning textbook, section 15.1.1, "When and why does unsupervised
pretraining work?":
http://www.deeplearningbook.org/contents/representation.html
Our work has exposed something unusual: our semi-supervised student
network learned a p(y|x) that obtained lower test error than the
teacher ensemble's p(y|x).
Our usual understanding of generative semi-supervised learning would
predict that the teacher network would obtain some test error rate
E_teacher. A model trained using purely supervised learning to mimic
the teacher would overfit somewhat, and obtain a test error rate
E_supervised, with E_supervised > E_teacher.
Finally, generative
semi-supervised learning would obtain less overfitting, and would
obtain some test error rate E_generative, with E_teacher <
E_generative < E_supervised.
Instead, to our surprise, we found that E_generative < E_teacher in many cases.
This means that the unsupervised component of the learning algorithm is
not merely reducing overfitting. It actually causes the student
network to become *better than the labels it was trained on*. This
result goes beyond the predictions made by our usual understanding of
generative semi-supervised learning.
We suspect that this means that semi-supervised learning via
generative modeling is not fully generic.
Instead, generative modeling
biases the classifier toward recognizing categories that are highly
salient in input space. For tasks like MNIST and SVHN, the inputs are
images of digits, which are written symbols. Humans designed written
symbols to be very visually salient and distinct from each other, so
that the most obvious feature of the input is the identity of the
symbol. This suggests that optical character recognition is a task
that is especially likely to benefit from regularization by generative
modeling. A model that uses the same input to predict a different
output (e.g., author identity, rather than digit identity) may not
benefit nearly as much from generative training.
Likewise, generative
training might be less effective on a different input domain, where
the input features have not been intentionally designed to make the
most important aspect of the data be highly salient.
For example,
generative training might not be as useful for analyzing gene
expression data.
A full resolution of this anomaly would require:
- Empirical work testing the effectiveness of generative
semi-supervised learning in different settings (e.g., predicting
author identities of MNIST inputs, classifying gene expression data)
- Theoretical work to explain more concretely how generative
semi-supervised learning is able to reduce the test error
While such additional empirical and theoretical work would be
interesting, such an in-depth investigation of the underpinnings of
generative semi-supervised learning would be beyond the scope of our
current paper.
Nice, well-written, and interesting paper. Very general question: what are the most important challenges in the context of privacy preserving specifically in deep learning that you perceive?
In our view, the most important high-level challenge is to develop ML models that learn while protecting the privacy of training data, without sacrificing accuracy, performance, or software-engineering simplicity, and with a satisfactory definition of "privacy". This is particularly important because deep learning often relies on vast quantities of data for training, and this data is often sensitive. It is particularly difficult for deep learning because we cannot make some convenient simplifying assumptions (e.g., convexity). This is part of why in previous works, the accuracy losses to guarantee privacy were significant.
In addition, deep learning is a "moving target"---the field is so active that we cannot make very strong bets on which approaches will be most successful in the long run and deserve the biggest attention. At a lower level, these difficulties result in technical problems, such as the need for accurate tracking of privacy losses (which our moments account addresses) and of algorithmic and architectural ideas (of which our paper is an example), sometimes specialized to particular learning tasks or models.
Altogether a very good paper, a nice read, and interesting. The work advances the state of the art on differentially-private deep learning, is quite well-written, and relatively thorough.
One caveat is that although the approach is intended to be general, no theoretical guarantees are provided about the learning performance. Privacy-preserving machine learning papers often analyze both the privacy (in the worst case, DP setting) and the learning performance (often under different assumptions). Since the learning performance might depend on the choice of architecture; future experimentation is encouraged, even using the same data sets, with different architectures. If this will not be added, then please justify the choice of architecture used, and/or clarify what can be generalized about the observed learning performance.
Another caveat is that the reported epsilons are not those that can be privately released; the authors note that their technique for doing so would change the resulting epsilon.
However this would need to be resolved in order to have a meaningful comparison to the epsilon-delta values reported in related work.
Finally, as has been acknowledged in the paper, the present approach may not work on other natural data types. Experiments on other data sets is strongly encouraged.
Also, please cite the data sets used.
Other comments:
Discussion of certain parts of the related work are thorough.
However, please add some survey/discussion of the related work on differentially-private semi-supervised learning.
For example, in the context of random forests, the following paper also proposed differentially-private semi-supervised learning via a teacher-learner approach (although not denoted as “teacher-learner”). The only time the private labeled data is used is when learning the “primary ensemble.”  A "secondary ensemble" is then learned only from the unlabeled (non-private) data, with pseudo-labels generated by the primary ensemble.
G. Jagannathan, C. Monteleoni, and K. Pillaipakkamnatt: A Semi-Supervised Learning Approach to Differential Privacy. Proc. 2013 IEEE International Conference on Data Mining Workshops, IEEE Workshop on Privacy Aspects of Data Mining (PADM), 2013.
Section C. does a nice comparison of approaches. Please make sure the quantitative results here constitute an apples-to-apples comparison with the GAN results.
The paper is extremely well-written, for the most part. Some places needing clarification include:
- Last paragraph of 3.1. “all teachers….get the same training data….” This should be rephrased to make it clear that it is not the same w.r.t. all the teachers, but w.r.t. the same teacher on the neighboring database.
- 4.1: The authors state: “The number n of teachers is limited by a trade-off between the classification task’s complexity and the available data.” However, since this tradeoff is not formalized, the statement is imprecise.
In particular, if the analysis is done in the i.i.d. setting, the tradeoff would also likely depend on the relation of the target hypothesis to the data distribution.
- Discussion of figure 3 was rather unclear in the text and caption and should be revised for clarity. In the text section, at first the explanation seems to imply that a larger gap is better (as is also indicated in the caption).
However later it is stated that the gap stays under 20%. These sentences seem contradictory, which is likely not what was intended.
Thank you for your time and for your review.
Current deep learning models do not come with any accuracy guarantees, unfortunately.
In particular, for tasks such as MNIST and SVHN, none of the state-of-the-art methods, even those not concerned about privacy, provide accuracy guarantees. With our current level of theoretical understanding, performance in terms of accuracy has to be established empirically, as is current practice in ML research, and as we have done here.
In order to further demonstrate the general applicability of our approach, we performed experiments on two additional datasets. The UCI Adult dataset is made up of census data, and the task is to predict when individuals make over $50k per year. The UCI Diabetes dataset includes de-identified records of diabetic patients and corresponding hospital outcomes, which we use to predict whether diabetic patients where readmitted less than 30 days after their hospital release. We looked at the UCI datasets because some of them appear in previous papers on privacy;
among the many UCI datasets, in the limited time available, we quickly decided to study two that looked interesting. These UCI datasets seemed at least reasonable, and not random, as they had been used for similar ML applications, so they were the first (and only) datasets that we tried for our new experiments. They turned out to be well-suited to our techniques, even though we had no idea whether this was the case before running our experiments.
While our experiments on MNIST and SVHN used convolutional neural networks and generative adversarial networks, we instead used random forests to train our teacher and student models for both of the UCI datasets. Our new results on these datasets show that despite the differing data types and architectures, we are able to provide meaningful privacy guarantees---with epsilon bounds below 2.7---at modest costs in accuracy---with reductions of at most 2% between a non-private model and our privacy-preserving student. We added them to the revised draft of our paper posted today.
In a real deployment, noise needs to be added to the differential privacy guarantees themselves before releasing them. In order to allow for a meaningful comparison with related work, we provided these additional values in our updated draft. We note that doing so does not impact our results’ competitiveness with prior work.
We added a discussion of prior work by Jagannathan et al. in the revision of our paper. A key difference is that their approach is tailored to decision trees. Our approach is applicable to any machine learning algorithm, including more complex ones. In the specific case of decision trees, our approach can also work well, as demonstrated by our new, added results for the Adult and Diabetes datasets performed with random forests.
Another key difference is that their approach modifies the classic model of a decision tree to include the Laplace mechanism.
Thus, the privacy guarantee does not come from the disjoint sets of training data analyzed by different decision trees in the random forest, but rather from the modified architecture.
In contrast, partitioning is essential for our approach's privacy guarantees. We know of no other paper that is similarly comparable to our work, and could not find any via a reverse citation search based on Jagannathan et al.
Regarding the comparison of Appendix C with GANs, we made it clearer that our goal was to reduce the number of labels required from the teacher ensemble. This makes the comparison more straightforward.
We thank you for all remaining comments not discussed in this response: we agree with all of them and modified the draft accordingly.
This paper discusses how to guarantee privacy for training data. In the proposed approach multiple models trained with disjoint datasets are used as ``teachers'' model, which will train a ``student'' model to predict an output chosen by noisy voting among all of the teachers.
The theoretical results are nice but also intuitive. Since teachers' result are provided via noisy voting, the student model may not duplicate the teacher's behavior.
However, the probabilistic bound has quite a number of  empirical parameters, which makes me difficult to decide whether the security is 100% guaranteed or not.
The experiments on MNIST and SVHN are good.
However, as the paper claims, the proposed approach may be mostly useful for sensitive data like medical histories, it will be nice to conduct one or two experiments on such applications.
Thank you for your time and for your review.
As in other work on differential privacy, we provide probabilistic guarantees, and these are parameterized to make trade-offs explicit. Our method provides a lower probability of privacy failure than other methods at comparable utility. (You may find a detailed explanation of the different parameters of our analysis in our answer dated from November 22nd on OpenReview.)
An important advantage of MNIST and SVHN, at this stage of development of the field, is that they enable rapid experimentation and benchmarking---no dataset that contains sensitive data or that imitates sensitive data seems to be a standard "fruit fly" to the same degree.
However, to further demonstrate the applicability of our approach to other natural data types, like the ones encountered in medical applications, we have followed your excellent suggestion of looking at specific applications using medical data. For this purpose we have performed additional experiments on the UCI Diabetes dataset, and have added the results to the new, revised version of our submission. The results show that our approach can apply well to medical data, and provide strong, meaningful privacy guarantees without reducing the accuracy of key medical-domain tasks, such as diagnostic classification and relapse prediction [https://arxiv.org/pdf/1602.04257.pdf]. (You may find additional details in our response to another review.)
This paper addresses the problem of achieving differential privacy in a very general scenario where a set of teachers is trained on disjoint subsets of sensitive data and the student performs prediction based on public data labeled by teachers through noisy voting. I found the approach altogether plausible and very clearly explained by the authors. Adding more discussion of the bound (and its tightness) from Theorem 1 itself would be appreciated. A simple idea of adding perturbation error to the counts, known from differentially-private literature, is nicely re-used by the authors and elegantly applied in a much broader (non-convex setting) and practical context than in a number of differentially-private and other related papers. The generality of the approach, clear improvement over predecessors, and clarity of the writing makes the method worth publishing.
Thank you for your encouraging review. In the revised version of our paper posted today, we added more context to the bound on the moment generating function provided in Theorem 1. Our analysis is rather conservative in that it pessimistically assumes that, even if just one example in the training set for one teacher changes, the classifier produced by that teacher may change arbitrarily.
Therefore, our analysis does not require any assumptions about the workings of the teachers; we regard this point as an advantage of our approach.
Nevertheless, we expect that stronger privacy guarantees may perhaps be established. We will add a discussion of this subject to the paper.

Meta-review:
role: PC
data: comment: The paper presents a general teacher-student approach for differentially-private learning in which the student learns to predict a noise vote among a set of teachers. The noise allows the student to be differentially private, whilst maintaining good classification accuracies on MNIST and SVHN. The paper is well-written.

Decision: Accept (Oral)