Better Accuracy with Quantified Privacy: Representations Learned via Reconstructive Adversarial Network

Abstract: The remarkable success of machine learning, especially deep learning, has produced a variety of cloud-based services for mobile users. Such services require an end user to send data to the service provider, which presents a serious challenge to end-user privacy. To address this concern, prior works either add noise to the data or send features extracted from the raw data.  They struggle to balance between the utility and privacy because added noise reduces utility and raw data can be reconstructed from extracted features.
      
      This work represents a methodical departure from prior works: we balance between a measure of privacy and another of utility by leveraging adversarial learning to find a sweeter tradeoff. We design an encoder that optimizes against the reconstruction error (a measure of privacy), adversarially by a Decoder, and the inference accuracy (a measure of utility) by a Classifier. The result is RAN, a novel deep model with a new training algorithm that automatically extracts features for classification that are both private and useful.  
      
      It turns out that adversarially forcing the extracted features to only conveys the intended information required by classification leads to an implicit regularization leading to better classification accuracy than the original model which completely ignores privacy. Thus, we achieve better privacy with better utility, a surprising possibility in machine learning! We conducted extensive experiments on five popular datasets over four training schemes, and demonstrate the superiority of RAN compared with existing alternatives.

Reviews and author responses:

We thank a lot for the comments with cares and insights, which are helpful for improving the quality and readability of our writing. We have addressed all the comments as follows:

Response #1: In the revision, we have added the following justification and explanations on privacy quantification in Section 2, 4 and 5.

First, there is no single standard definition of data privacy-preserving problems and corresponding adversary attacks. And a fundamental problem in it is the natural tradeoff between privacy and utility, which is affected by different data privacy-preserving methods. Our key contribution in this paper is the RAN framework and the training algorithm, which can accommodate different choices of privacy attackers and privacy quantification.  Second, finding the right measurement for privacy is an open problem in itself. To evaluate RAN, one has to pick some quantifications. In the present paper, we chose the “reconstructive error” as the quantification of privacy because it is the most intuitive one to measure the risk of disclosing sensitive background information in the raw data for the given perturbed data (Encoder output).  Third, in the future, we will evaluate RAN using other quantifications of privacy as well in a definitely defined application. For example, we could measure the privacy by the hidden failure, i.e., the ratio between the background patterns that were discovered based on RAN’s Encoder output, and the sensitive patterns founded from the raw data, in the object recognition application.

Response #2: Great help. In the revision, we have added more experiments (with more \lambda settings in RAN) to plot the full Pareto Front of three baselines and RAN, and revised the explanations in Section 3.1. And we also noted that the parameter \lambda can be fine-tuned, e.g., exponentially varied, to read a better tradeoff. 

Response #3: Thanks for pointing out the problems in Eq. 1. The utility is evaluated as the accuracy of a Classifier, i.e., the probability Yi=Yi’, which is a commonly used metric . And we adopted some randomness, e.g., dropout, in the parametric discriminative models (Encoder and Classifier).
 
Response #4: We have added more clarifications on the privacy definition in Section 2.1. In particular, the privacy of Max Min |Ii-Ii’|^2 is defined for each data rather than a dataset, which is different from any anonymization based data privacy-preserving techniques. 
 
Response #5: Although we can plug in any adversary architecture (Decoder) and privacy quantification in RAN, this paper adopts the worst possible Decoder to mirror the Encoder’s architecture. That is, we assume a powerful adversary that knows the Encoder in the training. “An exactly reversed model” stands for a layer-by-layer deconvolutional model (Decoder) with known Encoder’s convolution filter number and size, pooling size and each layer’s connection relationship. In the revision, we have added above clarification in section 2.3.
 
Response #6: Thanks for the comments. We call Eq. 3 and Eq.4 adversarial, as explained in out intuition, they need not be opposite all the time. And we agree that the resulting model is highly affected by the setting of hyper-parameters n and k. In particular, we have compared the settings of k=1, k=2, k=3, and k=4 for each task and finally select the best overall value k=3. As for the number of epoch n, it depends on the usual practices of developers for an acceptable converged result. In our experiments, we use n=10,000 for MNIST, UbiSound and Har with batch size=128, and adopt n=20,000 for CIFAR-10 and ImageNet with batch size=256 and batch size=512, respectively. 

In fact, we have already conducted exhaustive micro-benchmark experiments to determine the current design of RAN. For example, we adopt different options of model architectures, nine weight updating schemes on when and what order to update Encoder, Decoder and Classifier,  and several settings of the important hyper-parameters (e.g., “n” and “k”) to select the empirically optimized one. However, we didn’t present the micro-benchmark results in this paper due to the space limit. In the revision, we have added more explanations on the selection of n and k in Section 2.4.

We thank a lot for the comments with cares and insights, and appreciate your efforts in reviewing our paper, which is helpful for improving the quality and readability of our writing. We are also glad that you support our paper.

We agree that it is essential to justify how the reconstruction error works as a measure of privacy in this paper. In the revision, we have added the following justification on privacy quantification in Section 2, Section 4 and Section 5. We also note that the proposed reconstructive adversarial network (RAN), is not an extension of GAN but only borrows GAN’s thoughts on adversarial training several neural networks, for the data privacy-uniquely problem.

First, there is no single standard definition of data privacy-preserving problems and corresponding adversary attacks. And a fundamental problem in it is the natural tradeoff between privacy and utility, which is affected by different data privacy-preserving methods. Our key contribution in this paper is the RAN framework and the training algorithm, which can accommodate different choices of privacy attackers and privacy quantification. 

Second, finding the right measurement for privacy is an open problem in itself. To evaluate RAN, one has to pick some quantifications. In the present paper, we chose the “reconstructive error” as the quantification of privacy because it is the most intuitive one to measure the risk of disclosing sensitive background information in the raw data for the given perturbed data (Encoder output). 

Third, in the future, we will evaluate RAN using other quantifications of privacy as well in a definitely defined application. For example, we could measure the privacy by the hidden failure, i.e., the ratio between the background patterns that were discovered based on RAN’s Encoder output, and the sensitive patterns founded from the raw data, in the object recognition application.


We thank the comments with cares and insights, which are helpful for improving the quality and readability of our paper. We are glad that you support our paper. We have addressed all the comments as follows:

Response #1: In the revision, we had added a new experiment to zoom in on two categories for clearer utility visualization. In particular, we show the DNN’s deep features and RAN’s Encoder output to illustrate how they push the features to cluster with the “car with/without road” & “sailboat with/without water” images in the feature space.
 
Response #2: We agree that we should provide more details about the decoders. Generally, we set the Decoder to mirror the Encoder's architecture. That is, we assume a powerful adversary that knows the Encoder in training. Because the Encoders are different for different tasks, the Encoders are different too. In particular, we select the architecture of Encoder plus Classifier to be LeNet for MNIST, Ubisound and Har, to be AlexNet for CIFAR-10, and to be VGG-16 for ImageNet. The architectures of Encoder in four cases are different, so the Decoder is varied as well. In the revision, we have added above explanations about Decoder in Section 2.3 and in experiment settings of Section 3.

Response #3: We agree that the description of three baselines should be more precise, especially the DNN and DNN(resized) baseline. In the revision, we have added explanations on the difference/similarity between DNN (resized) and DNN baselines. And explain why we include them as baselines to compare RAN against in Section 3.1.
 
Response #4: We have added more explanations in Section 3.1 about how “the proposed algorithm works as an implicit regularization leading to better classification accuracy than the original model which completely ignores privacy”. As shown in Figure 3, the utility of RAN’s Encoder output is higher than that of DNN. Here the DNN model stands for the non-private feature extractor followed by a non-private classifier.
 
Response #5: We agree that it is necessary to conduct experiments to compare RAN’s performance concerning privacy and accuracy with/without a different kind of layers so that we can back up the argument mentioned in Section 2.2. On the one hand, we have already conducted exhaustive micro-benchmark experiments to determine the current design of RAN. For example, we select different model architectures (layers and building blocks), weight updating schemes of different parts (when and how to update Encoder, Decoder and Classifier) and settings of some important hyper-parameters (the setup of “n” epochs and “k” steps, learning rate) to select the empirically optimized one. However, we only present the most important results in this paper due to the space limit. On the other hand, for all the arguments in Section 2.2, we have added the citation to support them.
 
Response #6: We agree that it is important to justify how the reconstruction error works as a measure of privacy in this paper.  In the revision, we have added the following explanation and justification on privacy quantification in Section 1, Section 2, Section 4 and Section 5.

First, there is no single standard definition of data privacy-preserving and corresponding adversary attacks. And a fundamental problem is the natural privacy-utility tradeoff which is affected by different data privacy-preserving methods. We note that our principal contribution in this paper is the RAN framework and the training algorithm, which can accommodate different choices of privacy attacker and privacy quantification. 

Second, finding the right measurement for privacy is an open problem in itself. To evaluate RAN, one has to pick some quantifications. In the present paper, we chose the “reconstructive error” because it is the most intuitive one to measure the risk of original data disclosure given perturbed data (Encoder output). 

Third, in the future, we will evaluate RAN using other quantifications of privacy as well in a defined application. For example, we could measure the privacy by the hidden failure, i.e., the ratio between the background patterns that were discovered based on RAN’s Encoder output, and the sensitive patterns founded from the raw data, in an object recognition task.
 
Response #7: Thanks for pointing out the citation problem in Section 3.1. In the revision, we have added explanation and cited more articles about several attacks for how the raw data can cause privacy risks in Section 1. For example, underlying correlation detection, re-identification and other malicious mining. As for the “Noisy Data” method, we have added the citation on differential privacy in Section 3.1.
 
Response #8: We have re-plotted Figure 3 and Figure 4 to improve the readability.


Privacy concerns arise when data is shared with third parties, a common occurrence. This paper proposes a privacy-preserving classification framework that consists of an encoder that extracts features from data, a classifier that performs the actual classification, and a decoder that tries to reconstruct the original data. In a mobile computing setting, the encoder is deployed at the client side and the classification is performed on the server side which accesses only the output features of the encoder. The adversarial training process guarantees good accuracy of the classifier while there is no decoder being able to reconstruct the original input sample accurately. Experimental results are provided to confirm the usefulness of the algorithm.

The problem of privacy-preserving learning is an important topic and the paper proposes an interesting framework for that. However, I think it needs to provide more solid evaluations of the proposed algorithm, and presentation also need to be improved a bit.

Detailed comments:
I don’t see a significant difference between RAN and DNN in Figure 5. Maybe more explanation or better visualization would help.
The decoder used to measure privacy is very important. Can you provide more detail about the decoders used in all the four cases? If possible, evaluating the privacy with different decoders may provide a stronger evidence for the proposed method.
It seems that DNN(resized) is a generalization of DNN. If so, by changing the magnitude of noise and projection dimensions for PCA should give a DNN(resized) result (in Figure 3) that is close to DNN. If the two NNs used in DNN and DNN(resized) are different, I believe it’s still possible to apply the algorithm in DNN(resized) to the NN used in DNN, and get a full trace in the figure as noise and projection changes, which would lead to more fair comparison.
The abstract mentioned that the proposed algorithm works as an “implicit regularization leading to better classification accuracy than the original model which completely ignores privacy”. But I don’t see clearly from the experimental results how the accuracy compares to a non-private classifier.
Section 2.2 mentioned how different kind of layers would help with the encoder’s utility and privacy. It would be better to back up the argument with some experiments.
I think it needs to be made clearer how reconstruction error works as a measure of privacy. For example, an image which is totally unreadable for human eye might still leak sensitive information when fed into a machine learning model. 
In term of reference, it’s better to cite more articles with different kind of privacy attacks for how raw data can cause privacy risks. For the “Noisy Data” method, it’s better to cite more articles on differential privacy and local differential privacy.
Some figures, like Figure 3 and 4, are hard to read. The author may consider making the figures larger (maybe with a 2 by 2 layout), adjusting the position of the legend & scale of x-axis for Figure 3, and using markers with different colors for Figure 4. 


Summary: The paper studies the problem of training deep neural networks in the distributes setting while ensuring privacy. Each data sample is held by one individual (e.g., on a cell phone), and a central algorithm trains a learning model on top of this data. In order to protect the privacy of the individuals, the paper proposes the use of multi-layer encoders (E) over the raw data, and then send them across the server. The privacy is ensured by exemplifying the inability to reconstruct the original data from the encoded features, via running a reverse deep model (X). The notion of privacy is quantified by the Euclidian distance between the reconstructed vector via the best X and the original feature vector, maximized over E. The overall framework resembles a GAN, and the paper calls it RAN (Reconstructive Adversarial Network).

Positive aspects: The problem of training privacy preserving deep models over distributed data has been a significant and important challenge. The current solutions that adhere to differential privacy based approaches are not yet practical. In my view, it is a very important research question.

Negative aspects: One major concern I have with the paper is the notion of privacy considered. The notion of privacy considered in the paper makes two assumptions which I am not comfortable with: i) The protection that the notion assures is against reconstruction attacks. There has been a large body of work which shows that weaker attacks like membership attacks can be equally damaging, ii) Privacy is a worst-case guarantee. I do not see the GAN style approach taken by the paper, ensures this. 

The privacy definition employed in this work is problematic. The authors claim that "Privacy can be quantified by the difficulty of reconstructing raw data via a generative model". This is not justified sufficiently. Why larger reconstruction error achieves stronger privacy protection? I could not find any formal relationship between reconstruction error and privacy. 

The proposed method is not appropriately compared with the other methods in experiments.  In Fig. 3 the author claim that the proposed method dominates the other methods in terms of privacy and utility but this is not correct. At the specific point that the proposed method is evaluated with MNIST and Sound, it achieves better utility and better "privacy". However, the Pareto front of the proposed method is concentrated on a specific point. For example, the proposed method does not achieve high "privacy" as "noisy" does. In this sense, the proposed method is not comparable with "noisy". In my understanding, this concentration occurs because the range of \lambda is inappropriately set. This kind of regularization parameter should be exponentially varied so that the privacy-utility Pareto front covers a wide range. 

--
Minor:
In Eq. 1, the utility is evaluated as the probability Yi=Yi'. What randomness is considered in this probability?
In Eq 2, privacy is defined as maxmin of |Ii - Ii'|. Do you mean privacy guaranteed by the proposed method is different for each data? This should be defined as expectation over T or max over T. 

In page 4. "The reason we choose this specific architecture is that an exactly reversed mode is intuitively the mode powerful adversarial against the Encoder." I could not find any justification for this setting. Why "exactly reversed mode" can be the most powerful adversary? What is an exactly reversed mode?

Minimization of Eq. 3 and Eq. 4 contradict each other and the objective function does not converge obviously. The resulting model would thus be highly affected by the setting of n and k.  How can you choose k and n?

Meta-review: Following the unanimous vote of the reviewers, this paper is not ready for publication at ICLR. A significant concern is that the definition of privacy used here is not adequately justified. This opens up issues of: 1) possible attacks, 2) privacy-guarantees that are not worst-case, among others. 

Decision: rejected-papers