Scrunch: Preventing sensitive property inference through privacy-preserving representation learning
Authors: Vittorio Prodomo
Roberto González-Sánchez
Marco Gramaglia
Venue: Submitted to ICLR 2023 (2023)
Keywords: privacy
machine learning as a service
center loss

Abstract: Many tasks that are commonly performed by devices attached to the Internet are currently being offloaded to the cloud, using the Machine Learning as a Service (MLaaS) paradigm. While this paradigm is motivated by the reduced capacity of mobile terminals, it also hinders privacy associated with the data exchanged over the network. Thus, the data exchanged among parties shall be conveniently anonymized to prevent possible confidentiality and privacy issues. While many privacy-enhancing algorithms have been proposed in the past, they are usually relying on very complex models that make difficult their applicability to real-world systems or envision too friendly attacker models. In this paper, we propose a deep learning system that creates anonymized representations for the data, while keeping the accuracy for the targeted MLaaS task high, assuming that the attacker can re-train an adversarial model. Our results show that the proposed algorithm i) is effective yet it uses a lighter approach than state-of-the-art ii) considers less friendly attacker models, and iii) outperforms the benchmark under different privacy metrics.

Under review as a conference paper at ICLR 2023
Scrunch: PREVENTING SENSITIVE PROPERTY INFER- ENCE THROUGH PRIVACY-PRESERVING REPRESENTA- TION LEARNING
Anonymous authors
Paper under double-biee brindd review
ABSTRACT
Many tasks that are commonly performed by devices attached to the Internet are currently being offloaded to the cloud, using the Machine Learning (MLaaS) paradigm. While this paradigm is motivated by the reduced capacity of mobile terminals, it also hinders privacy associated with the data exchanged over the network. Thus, the data exchanged among parties shall be conveniently anonymized to prevent possible confidentiality and privacy issues. While many privacy-enhaneee algorithms have been proposed in the past, they are usually re- lying on very complex models that make difficult their applicability to real-world systems or envision too friendly attacker models. In this paper, we propose a deep learning system that creates anonymized representations for the data, while keeping the accuracy for the targeted MLaaS task high, assuming that the attacker can re-train an adversarial model. Our results show that the proposed algorithm i) is effective yet it uses lighter approach than state-of- of-te-of-the-art ii) considers less friendly attacker models, and iii) outperforms the benchmark under different pri- vacy metrics.
Service
INTRODUCTION
The complexity and size of ML models growing over time. Recent examples, such as GTP-3 with 175B parameters [OpenA] (2022 (accessed September 28, 2022 or Megatron- 530B (Nvidia (2022 (accessed September 28, 2022)], have presented models that are impossible to generate even maintain for most companies in the world, not to speak about academia users with personal devices. Moreover, it is expected similar growth the next years [Fedus et al. (202T)]- This progression, together with the slowdown in the production of new hardware severely limits the capacity of small (and even big) enterprises to use the last advances in Natural Language Processing (NLP), image recognition, other complex ML tasks.
with
In this scenario, big tech companies have started to offer their models in a Machine Learning as Service (MLaaS) fashion. That is, they run the gigantic ML models on their premises and allow customers to query the model for a pre-negotiated fare. This model is convenient for both customers that do not have the ability to create their own complex model (i.e., because they do not have tagged dataset), and for those that need to execute (even simple) ML tasks on limited devices such as mobile phones or IoT devices.
However, to perform MLaaS task, the customer should send the raw data (e.g., an image) to the service provider. While this operation may not present big problems certain tasks (e.g.. connected vehicle sending telemetry data for predictive maintenance), it certainly has heavy pri- vacy/confidentiality implications in others (e.g.., a surveillance system requesting image classifica- tion services).
Alternatively, the service provider could give the model to the customer to avoid data transfer. Nonetheless, this is typically not feasible in the case of limited devices or huge models. And even in cases where the customer could execute the model, the MLaaS provider may have concerns as the customer could blackbox or the model without the provider's permission.

---

Under review as a conference paper at ICLR 2023
In this paper, present Scrunch, technique that allows the usage of MLLaS without the privacy implications of sending raw data to third parties. In our technique, a previously trained model is split into two parts and then its first part is fine-tuned adding second loss function, to ensure the information after this point is only valuable to perform the task at hand, but not to perform any other task. The usage of pre-trained model allows the easy usage of already existing models without the need of training them from scratch.
After the two parts are both trained taking into account the new, combined loss function, the first part can be sent to the customers that can execute it even with limited resources, and only transfer the obtained data representations. The rest of the model stays within the service provider ensuring that customers cannot make non-legitimate usage of the entire model from the provider.
Scrunch is able to create crivacy-preservin data representations. It provides accuracy similar to the one of a neural network without privacy and, at the same time, provides higher privacy than state-of-the-ar privacy solutions.
In the remaining of this paper, we present the privacy model in Section 2 then, we implement our solution for two different Neural Network architectures and data sets and evaluate its performance in Section 3 In Section 4 we show how the model parameters affect its way of working. Finally,
Section concludes the paper.
1.1 STATE THE ART
The application of privacy-preservve techniques to data sharing and ML has been widely studied in the past years with solutions ranging from the already classic k-anonymity (Sweeney) (2002]], I-diversity [Machanavajhhala et al. (007)] or t-closeness [Li et al. (2007) to more novel solutions such 7-anonymity [Jha et al. (2020)). Among all of them, Differential Privacy (DP) (Dwork et al.
[2006]] is, with absolute certainty, the most accepted and used by the MI. community.
DP grants formal guarantee about how likely the data is leak sensitive information, i.e. info beyond what is legitimately intended to be publicly available by the data owner. The problem to be solved in our scenario, instead, concerns 'inference privacy', i.e. reducing the amount of information that is sent/published in the first place. In addition, applying DP e.g. in the form of noise- to the data with no further tweaks usually tends to quickly degrade the whole informational content, including what should be kept usable.
Other approaches that try to preserve the privacy of exchanged data are those that employ advanced cryptographic techniques. Two particularly researched approaches today are Fully Homomorphic Eneryption (FHE) and Secure Multi-Party Computation. Thanks to FHE, direct inference on en- crypted data becomes possible [Gilad-Bachraad And since data never decrypted, its privacy is guaranteed. FHE usually suffers from an aceu- racy drop with complex networks, since it works by approximating neural network with low degree polynomial function. But the real major drawback is the computational cost: the encryption schemes' complexity makes the inference time increase by many orders of magnitude, making it impractical for real-time use cases. Another cryptographic approach is Secure Multi-Party Compu- tation (SMC), which makes possible for two entities to compute function over their inputs while maintaining those inputs perfectly private [Mohassel & Zhang (2017); 7): Liu et al. (2017)). Usually, SMC scenarios are based on garbled circuits, secret sharing, and oblivious transfer. SMC also suf- fers from high cryptographic complexity. Another popular field of research concerns about how to securely run an ML model on a cloud machine. Proposals from this field rely on trusted execu- tion environments such as Intel SGX and ARM TrustZone (Tramer & Boneh 2018); Hanzlik et al.) 2021) Nevertheless, such scenarios still require the client trust the cloud servers with their data.
et al. (2016); /eenty (2009); Bos et al. [.
Finally, more similar to our work, there is another sub-field of the Prreacy-r research com- munity that tries to generate privacy-preserve
data representations. AutoGAN [Nguyen et al. ] proposes non-linear dimension reduction framework based on GAN structure. On it, Generator and a Discriminator are iteratively trained in turn, in an adversarial manner, to enforce certain distance between original and potentially reconstructed data. Osia et al. (2018) protects the data against the execution of previously known ML tasks. Moreover, works like [Osia et al.) 2020)). Cloak (Mireshghallah al. 2020aa) and Shredder [Mireshghallah al. (020b) apply
2

---

Under review as a conference paper at ICLR 2023
Client
Server
Encoder
Mallcious Classifier
6
Data
3
Classifier
Anorymized
2
Prediction
5
Figure I: High-level scenario.
contrastive loss to privatize the data with small differences among them the network structure and application. Contrary to all them, we employ the center loss in our work, allowing that way better privacy protection without the need of complex Siamese Networks.
MODEL
The Scrunch model uses as a starting point a pre-trained neural network, then it splits into two parts and makes modifications it to improve the privacy provided by the final solution. In this solution, the computation of the inference is shared between a client, that executes the first part of the neural network (namely, the Encoder) and sends the output features to Server (i.c., in the cloud) that executes the second part (namely, the Classifier) and returns the MI. task answer.
Figure shows summary of how the general model works in inference: 1) The server sends an Encoder to the Client. 2) The client uses the encoder to generate anonymized representations of the raw data. 3) The client sends the anonymized data the Server. The server to solve the ML task at hand and 5) sends the prediction back to the client. 6) Even using the data available to the Server, the latter- or another, malicious entity cannot train different Classifier to solve different task, based solely on the privatized representations.
the Classifier
Following, describe how efficiently generate an Encoder and Classifier maximize the accuracy for the task to be solved, while decreasing the performance of other non-related tasks over the anonymized data.
2.1 ENCODER AND CLASSIFIER
We build on the intuition that features obtained in the middle of a neural network an abstract representation of the input data and can be used to share data in a privacy-preserving way. However, without additional modifications, such features may still contain significant amount of extra infor- mation, as demonstrated for example by the visualization techniques Dosoviiskki & Brox (0016)].
Moreover, is well known that, the deeper we go through the layers of neural network, the more specialized, abstract (Yosinski et al. (2014)) and specific to the main task the features become. Furthermore, in most networks design, going deeper means also having to deal with much fewer dimensions. This all collaterally contributes to data privacy: whatever extra information was con- tained the data - beyond what actually useful to perform main task - goes gradually and irreversibly lost [Osia et al. 2017); Malekzadeh et al. (2018)).
In the extreme case, when the network split is done after the last layer, the complete MI. task would be executed by the client, obtaining that way "perfect privacy". However, this situation is not realistic in most of the use cases, either because the client cannot run the complete neural network due to hardware limitations, because the service provider does want to share the complete model with the client.
Thus, rule of thumb, should first choose the split point that provides the client with the heaviest model it can support (and the service provider is willing to share). This alone would al- ready grant some degree of privacy depending on the chosen point regarding any other extra
3

---

Under review as a conference paper at ICLR 2023
information the data carries. Then, in addition to this especially in cases where one is constrained to the lower levels of the network- -, other approaches may further enhance the privacy of the data.
2.2 ADDING PRIVACY
In a scenario where no knowledge about potential sensitive features is assumed (i.e. no private labels), the only reasonable choice is to try and reduce the general amount of information contained in the data, while constraining the accuracy of the main task to be as high as possible. In a more formal way, considering the input data and their corresponding we would like to reduce their Mutual Information (MI) as much as possible, while still keeping the cross-entropy loss of the whole model as low as possible. The latter is embodied in the typical softmax cross-entropy loss, computed on the output of the very last layer of the intact model: basically keeps the points belonging to different classes well separated in the encoding space. In other words, it tries to maximize the inter-class distance.
preseeevign representations,
In addition to this, we would also like to minimize the inter-class distance, to reduce any extra information contained in the structures of same-class points. Other works have tried to achieve this
by employing siamese network architectures and contrastive or triplet losses [Osia et al. (2020);
Mireshghallah et al. (. The problem is these methods (Sun et al.] 2014); Wen et al. 201 6a);
Schroff al. (2015) suffer from non-negligible data expansion since the training set must be recombined in specific pairs. Furthermore, since these losses need pairs of points to be computed, two forward passes are needed, thus increasing the training time. That is why we chose instead to employ the Center Loss function, originally proposed by [Wen et al. (2016b)]. It is worth noting that, same as for triplet and contrastive losses, this loss was not primarily designed for purposes, but to improve the discriminative power of the deeply learned features. As far as we know, this is the first time it actually applied to such goal. The center loss is defined as:
1 m
Lo
Σ = 2 i=1
||=
(1)
where m is the total number of classes in the dataset, is the i-th encoding, and Cy: is the center of the y4 class. Hence, the term introduced by Le minimizes the euclidean distance between any encoding classified to that class and the center of such class. We apply this function done by Wen (ct al. (2016b), combining with the usual softmax categorical cross coos-cyy Ls. The two losses are governed by weight factor À as follows!]
C= + (1-
(2)
In other words, we train the model with Joint Supervision, as discussed in the work by Wen et al. (2016)). The variable À plays fundamental role in steering the encoding anonymization process, and weights how the encodings are anonymized. First, we remark that these two terms need to be jointly optimized (so, configurations where À is 0 or meaningful in a real scenario), but À also has meaning for the anonymization process. When À = 0, the model is trained to only optimize the cross-entropy among classes, as a typical classifier would do. Thus, encodings may still exhibit intra-class variations that may leak sensitive information, as shown by Dosovitskiy & Brox (2016). Basically, when À O, no anonymization is applied beyond what is already provided by the deep features. Instead, when X = 1, no effort is made by the network to separate the classes, thus yielding poor results on the machine learning task. Hence, the correct parametrization of X is fundamental to steering the operation of the system toward an optimal balance between accuracy and privacy.
asymptotic cases that not necessarily
Assuming have pre-trained model for our 'Public Task', and once the split point has been decided, Figure presents in this case specifically for the VGG16 model an example of how our pipeline would work when modifying the architecture:
2
1.
The architecture is rearranged so that it becomes multi-output network, with the activa- tions of the split point going two separate ways:
The natural continuation of the original network, i.e. the Dense Part
For more details about the loss function implementation, please refer to Wen al. (016b)
4

---

Under review as a conference paper at ICLR 2023
Back-Propagation
Crosp-Erirspy Badk Badk-Propagelon
Original Network
Fixed Auto Encoder
Center Layer
Center
Dimensionality Reduction
Our Addition
Combined
Loss
VGG16
Crose- Entropy
Convolutional Part
Loss
Input Images
Mid Activations
2
Figure 2: Model Architecture
The new added branch, composed of fixed pre-trained Autoencoder for dimen- sionality reduction followed by the main component of the approach, i.e. the Center Loss Layer
2. The whole network is carefully fine-tuned with low learning rate and Joint Supervision
During the forward pass, the CL layer "learns" new, more accurate class centroids, based on the current batch. Such centroids are then used to compute the Center Loss
The weights in the Dense part are adjusted thanks the Cross-Entropy back- propagation, as usual
The Convolutional Part, instead, is trained by the nack-propagation of both losses, that it produces features that retain both privacy and utility
3. Once the training is finished, the first part of the network will be frozen and deployed on the client side as a black-box anonymizer
3 EXPERIMENTAL EVALUATION
We implemented Scrunch work with two different network architectures and datasets. Further- more, we tested it against some of the privacy metrics that are in line with the attacker model pre- sented Sec. benchmarking against the results obtained by another privacy-preserving solution: Shredder. For this, we re-implemented the original Shredder source code in Tensorflow/Keras, as also Scrunch is implemented using the same framework.
3.1 DATASETS AND NETWORK ARCHITECTURES
We employ two different architectures both for image classification to test our approach: a LeNet-5 neural network with the MNIST dataset, for which also provide direct comparison with another state-of-thc-art approach [Mireshghallah et al. [2020b)] and VGG16 neural network, with the CelebA dataset.
The LeNet-5 network takes as input a 32 X 32 X image. The channel is just one because the network is designed use greyscale images. The architecture then consists of 5 learnable layers. The first three of these are convolutional (Conv) layers with 6, 16, and 120 filters, respectively. All three layers use kernel size of (5,5). After the 1st and the 2nd Conv layers, we also find Max Pooling (MP) layer. Finally, the last two are Fully Connected (FC) layers. The first has 84 neurons, while the second and last FC layer has usually neurons since the digits in the MNIST dataset are 10. Lastly, such layer is usually followed by Softmax layer that classifies the images into the corresponding classes.
The MNIST dataset [LeCun & Cortes 2010 used to test this case is composed of greyscale images of size 32x32 pixels, representing handwritten digits going from '0' to '9'. The training set contains 60,000 different examples of images, while the test set contains another 10,000 example images for model evaluation. The labels are simple integer numbers from to that indicate the specific digits in the images
5

---

Under review as a conference paper at ICLR 2023
The other network in use is the well-known VGG16 [Simonyan & Zisserman (2014) a 16-layer- deep CNN (Convolutional Neural Network) primarily designed to perform image classification. In particular, use the pre-trained version of VGG16 trained on the huge ImageNet dataset (Deng et al. [ and we first fine-tune it for the 'public task', via transfer learning. The network consists of first, convolutional part, and then a second, fully connected part. The first part is composed of 5 macro-blocks, Each of these blocks is stack of 2 or 3 Convolutional layers, always followed by a Max Pooling Layer. For each block, all the conv layers have the same number of filters. From the 1tt to the 5th block we have, respectively, 64, 128, 256, 512, and 512 filters. The 2nd part of the network is simply made up of Flatten layer followed by two Dense- cense-Dropout blocks and the final Dense Layer. The first two Dense layers have both 4096 neurons, and both dropouts have a probability of 0.5, while the last Dense Layer has a number of neurons that depends on the number of classes of the specific task at hand.
In this case, we use the CelebA dataset [Liu et al. (2015)). The original CelebA dataset consists of 202.599 images of celebrities' faces, of variable sizes. We crop these images to fixed size of 128x3 pixels. The images are colored, hence the three channels. The dataset comes with a huge set of labels: 10,177 identities; 5 landmark locations for each image; 40 binary attributes annotations per image. We limit ourselves to two sets of binary labels: the gender and a label that indicates whether the person in the photo is smiling
not. We use gender as our primary/public task, and purposefully choose a simple binary attribute such as smiling/not-smillngg as our malicious/private task. We do this in order to prove that our approach works for hindering even such a simple task, as opposed to choosing something that would more likely be private information - and intuitively more difficult to leak such as identity.
3.2 PRIVACY METRICS
Mutual information (MI) is an information-theoreti notion. Assuming two data points x and y.
MI(x.y) quantifies the average amount of information that leaks (i.e. is learnable) from y about
This measure is commonly employed in literature [Cuff' & Yu (006); Kalantari et al. 2017
both as an anonymity metric when dealing with database potential leakage [angg et al. (2016);
Liao et al. [2017)] and to better explain the behavior of neural networks [Saxe et al. (2019)). In our experiments, we calculate the Mutual Information Drop experienced when comparing the MI between raw data and simple mid-network features (no enhanced privacy) with the MI between raw data and the data representations obtained after the encoder.
Moreover, since the encoder is public (to all possible clients, and the server itself), malicious entity could try to retrain a Classifier in order to solve different task than the one requested by the Client. Thus, we compute the normalized Private Task Accuracy Drop of an adversary classifier. That is, how worse the privacy task results are when trained on deep features or data representations, with respect to typical classifier free to train the private task on the raw input data.
Proving that the chosen 'sensitive' task fails does not guarantee that any other potentially sensitive information is safe, of course. Hence, we resort to also testing how well a full decoder can be trained on some leaked data, in order to reconstruct the original input images from the data representations. Intuitively, if the reconstructed images are too similar to the original ones, it means that the extra information contained in the image may potentially leak from the data representations obtained. We not only provide a visual comparison for such similarity but also a quantitative measure, in the form of the Structural Similarity Index Measure (SSIM) [Wang et al. (, metric specifically designed to mimic human perception of 'mmage similarity" In its simplest form, SSIM is represented by number between -1 and 1 usually rescaled between and 1 where - means "completely different" and I means "exactly the same".
The architectures of the adversary classifier and decoder are the following:
For the classifier, we take what remains of the VGG16 or the LeNet-5 network after the chosen split point, and retrain it with the private labels and the anonymized data represen- tations of holdout set.
For the decoder, we employ a simple custom upsampling network for the LeNNSSS/MNIST case and the AlexNet decoder (Dosovitskiy & Brox (2016)) for the VGG16/CelebA case.

---

Under review as a conference paper at ICLR 2023
Table 1: Accuracy and privacy results.
Normalized Public
Private Task
Mutual Info
95th Percentile
Dataset
Algorithm
Task Accuracy (%)
Accuracy Drop (%)
Drop (%)
SSIM Drop (%)
NoPrvary/Deer-eatuues
100.0
00.36
N/A
10.19
MNIST
Shredder
98.93
23.98 (1.1
66.97
91.34 (75.57)
Scrunch
99.34
60.56
79.66
81.75
NoPivacylDeeteenues
100.0
29.86
N/A
39.94
CelebA
Scrunch
99.73
40.02
29.84
70.8
We train it with a holdout set of anonymized data representations, and their original coun- terparts as labels.
3.3 PRIVACY RESULTS
Table T shows the obtained privacy results against the metrics we introduced above. Higher values mean better performances, in all columns. The public accuracy is normalized by the accuracy that the models reach absence of any privacy-preserving approach, for their public tasks. The private accuracy loss is computed with respect the accuracy that full, free model would obtain by training with the private labels. We tested two popular datasets, MNIST and CelebA, using LeNet-5 and VGGI6 respectively as the backbone network for the classification tasks. Both networks have been split right after their convolutional part (layer block5_poo) for VGG16 and conv _3 for LeNet-5). For the training of the encoder, we split the original dataset into Training, Validation, and Test with a 40%, 40%, 20% proportion for both the solutions. When training the adversary attacker classifiers we also use a similar split. For Scrunch the best results are obtained with = 0.9, with a learning rate of 10-5 The adversary classifiers and decoders are trained with learning rates of 10-5 and 10- resp.reisveeeeiie respectively. Shredder is configured as in the original paper.
Results show that for both the datasets, Scrunch can basically retain all the accuracy on the public task as if there was no privacy solution applied, with a drop that is below 1 1% for MNIST and 0.26% for CelebA. Also, Shredder is on par with these results. However, Scrunch proves to be much more powerful in dropping the accuracy in the private tasks than Shredder. For the MNIST case, where both implementations can be compared. we more than double the privacy drop obtained by Shredder, approximating the performance of a random guesser.
This is also partially explained by the loss in the mutual information (13% higher in Scrunch), which quantifies the amount of information about the original data points - that leaks from the data representations. Similar considerations apply also to the CelebA dataset (for which we do not have a working Shredder implementation). Here we can drop the accuracy on the private task, while the mutual information is less affected due the high dimensionality of the input raw data.
It is fundamental to remark here that, besides obtaining better privacy metrics than a state-of-the-ar solution, Scrunch does that while enforcing a much more stringent privacy scenario. While Shredder does not retrain the adversarial attacker on possibly similar data with available labels (an aspect that more appropriate in many cyber-attack scenarios), in our approach we let the adversary re- re-train on leaked data, to improve the effectiveness of the attack. Still, even in this extremely challenging scenario, we can obtain better results than Shredder for MNIST and a significant decrease in the accuracy for the private task for CelebA.
Figure provides an overview of the visual reconstructions. Without privacy, the raw data can be reconstructed in a more or less straightforward way from the decoders. depending the network complexity and the chosen split point. Particular mention should go to the VGG16/CelebA case (right image): while the simple embeddings i.e. with privacy are unmistakably much less recognizable than their original counterparts, it is still possible, even for a human observer, to dif- ferentiate sensitive features (e.g. whether the person is smiling). Such sensitive feature is then lost when Scrunch is applied, leaving the public task the only one feasible. The left image provides a quick intuition of how Scrunch works with respect Shredder and other naive solutions. Scrunch "learns" how to blend together samples from the same public classes, making them indistinguishable
This is the obtained value when the adversary classifier is retrained with holdout pool of data
7

---

Under review as a conference paper at ICLR 2023
Female
Female
Male
Male
Class
5 Class
Original
Original
{
3
7
8
9
No Privacy
1
2
3
7
8
9
Privacy
Shredder
t
1
3
7
Scrunch
6
8
7
7
?
Smiling
Smillng
(a) MNIST dataset
(b) CelebA dataset
Figure 3: Example images reconstructed from the data representations obtained after the encoder.
10
Males
Averace Intra Llar--lless Distance
Females
10P
0.59
0.993
0.998
Value
Figure 4: Aitra-Class distance for different values of
from the private ones. Shredder, instead, works by learning "where to add noise" on each sample, technique that proved to be less resilient in our benchmarks.
PARAMETRIZATION
As discussed in Section 2 Scrunch can be steered through the configuration of While in Section discussed the results with the best X, in this section discuss the effect of À steer the operation of the system, using the datasets introduced above.
In Scrunch, À is used to weight the categorical cross entropy and the center loss. F the intra-cluster distance for the two public categories of the CelebA dataset. If we take as reference the results for À increasing it allows for reducing the intra-cluster distance by order of magnitudes. By reducing the intra-cluster distance, using the same data for tasks other than the public one results in lowered accuracy, as discussed earlier.
We further dig into the result by analyzing how À affects the utility versus privacy trade-off, that is how the data transformation for privacy purposes affects the accuracy of the public task. We depict it in Figure 5) indicating the confidence interval in the accuracy averaged over 10 repetitions for the CelebA dataset. We can clearly observe two areas: the first for higher that yields higher privacy at cost of lower public accuracy. The public accuracy plateaus instead in the second region, where decreasing À does not help the public accuracy, while quickly increases the private one. Thus, by effectively control this trade-off.
changing À
Another way assessing how the inter-class distance reduced by lambda by plotting the 2-D embeddings using t-SNE for different À values. We depict it in Figure 6 for the MNIST dataset, where the upper row colored to represent the clusters for the private task and the bottom row is representing the public one. While there is clear separation for lower À for both public and private clusters, the effect growing À blends points belonging to the same public cluster but different private ones, achieving thus drastic drop in accuracy for the private tasks shown in Table T
8

---

Under review as a conference paper at ICLR 2023
1.0
97.8%
Pubbblc Accuracy
0.7
0.6
0.8
97.4%
0.9
0.85
0.6
97.2%
0.98
A
97.0%
99d
0.4
89
0.995
96.8%
Adversary Baseline
0.2
96.6%
Public Baseline
60.0%
62.0%
64.0% Private Accuracy
66.0%
68.0%
70.0%
Figure Utility/Priveey Trade-off
5
55 055
Raw Data Points
À = 0.0
À = le - 05
À = 0.01
À = 0.9
Figure 6: t-SNE of encodings for different Lambda values (MNIST dataset)
4.1 LIMITATIONS
While increase in the value of comes with significant increase in the privacy obtained, also comes with a major difficulty for the neural network to converge during the training. For example, over 70% of the training phases failed converge and started overfitting, triggering an early stop after 70 epochs (on average) when À 0.999, while all the trainings converged in less than 40 0.7. Moreover, the average time to complete training also grows with A. Still,
epochs for while the increase in training time appears to be linear, the decrease in the intra-cluster distance is exponential.
DISCUSSION AND CONCLUSION
This paper presents Scrunch, an ML technique to learn privacy preserving data representations of data that can be used in MLaaS scenarios. In Scrunch, the client locally executes an encoder over the raw data and only shares the privatized data representations with the MLaaS Server, ensuring that way that the server (or other malicious entities) cannot use the data to perform any task different than the one requested by the Client.
In Scrunch, the encoder used by the client, and the classifier used by the server are generated by splitting an already trained neural network, modifying by adding new layer that ensures the intra-class distance minimization (i.c., the center loss) and retraining the whole model using joint- supervision.
Scrunch has demonstrated to provide much better privacy than state-of-the-arr solutions (38% and 13% Private Task Accuracy Drop and Mutual Information Drop, respectively) while keeping the accuracy for the public task virtually unaffected in spite of ensuring much more realistic privacy model. Finally, we show how Scrunch can be parameterized to steer its operation, for instance by trading privacy for accuracy and reducing the training time.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Joppe W Bos, Kristin Lauter, Jake Loftus, and Michael Naehrig. Improved security for ring-based fully homomorphic encryption scheme. In IMA International Conference on Cryptography and Coding, pp. 45-64. Springer, 2013.
Paul Cuff and Lanqing Yu. Differential privacy as a mutual information constraint. In Proceedings of the 2016 ACM SIGSAC Conference Computer and Communications Security, pp. 43-54, 2016.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hier- archical image database. 2009 IEEE Conference on Computer Vision and Pattern Recognition, pp. 2009. doi: 10.1 (09////V..0009.220066
Alexey Dosovitskiy and Thomas Brox. Inverting visual representations with convolutional networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4829- 4837, 2016.
Cynthia Dwork, Frank McSherry, Kobbi Nissim, and Adam Smith. Calibrating noise to sensitivity in private data analysis. In Shai Halevi and Tal Rabin (eds.), Theory of Cryptography, pp. 265-284, Berlin, Heidelberg. 3 2006. Springer Berlin Heidelberg.
William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity, 2021.
Craig Gentry, Fully homomorphic encryption using ideal lattices. In Proceedings of the forty-first annual ACM symposium on Theory of computing, pp. 169-178, 2009.
Ran Gilad-Bachrach, Nathan Dowlin, Kim Laine, Kristin Lauter, Michael Naehrig, and John Werns- ing. Cryptonets: Applying neural networks to encrypted data with high throughput and aceuracy. In International conference machine learning, pp. 201-210. PMLR, 2016.
Lucjan Hanzlik, Yang Zhang, Kathrin Grosse, Ahmed Salem, Maximilian Augustin, Michael Backes, and Mario Fritz. MIcapsule: Guarded offline deployment of machine learning as a ser- vice. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3300-3309, 2021.
N. Jha, T. Favale, L. Vassio, M. Trevisan, and M. Mellia. z-anonymity: Zero-delay anonymiza- tion for data streams. In 2020 IEEE International Conference Big Data (Big Data), pp- 3996-4005, Los Alamitos, CA, USA, dec 2020. IEEE Computer Society. doi: 10.1109/ SigDaata500222200.993882222 URL /ectps://doi.oe het.l./eeeeccppetrrreeeeoo.ooo/. 2020. 93 9378422
Kousha Kalantari, Lalitha Sankar, and Oliver Kosut. On information- thhooretccc privacy with general distortion cost functions. In 20/7 IEEE International Symposium on Information Theory (ISIT), pp. 2869. JEEE, 2017.
Yann LeCun and Corinna Cortes. MNIST handwritten digit database. 2010. URL http: lecun. www./exbb//m.sseeecceeeee//e/eee
N. Li. T. Li, and S. Venkatasubramanrian. t-closeness: Privacy beyond k-anonymity and I-diversity. In 2007 IEEE 23rd International Conference on Data Engineering, pp. 106-115, 2007.
Jiachun Liao, Oliver Kosut, Lalitha Sankar, and Flávio P Calmon. A general framework for infor- mation leakage, 2017.
Jian Liu, Mika Juuti, Yao Lu, and Nadarajah Asokan. Oblivious neural network predictions via minionn transformations. In Proceedings of the 2017 ACM SIGSAC conference on computer and communications security, pp. 619-631, 2017.
Ziwei Liu, Ping Luo. Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of International Conference on Computer Vision (ICCV) December 2015.
10

---

Under review as a conference paper at ICLR 2023
Ashwin Machanavajhhala, Daniel Kifer, Johannes Gehrke, and Muthuramakrishnan Venkitasub- ramaniam. 1-diversity: Privacy beyond k-amonymity. ACM Trans. Knowl. Discov. Data,
(1):3-es, es, March 2007. ISSN 1556-4681. doi: 10.1145/1217799.117700. URL https:
7/doi //doi.org/10. 0155/221299.12173772
Mohammad Malekzadeh, Richard G Clegg. and Hamed Haddadi. Replacement autoencoder: A privacy-preserving algorithm for sensory data analysis. In 2018 IEEE/ACM Thirl International Conference Internet-of-Things Design and Implementation (IoTDI), pp- 165-176. IEEE, 2018.
Fatemehsadat Mireshghallah, Mohammadkazem Taram, Ali Jalali, Ahmed Taha Elthakeb, Dean Tullsen, and Hadi Esmaeilzadeh. A principled approach learning stochastic representations for privacy in deep neural inference. arXiv preprint arXiv:2003. /2 1244,, 2020a.
Fatemehsadat Mireshghallah, Mohammadkazem Taram, Prakash Ramrakhyani, Ali Jalali, Dean Tullsen, and Hadi Esmaeilzadeh. Shredder: Learning noise distributions to protect inference privacy. In Proceedings of the Twenty-Fifth International Conference Architectural Support for Programming Languages and Operating Systems, pp. 3-18, 2020b.
Payman Mohassel and Yupeng Zhang. Secureml: A system for scalable privacy-preserve machine learning. 2017 IEEE symposium on security and privacy (SP), pp. 19-38. IEEE, 2017.
Hung Nguyen, Di Zhuang, Pei- Yuan Wu, and Morris Chang. Autogan-based dimension reduction for privacy preservation. Neurocomputing, 384:94-103, 2020.
Nvidia. Using deepspeed and megatron train megatron-turing nlg 530b, the world's largest and most powerful generative language model.
using-deepspeed and- htt.::/www.........aao.aaa.aaaaa 2022 (accessed September 28, 2022).
OpenAL Openai unveils 175 billion parameter gpt-3 language model. htttps:/syycced
mediumm. catmm//peaaiuuueee 2022 (accessed September 28, 2022).
Seyed Ali Osia, Ali Shahin Shamsabadi, Ali Taheri, Kleomenis Katevas, Hamid R Rabiee, Nicholas D Lane, and Hamed Haddadi. trivacy-preservie deep inference for rich user data on the cloud. arXiv preprint arXiv: arXiv:1710.01722 2017.
Seyed Ali Osia, Ali Taheri, Ali Shahin Shamsabadi, Kleomenis Katevas, Hamed Haddadi, and Hamid R Rabiee. Deep private-feature extraction. IEEE Transactions on Knowledge and Data Engineering, 32(1):54-66, 2018.
Seyed Ali Osia, Ali Shahin Shamsabadi, Sina Sajadmanesh, Ali Taheri, Kleomenis Katevas, Hamid Rabiee, Nicholas D Lane, and Hamed Haddadi. A hybrid deep learning architecture for privacy-preservin mobile analytics. IEEE Internet of Things Journal, 755:44005-4588 2020.
Andrew M Saxe, Yamini Bansal, Joel Dapello, Madhu Advani, Artemy Kolchinsky, Brendan D Tracey, and David D Cox. On the information bottleneck theory of deep learning. Journal of Statistical Mechanics: Theory and Experiment, 2019(12):124020, 2019.
Florian Schroff, Dmitry Kalenichenko, and James Philbin. Facenet: A unified embedding for face recognition and clustering. In Proceedings the IEEE conference on computer vision and pattern recognition, pp. 815-823, 2015.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv: 1556, 2014.
Yi Sun, Yuheng Chen, Xiaogang Wang, and Xiaoou Tang. Deep learning face representation by joint iddntification-verfiiattto Advances in neural information processing systems, 27, 2014.
Latanya Sweeney. K-anonymity: A model for protecting privacy. Int. J. Uncertain. Fuzzi- ness Knsel.-Based Syst., 10(5):557-570, October 2002. ISSN 0218-4885, doi: 10.1142/ 80218488502001648 Utt.httts:://aa 1ttp://////oii..ooo00111.1/../// 01444///5222288882200111111
11

---

Under review as a conference paper at ICLR 2023
Florian Tramer and Dan Boneh. Slalom: Fast, verifiable and private execution of neural networks in trusted hardware. arXiv preprint arXiv:1806.03287, 2018.
Weina Wang, Lei Ying, and Junshan Zhang. On the relation between identifiability, differential privacy, and mutual-inforration privacy. IEEE Transactions on Information Theory, 62(9):5018- 5029, 2016.
Z. Wang, A. C. Bovik, H. R. Sheikh, and E. P. Simoncelli. Image Quality Assessment: From Error Visibility Structural Similarity. IEEE Transactions on Image Processing, 13(4):600-612, April 2004. doi: 10. 00000/T1..003.8198660
Yandong Wen, Zhifeng Li, and Yu Qiao. Latent factor guided convolutional neural networks for age-invariant face recognition. Proceedings of the IEEE conference on computer vision and patter recognition, pp.
2016a.
Yandong Wen, Kaipeng Zhang, Zhifeng Li, and Yu Qiao. A discriminative feature learning approach for deep face recognition. In European conference on computer vision, 499-515. Springer, 2016b.
Jason Yosinski, Jeff Clune, Yoshua Bengio, and Hod Lipson. How transferable are features in deep neural networks? Advances in neural information processing systems, 27, 2014.
12


Reviews:
[{"review_id": "uDicmGrfR9", "reviewer": "Reviewer_rRGt", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "As MLaaS rises on cloud, data privacy becomes an issue. Clients dont want to send their raw data directly to the server and server does not want clients to use ML models without permission. This work proposes a framework named Scrunch by using encoder sent from server for clients to generate anonymized representations of raw data and preventing malicious client to train another one for other tasks. ", "summary_of_the_review": "The idea to use features generated by neural network is a good perspective, but this work is considering less on security model and features themselves are still risky for information leakage. Except the smart watch example, I think the proposed model does not even work for NLP applications. As we know, words like “the” will appear frequently and encoding vectors will be quite similar if server provides the same encoder. It is possible for server to deduce frequent words from received features and server is also able to “reverse engineering” whole sentences or passages with cryptographic analysis. Upon the generated features, I think they still need to apply somehow encrypted techniques before sending. Hence, I think this work is not sufficient to address data privacy issue for MLaaS systems on the cloud.", "strength_and_weaknesses": "Strength:\n\n1. Accuracy performance is not significantly affected.\n\nWeakness:\n\n1. This work does not define security model.\n\n2. Using middle features of neural network is still risky to leak information. For example, if client data are collected from smart watch, which contains physiological time-series data, after encoding, it may still contain trends how data change in time.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "YdlsfLEpPWe", "reviewer": "Reviewer_2RqB", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work studies the problem of privacy preserving representation learning. The goal here is to enable secure inference without revealing the client data. The central idea is to split a neural network into two parts — and sending the first part to client. The client applies raw data through the first part ofthe neural network (with some additional loss function) and then sends the encoded data to the server for final inference. The key idea is that by controlling the information at the encoding stage privacy of protected attributes is preserved.\n\nThe experimental analysis compares this method with another method in prior art, namely Shredder, and the non-private counterpart. There two metrics of privacy employed — one to see how much information is revealed by the first part and second one to see the see how much information is revealed in the encoding step.", "summary_of_the_review": "I think the paper is an incremental effort over the state of the art methods. The experimental section is rather limited for an experimental paper. It needs a lot more benchmarks and results on multiple datasets and baselines. ", "strength_and_weaknesses": "I do not think the idea of splitting a network into two parts is a new one, especially in the context of privacy. The idea of adding additional constraints in the encoding process to control mutual information flow is an interesting take. Unfortunately, I am not convinced about its effectiveness.\n\nFor a primarily emprical paper, the work considers rather simplistic datasets and only two of them. In the second dataset, they do not compare it with any baselines and in the first dataset it is compared with 1 baseline. There are lot of other experiments that can be performed and baselines that can be and should be compared with. For example, why not compare with fully homomorphic encryption based approaches?\n\nOverall I think the paper is rather incremental with no “aha” factor that is needed in a conference such as ICLR.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "8W-8WpEn22-", "reviewer": "Reviewer_QZoH", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper presents a new approach called \"Scrunch\" to perform privacy-preserving training in MLaaS settings. The key idea is to split the model architecture into two parts (one for client called Encoder and another for the server called Classifier) and optimize both parts to maintain predictive performance while explicitly minimizing information leakage via Center Loss. The paper presents results on two models (LeNet-5 and VGG-16) and two datasets (MNIST and CelebA).", "summary_of_the_review": "The paper is well-written and easy to follow, but has serious technical flaws and the results are unconvincing.", "strength_and_weaknesses": "Strengths:\n1) The paper is clear and mostly easy to understand\n2) The problem statement is well-defined\n3) Good use of figures and tables\n4) Interesting evaluation metrics\n\nWeaknesses:\n\n1) The proposed approach is severely flawed due to the following reasons. \n\na) Firstly, the title and abstract suggest that the goal is to perform privacy-preserving \"inference\" in the MLaaS scenario. However, the proposed approach requires finetuning the encoder component, which can done only during the \"training\" phase. \n\nb) Secondly, to perform this training (to compute the cross-entropy loss), the server needs access to the private labels of the client along with the \"smashed\" representations. Next, to compute the center loss, the server also needs access to the class centers. Finally, the encoder is initially shared by the server to the client and the server also knows the loss at the split point for each iteration. Using this information, the server can easily recreate a \"surrogate\" encoder, which will be exactly identical to the client encoder. Thus, the server has all the information including the entire neural network, class labels, and class centers in the feature space. The only unknown is the input samples, which should be easy to reconstruct using well-known \"model inversion\" attacks.\n\nc) Finally, and in fact fatally, the threat model for the proposed approach is completely messed up. If the assumption is that the server is \"curious\" to know about the raw data, why would it implement the center loss honestly? The server can very well turn off the center loss to zero and back propagate only the cross-entropy loss and the client has no way of verifying it. Moreover, the encoder becomes privacy-preserving only for the inference phase. Thus, there will no defense against privacy leakage during the training phase.\n\n2) Following are some of the other issues with the paper:\n\na) There are several ambiguous statements, e.g. “In other words, it tries to maximize the inter-class distance. In addition to this, we would also like to minimize the inter-class distance, to reduce any extra information…” should probably have said intra-class distance in the latter part instead. These errors affect the clarity of the paper and may negatively affect appreciation of its contributions. \n\nb) What is the private task for the MNIST dataset? It is well-stated for CelebA, together with supporting visualizations, but this information is not provided for MNIST although results for the private task accuracy drop on MNIST are reported in Table 1.\n\nc) The experimental evaluation seems somewhat simplistic: the paper claims that the proposed method is effective at preventing privacy invasion via expression recognition (smiling vs not smiling) from a public task of gender recognition (which is also binary and arguably naturally privacy-preserving). However, proof in a more challenging setting (from both public and private task perspectives) e.g., identity inference (public) vs gender (private) would be more convincing, as such challenging settings are likely to be of more real-world interest to attackers. \n\nd) The two models evaluated in this work are relatively shallow and architecturally-similar (basic CNNs). It would be more beneficial to consider more sophisticated architectures such as ResNets, DenseNets or Vision Transformer, which are significantly deeper and more commonly-used architectures today. Some experimentation extending the proposed method to actual large-scale networks such as those mentioned above would provide compelling evidence about the benefits of the method.\n\ne) The paper notes that the choice of lambda is particularly important. More investigation on this point would be a good addition.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject