MixCon: Adjusting the Separability of Data Representations for Harder Data Recovery
Authors: Xiaoxiao Li
Yangsibo Huang
Binghui Peng
Zhao Song
Kai Li
Venue:  (2021)
Keywords: Data Recovery
Data Separability
Distributed Deep Learning

Abstract: To address the issue that deep neural networks (DNNs) are vulnerable to model inversion attacks, we design an objective function to adjust the separability of the hidden data representations as a way to control the trade-off between data utility and vulnerability to inversion attacks. Our method is motivated by the theoretical insights of data separability in neural networking training and results on the hardness of model inversion. Empirically, we show that there exist sweet-spots by adjusting the separability of data representation, such that it is difficult to recover data during inference while maintaining data utility.  

Under review as a conference paper at ICLR 2021
MIXCON: ADJUSTING THE SEPARABILITY OF DATA REPRESENTATIONS FOR HARDER DATA RECOVERY
Anonymous authors
Paper under double-blind review
ABSTRACT
To address the issue that deep neural networks (DNNs) are vulnerable to model inversion attacks, we design an objective function, which adjusts the separability of the hidden data representations, as a way to control the trade-off between data utility and vulnerability to inversion attacks. Our method is motivated by the theoretical insights of data separability in neural networking training and results on the hardness of model inversion. Empirically, by adjusting the separability of data representation, we show that there exist sweet-spots for data separability such that it is difficult to recover data during inference while maintaining data utility.
INTRODUCTION
Over the past decade, deep neural networks have shown superior performances in various domains, such visual recognition, natural language processing, robotics, and healthcare. However, recent studies have demonstrated that machine learning models are vulnerable in terms of leaking private data He et al. (2019); Zhu et al. (2019); Zhang et al. (2020b). Hence, preventing private data from being recovered by malicious attackers has become an important research direction in deep learning research.
Distributed machine learning Shokri & Shmatikov (2015); Kairouz et al. (2019) has emerged as an attractive setting to mitigate privacy leakage without requiring clients to share raw data. In the case of an edge-cloud distributed learning scenario, most layers are commonly offloaded to the cloud, while the edge device computes only a small number of convolutional layers for feature extraction, due to power and resource constraints Kang et al. (2017). For example, service provider trains and splits a neural network at "cut layer," then deploys the rest of the layers clients Vepakomma et al. (2018). Clients encode their dataset using those layers, then send the data representations back to cloud server using the rest of layers for inference Teerapittayanon et al. (2017); Ko et al. (2018), 8);; Vepakomma et al. (2018). This gives an untrusted cloud provider to steal sensitive inference data from the output of "cut layer" on the edge device side, i.c. inverting data from their outputs Fredrikson et al. (2015); Zhang et al. (2020b); He et al. (2019).
malicious participant a chance
In the above distributed learning setup, we investigate how to design a hard-to-invert data represen- tation function (or hidden data representation function), which is defined the output of the neural network's 's intermediate layer. We focus on defending data recovery during inference. The goal is to hide sensitive information and protect data representations from being used to reconstruct the original data while ensuring that the resulted data representations are still informative enough for de- cision making. We use the model inversion attack that reconstructs individual data He et (2019); Zhang et al. (2020b) to evaluate defense performance and model accuracy to evaluate data utility. The core question here is how achieve the goal, especially protecting individual data from being recovered.
We propose data separability, also known as the minimum (relative) distance between (the represen- tation of) two data points, as a new criterion to investigate and understand the trade-off between data utility and hardness of data recovery. Recent theoretical studies show that data points are separable in the hidden embedding space of DNN model, it is helpful for the model to achieve good clas- sification accuracy Allen-Zhu et al. (2019b). However, larger separability is also easier to recover inputs. Conversely, if the embeddings are non-separable or sometimes overlap with one another, it is challenging to recover inputs. Nevertheless, the model may not be able to leam to achieve good performance. Two main questions arise. First, is there an effective way to adjust the separability of data representations? Second, are there "sweet spots" that make the data representations difficult for inversion attacks while achieving good accuracy.

---

Problem formulation. Formally, let Rd + RT denote the local feature extractor function, which maps an input data x E Rd to its feature representation h(x) E RT. The local feature extractor is shallow neural network in our setting. The deep neural network on the server side is Under review as a conference paper at ICLR 2021
This paper aims to answer these two questions by learning a feature extractor that adjust the separability of data representations embedded by a few neural network layers. Specifically, we pro- pose to add self-supervised learning-base -bsed novel regularization term to the standard loss function during training. We conduct experiments on both synthetic and benchmark datasets to demonstrate that with specific parameters, such learned neural network is indeed difficult to recover input data while maintaining data utility.
Our contributions can be summarized as:
To the best of our knowledge, this is the first proposal to investigate the trade-off between data utility and data recoverability from the angle of data representation separability;
We propose a simple yet effective loss term, Consistency Loss MixCon for adjusting data separability;
We provide the theoretical-guided insights of our method, including a new exponential lower bound on approximately solving the network inversion problem, based on the Expo- nential Time Hypothesis (ETH); and
We report experimental results comparing accuracy and data inversion results with/without incorporating MixCon. We show MixCon with suitable parameters makes data recovery difficult while preserving high data utility.
The rest of the paper is organized follow. We formalize our problem in Section 2. In Section present our theoretical insight and introduce the consistency loss. We demonstrate the experiment results in Section We defer the technical proof and experiment details Appendix.
PRELIMINARY
Distributed learning framework. We consider distributed learning framework, in which users and servers collaboratively perform inferences Teerapittayanon al. (2017); Ko al. (2018); Kang et al. (2017). We have the following assumptions: 1) Datasets are stored at the user sides. During inference, no raw data are ever shared among users and servers; 2) Users and servers use a split model Vepakomma et al. (2018) where users encode their data using proposed mechanism to tract data representations at a cut layer of a trained DNN. Servers take encoded data representations inputs and compute outputs using the layers after the cut layer in the distributed learning setting; 3) DNN used in the distributed learning setting can be regularized by our loss function (defined later).
Threat model. We consider the attack model with access to the shared hidden data representations during the client-cloud communication process. The attacker aims to recover data (i.e., pixel- wise recovery for images in vision task). To quantify the upper bound of privacy leakage under this threat model, we allow the attacker to have more power in our evaluation. In addition having access to extracted features, we allow the attacker to see all network parameters of the trained model.
denoted as g R + R which performs classification tasks and maps the feature representation to of C' target classes. The overall neural network Rd R and it can be written as f = o h.
Our overall objectives are:
Learn the feature representation mechanism (i.e. h function) that safeguards information from unsolicited disclosure.
Jointly learn the classification function g. and the feature extraction function to ensure the information extracted is useful for high-performance downstream tasks.
3 CONSISTENCY LOSS FOR ADJUSTING DATA SEPARABILITY
To address the issue of data recovery from hidden layer output, we propose a novel consistency loss in neural network training, shown in Figure 1. Consistency loss is applied the feature extractor
2

---

Under review as a conference paper at ICLR 2021
h to encourage encoding closed but separable representations for the data of different classes. Thus, the feature extractor can help protect original data from being inverted by an attacker during inference while achieving desirable accuracy.
Frbbowe
dist
polx
Encoder
lovel
# Aul?
& buf2
High-level
teatures
Space
Fmbeedding Space
Output Space
Figure I: Schematic diagram of our data representation encoding scheme in deep learning pipeline. We show simple toy example of classifying data points of triangles, squares, and circles. In embedding space (the middle block), data representations from different classes constrained to small ball with diameter Au, while they are separate from each other at least with distance 5n.
3.1 DATA SEPARATION AS A GUIDING TOOL
Our intuition is to adjust the information in the data representations to a minimum such that down- stream classification tasks can achieve good accuracy but not enough for data recovery through model inversion attacks He et al. (2019). The question is, what is the right measure on the amount of information for successful classification and data security? We propose use data separability as the measure. This intuition is motivated by the theoretical results of deep learning, In particular,
Over-parametent deep learning theory the well separated data requires a narrower network train,
In-approximation theory - the worse separability of the data, the harder of the inversion problem.
Definition 3.1 (Data separability). Let on denote the separability of hidden layer over all pairwise inputs ø1, "2 Rd i.e.
8n = min [h(x,) 122))l22- (controlling accuracy)
Let denote a set of pairs that supposed to be close in hidden layer. Let Aht denote the maximum distance with respect that set S
Ar
max [1(x;)) h(2,)1l2. (controlling invertibility) (iJ)s
Intuitively, expect the upper bound on data separability (Ax) relates to the invertibility and the lower bound on data separability (8₁) relates to the accuracy.
Lower bound on data separability implies better accuracy Recent line of deep learning theory Allen-Zhu et al. (2019b) indicates that data separability is perhaps the only matter fact for learnabil- ity (at least overparameterizeed neural network), leading into the following results.
Theorem 3.2. Allen Alen-Zhu al. (2019b) Suppose the training data points are separable, i.e., 8%. > O. If the width of L-layer neural network with ReLU gates satisfies m 2 poly(n, d. L, 1/5x), initializing from a random weight matrix W. (stochastic) gradient descent algorithm can find the global minimum of neural network function
Essentially, the above theorem indicates that we can (provably) find global minimum of neural network given well separated data, and better separable data points requires narrower neural network and less running time.
Upper bound on data separability implies hardness of inversion. When all data representation close each other, i.e. Air is sufficiently small, we expect the inversion problem is hard. We
3

---

Under review as a conference paper at ICLR 2021
support this intuition by proving that the neural network inversion problem is hard to approximate within some constant factor when assuming NP*RP.
Existing work Lei et al. (2019) indicates that the decision version of the neural network inversion problem is NP-hard. However, this insufficient since it is usually easy to find an approximate solution, which could leak much information on the original data. It is open question whether the approximation version is also challenging. We strengthen the hardness result and show that by assuming /P,RP, it hard to recovery input that approximates the hidden layer representation. Our hardness result implies that given hidden representations are close to each other, no polyno- mial time can distinguish their input. Therefore, impossible to recover the real input data in polynomial time.
Theorem 3.3 (Informal). Assume /R, there is no polynomial time algorithm that is able to give constant approximation thee neural network inversion problem.
The above result only rules out the polynomial running time recovery algorithm but leaves out the possibility a subexponential time algorithm. To further strengthen the result, we assume the well-known Exponential Time Hypothesis (ETH), which widely accepted in the computation complexity community.
Hypothesis 3.4 (Exponential Time Hypothesis (ETH) Impagliazzo et al. (1998)). There a ò such that the 3SAT problem cannot be solved in 0 time.
Assuming ETH, we derive an exponential lower bound on approximately recovering the input.
Corollary 3.5 (Informal), Assume ETH, there is no 20(n2 time algorithm that is able to give a constant approximation neural network inversion problem.
3.2 CONSISTENCY LOSS - MixCon
Follow the above intuitions, we propose a novel loss term MixCon loss - Lmixcon - to incorporate in training. MixCon adjusts data separability by forcing the consistency of hidden data represen- tations from different classes. This additional loss term balances the data separability, punishing feature representations that are too far or too close to each other. Noting that we choose to mix data from different classes instead of the data within class, in order to bring more confusion in embedding space and potentially hiding data label information".
MixCon loss : We add consistency penalties force the data representation of i-th data in different classes to be similar, while without any overlapping for any two data points.
Emixcon == (1-(
ΣΣ> i=1 DEE CgEC
Σ
(iitt(i,c1,.)) / c1, 02)).
(I)
practical choice for the pairwise distance is dist 02) ||b(x),,))) M( ( )1 where v is the i-th input data point in class := minzec Ic]. and that the order is not fixed for data point due to random shuffling in the regular training process. Thus Eq. can nearly perform as all-pair comparisons in training with data shuffling. The first term punishes large distance while the second term enforces sufficient data separability. In general, we could replace (dist + 3/dist) by convex functions with asymptote shape non-negative domain, that is, function with value reaches infinity on both ends of [0, oc).
balances the data separability. Note
We consider the classification loss
N
Lelass =
Mi.e log(ūi.) (cross entropy)
'The class consists all languages T. that have a polynomial-time randomized algorithm with following hchavior: If æ & L., then A always rejects x (with probability 1). x E L,, then A accepts x in I.
with probability at least 1/2.
We show the comparison Appendix D.I.
In practice, normalize []h(=)] To avoid division by zero, we can positive small ( 1)
and threshold distance the range of (e, 1/e].
4

---

Under review as a conference paper at ICLR 2021
where ys R is the one-hot representation of true label and 0 f(zz) R is the prediction score of data i E [J]....N.. The final objective function L == Celass + Lmixcon- We simulta- neously train h and 9. where À and / are tunable syper-parametenn associated with consistency loss regularization to adjust separability. We discuss the effect of À and /3 in experiments (Section 4).
4 EXPERIMENTAL RESULTS
Our hypothesis this work is MixCon loss can adjust the separability data hidden representa- tions. moderate reduction of representation separability can still keep data utility while making harder to recover from its representation. To show this hypothesis hold, we conduct two experiments. Synthetic experiments provide a straightforward illustration for the relationship among data separa- bility, data utility and reversibility. Benchmark experiments are used for image data classification and recovery evaluation.
4.1 DATA RECOVERY MODEL
To empirically evaluate the quality of inversion, we formally define the white-box data recovery (inversion) model He et al. (2019) used in our experiments. The model aims to solve an optimization problem in the input space. Given a representation 2 h(x) of testing data point, and public function h (the trained network that generates data representations), the inversion model tries to find the original input z::
xx arg min ((h(s), 2) 2) + R(s)
where is the loss function that measures the similarity between h(s)) and 2. and R the regu- larization term. We specify L and R used in each experiment later. We solve Eq. (3) by iterative gradient descent.
4.2 EXPERIMENTS WITH SYNTHETIC DATA
To allow precise manipulation and straightforward visualization for data separability, our experi- ments use generated synthetic data with a 4-layer fully-connected network, such that we can control the dimensionality. In this section, want answer the following questions:
Q1 What is the impact of having 3 in Eq.(1) to bound the smallest data pairwise distance? Q2 Is feature encoded with MixCon mechanism harder to invert?
Network, data generation and training. We defined the network as
y = g(softmax(f(x))), f(x) Wa (W3 oW2 (a (W + b1)) + b2) + bs) + ds
E R.O W. E 500x000 W2 E 22x500 W3 E 100x2 Wa E 222100 bj E R 500 by E
RR, by E 10000 by E R.. For a vector =, we use g(=) to denote the index such that |zil > |
Vj + i. We initialize each entry of W₁ and by from ((u,, 1), where uy ~NO,, a) and k E 41,
We generate synthetic samples v) from two multivariate normal distribution. Positive data are sampled from (0, and negative data are sampled from (N-11,1) Z), ending up with 800 training samples and 200 testing samples, where the covariance matrix / is an identity diagonal matrix. Lmixcon is applied to the 2nd fully-connected layer.
We train the network for 20 epochs with cross-entropy loss and SGD optimizer with 0.1 learning rate. We apply noise to the labels by randomly flipping 5% labels to increase training difficulty.
Testing setup.
We compare the results under the following settings:
Vanilla: training using only Leluss-
MixCon: training with MixCon loss with parameters (A, .9)1.
*A is the coefficient of penalty and | balancing term for data separahility.
5

---

Under review as a conference paper at ICLR 2021
cless
Nowe
clmm
dec
did
dec
den
(a) Vanilla E = 0
(b) Vanilla E = 20
(c) MixCon
(d) MixCon
(c) MixCon
(f) MixCon
0.01 E = 0 = 0.01 E = 20
B = =
B = 0 E = 20
Figure 2:
Data hidden representation h(z) E R² from the 2nd fulle-connected layer of synthetic data at different epoch (E). Two settings of MixCon are given default À = 0.1 but have different g. Compare to Vanilla, MixCon squeezes data representations smaller space over training. When 8 0. MixCon map all data to h(z)= 0): which not learnable.
MixCon B = 0.01
MixCon 3 =0 0
Vanilla
default
deeper net
wider net
default
deeper net wider net
Train Accuracy (%)
91.5
88.9
89.5
91.5
50.0
50.0
50.0
Test Accuracy (%)
91.5
88.0
88.5
90.5
50.0
50.0
50.0
Data utility (accuracy). Vanilla is equivalent to (A = 0, 8 = 0). Two MixCon "default" settings both use À = 0.1 but vary in 3 = 0.01 and B = 0. "Deeper"'"Wider" indicate increasing the depth / width of layers in the network server side 9(x).
Table 1:
We perform model inversion using Eq. (3) without any regularization term R(x) and L is the losss function. Detailed optimization process is listed in Appendix C.I.
Results. To answer QI that how 8 in (I,(1) affect the smallest data pairwise distance, we visualize the change of data representations initial and ending epochs in Figure First, in vanilla training (Figure a- a-b), data are dispersively distributed and enlarge their distance after training. The obvious difference for MixCon training (Figure c-f) is that data representations become more and more gathering through training. Second, we direct the data utility results of Vanilla and two "default" MixCon settings (A 0.1, B 0.01) and (X 0.1, B 0) to Table 1. When = O, MixCon achieves chance accuracy only it encodes all the hax) to hidden space (0,0) (Figure f). While having 3 >0 0 balancing the separability, MixCon achieves similar accuracy as Vanilla.
Based on Theorem 3.2, we further present two strategies to ensure reasonable accuracy while com- prise of reducing data separability by increasing the depth or the width of the layers 9(2), the network after the layer that is applied Lmixcon In practice, neurons after the 3nd layer for "deeper" g(x), and change the number neurons on the 3nd layer to 2048 for "wider" g(x)). We show the utility results in Table 1. Using deeper or wider g(2), MixCon 0.1, 3 = 0.01) improves accuracy. Whereas MixCon (A = 0.1, 3 = 0) fails, because zero data separability not learnable no matter how g(z) changes. This gives conformable answer that is an important factor to guarantee neural network to be trainable.
add two more fully- collecconnected layers with 100
To answer Q2, we evaluate the quality of data recovery using the inversion model. We use both square error (SE) and cosine similarity (CS) of x and x² to evaluate the data recov- ery accuracy. We show the quantitative inver- sion results in Table 2 with the mean and worst case values. Higher SE or lower CS indicates worse inversion. Apparently, data representa- tion from MixCon trained network is more dif- ficult to recover compared to Vanilla strategy.
MixCon
MixCon (0.1, 0)
Vanilla
(0.1, 0.01)
SE
1.92 (0.31)
2.08 (0.37)
2.35 (0.44) 0.161 (0.921)
CS
0.169 (0.960)
0.118 (0.939)
Table 2: Inversion results on synthetic dataset re- ported in mean(worst) format for the 200 testing sam- ples. Higher MSE or lower MCS indicates a worse in- version. (2, denoted in header.
4.3 EXPERIMENTS WITH BENCHMARK DATASETS
In this section, we would like answer the following questions:

---

Under review as a conference paper at ICLR 2021
ACCUROCY
Acunscy
AcCUFII
Decanec
Oetance
(a) MNIST
(b) Fashion.MNIS
(c) SVHN
Figure 3: Trade-off between data separability and data utility. We show testing accuracy and mean pairwise distance (data separability) on three datasets with different À and B. X and & show complementary effort on adjusting data separability. A sweet-spot can be found the (A, (3 resulting in small data separability and high data utility.
Original Image
Vanilla ACC- 99.156
MixCon (1-le-3)
MixCon (1 (l.e--)
MixCon (e-)) MixCon le-e-) MixConn (10, le-2) MisCon (100,1e-2)
ACC-9900R
ACC- 94F3
ACC- 978
ACC=0104R
ACC=08 3334
ACC- D45
ANNST
ACC 8012
ACC=ER,05
ACCw67 5055
ACCw67,577 7.57%
ACDw00 7652
ACC=57 573
ACC=5I 965
Vashion MINIST
Ledd
ACC-88.78%
ACC-87518
ACC-35
ACC-88 08%
ACC-88
ACC-85
ACC-64.51
SVIIN
(C1)
(C2)
(C3)
[C4)
(C5)
[O6]
(CF)
(CB)
Figure 4: Qualitative evaluation for image inversion results. (A, B) settings of MixCon denoted on the header. The corresponding lesting accuracy of each dataset is denoted the top of each row. Compared vanilla training, inversions from the MixCon model less realistic and distinguishable from the original images without significant accuracy dropping.
How does MixCon loss affect data separability and accuracy on image datasets?
Q4 Are there parameters (A, 3) in MixCon (Eq, (1)) to reach a "sweet-spot" for data utility and the quality of defending data recovery?
Network, datasets and training setup. The neural network architecture used in the experiments LeNet5 LeCun (2015)3. Lmixcon applied to the outputs of the 2nd convolutional layer blocks of LeNet5. The experiments use three datasets: MNIST LeCun et al. (1998), Fashion-MNIST Xiao al. (2017), and SVHN Netzer al. (2011).
Neural network is optimized using cross-entropy loss and SGD optimizer with learning rate 0.01 for 20 epochs. We do not use any data augmentation manual learning rate decay. MixCon loss is applied to the output of 2nd convolutional layer blocks in LeNet5. We use mini-batch training and each batch contains 40 data points from each class. We train the model with different pairs of 3)
$ We change input channel for SVHN dataset.
'In the regular mini-batch training using data shuffling, when calculating the MixCon loss, we truncate the size of batch to plC and each mini-batch contains equal number of samples of each class, Here p is the number of training points of the smallest class and [C] the numher of classes.

---

Under review as a conference paper at ICLR 2021
MNIST
FashaionMNIST
SVHN
Vanilla
MixCon
Vanilla
MixCon
Vanilla
MixCon
(A 1.0.5 10
1.0.8- 10-
x 0.5. B 10
Acc (%)
90.1
98.6
81.8
88.9
68.4
88.2
SSIM
0.64 + (3.1130333
0.14 + 0.1104488)
0.43 0.17(0.78)
0.17 + 0.09(0.52)
0.76 + 19:0.92)
0.61 0.15(0.84)
PSIM
0.78 1: 0.05(0.88)
0.44 0.07(0.69)
0.71 1 0.13(0.92)
0.42 1: (.08(0.66)
0.69 0.07(0.81)
0.59 1: 0.07(0.72)
Table 3: Quantitative evaluations for image recovery results. For fair evaluation, we match the data utility (accuracy) for Vanilla and Structual similarity index metric (SSIM) and perceptual similarity (PSIM) are measured on 100 testing samples. Those scores are presented in mean + std and worst case (in parentheses) format. Lower scores indicate harder to invert.
in Eq. (1) for the following testing. Specifically, we vary À from: (0.01,0.1,0.) 1,,,5, 10, 100) and | from: {10 2, 10000- 1004 10.110050000 10 10 10 $].
Testing setup. We record the testing accuracy and pairwise distance of data representation under each pair of (A, B) for each dataset. Following a recent model inversion method He et al. (2019), we define L in Eq. (3) as f2-loss function, R as the regularization term capturing the total variation of a 2D signal defined as R(aa)
((r+1, az,))² 2 + (aij+1 a.. )2)_) The inversion attack is applied to the output of 2nd convolutional layer blocks in LeNet5 and find the optimal of Eq. (3) though SGD optimizer. Detailed optimization process is listed in Appendix C.2.
We use metrics normalized structural similarity index metric (SSIM) Wang et al. (2004) and percep- tual similarity (PSIM) Johnson et al. (2016) to measure the similarity between the recovered image and the original image. The concrete definitions of SSIM and PSIM are listed in Appendix C.3.
Results To answer Q3, we plot the complementary effects of À and B in Figure 3. Note that B bounds the minimal pairwise of data representations, and À indicate the penalty power on data sep- arability given by MixCon. Namely, a larger À brings stronger penalty of MixCon, which enhances the regularization of data separability and results in lower accuracy. Meanwhile, with a small B, is not necessary to be very large, smaller B leads to smaller bound of data separability, thus resulting in lower accuracy. Hence, À and ,3 work together to adjust the separability of hidden data representations, which can affect on data utility.
To answer Q4, we evaluate the quality of inversion qualitatively and quantitatively through a model inversion attack defined in "Test setup" paragraph. Specifically, for each private input xr we execute the inversion attack on hmmixco (z) and (z) of testing images. As it is qualitatively shown in Figure 4, first, the recovered images using model inversion from MixCon training (such as given (A, 3) E {(1, 1 x 10-"), (10, 1 x 10-2), (100, x 10-2)}) are visually different from the original inputs, while the recovered images from Vanilla training still look similar to the originals. Second, with the same X (Figure 4 column c3-c5), the smaller the 3 it is, the less similar of the recovered images to original images. Last, with the same /3 (Figure 4 column c3 and c6-c8), the larger the À is, the less similar of the recovered images to original images.
Further, we quantitatively measure the inversion performance by reporting the averaged similarity between 100 pairs of recovered images by the inversion model and their original samples. We se- lect (X, B) to match the accuracy 7 of MixCon with Vanilla training (see Accuracy in Table 3), and investigate if MixCon makes the inversion attack harder. The inverted results (see SSIM and PSIM in Table 3) are reported in the format of mean + std and the worst case (the best-recovered data) similarity in parentheses for each metric. Both qualitative and quantitative results agree with hypothesis that 1) adding Liixxon in network training can reduce the mean pairwise distance (sep- arability) of data hidden representations; and 2) smaller separability make it more difficult to invert original inputs. By sweet spot, we can define as the set of (3, A) that suffers with negligible accu- racy loss (say within 1%) and the model inversion becomes significantly harder w.r.t computational complexity or breaks the attack (less similarity to the original input data per se). Thus by visiting through possible (A, 3). we are able to find a spot, where data utility is reasonable but harder for data recovery, such as (A = 100, 3 = 1e 2) for MNIST (Figure 4). Thus our proposed method is helpful if the user is willing to give up on some accuracy the hope of getting a more robust model.
AAccurayy reduction is within small tolerance, i.e., 1%.
8

---

Under review as a conference paper at ICLR 2021
5 DISCUSSION AND CONCLUSION
In this paper, we have proposed and studied the trade-off between data utiliity and data recovery from the angle of the separabilityy of hidden data representations in deep neural network. We propose using MixCon, a consistency loss term, as an effective way to adjust the data separability. Our proposal is inspired by theoretical data separability results and new exponential lower bound approximately solving the network inversion problem, based on the Exponential Time Hypothesis (ETH).
We conduct two sets of experiments, using synthetic and benchmark datasets, to show the effect of adjusting data separability on accuracy and data recovery. Our theoretical insights help explain our key experimental findings: MixCon can effectively adjust the separability hidden data representa- tions, and one can find "sweet-spot" parameters for MixCon to make it difficult to recover data while maintaining data utility. Our experiments are limited to small benchmark datasets in the domain of image classifications. It will be helpful to conduct experiments using large datasets in multiple do- mains to further the study of the potential of adjusting data separability of data representations to trade-off between data utility and data recovery.
REFERENCES
Martin Abadi, Andy Chu, Ian Goodfellow, H Brendan McMahan, Ilya Mironov, Kunal Talwar, and Li Zhang. Deep learning with differential privacy. In Proceedings of the 2016 ACM SIGSAC Conference on Computer and Communications Security, pp. 308 -18,,,181201 2016.
Zeyuan Allen-Zhu, Yuanzhi Li, and Yingyu Liang. Learning and generalization in overparameter- ized neural networks, going beyond two layers. In Advances in neural information processing 2019a.
systems (NeurIPS), pp.
Zeyuan Allen-Zhu, Yuanzhi Li, and Zhao Song. convergence theory for deep learning via over- parameterization. In /CML, 2019b.
Zeyuan Allen-Zhu, Yuanzhi Li, and Zhao Song. On the convergence rate of training recurrent neural networks. In NeurlPS, 2019c.
Sanjeev Arora and Boaz Barak. Computational complexity: a modern approach. Cambridge Uni- versity Press, 2009.
Sanjeev Arora, Rong Ge, Ravi Kannan, and Ankur Moitra. Computing a nonnegative matrix factorizatioon-prvvabll. In STOC, 2012.
Sanjeev Arora, Simon Du, Wei Hu, Zhiyuan Li, and Ruosong Wang. Fine-grained analysis of op- timization and generalization for overparameterized two-layer neural networks. In International Conference on Machine Learning (ICML), pp. 322-332, 2019a.
Sanjeev Arora, Simon S Du, Wei Hu, Zhiyuan Li, Russ Salakhutdinov, and Ruosong Wang. On exact computation with an infinitely wide neural net. In Advances in Neural Information Processing Systems (NeurIPS), pp. 8141-8150, 2019b.
Ainesh Bakshi, Rajesh Jayaram, and David P Woodruff. Learning two layer rectified neural networks in polynomial time. In Conference on Learning Theory (COLTT), pp. 195-268. PMLR, 2019.
Frank Ban, Vijay Bhattiprolu, Karl Bringmann, Pavel Kolev, Euiwoong Lee, and David P Woodruff. A ptas for lowlow rank approximation. In Proceedings of the Thirtieth Annual ACM-SIAM Sym- posium Discrete Algorithms (SODA), pp. 747-766. SIAM, 2019.
Arnab Bhattacharyya, Suprovat Ghoshal, Karthik C. S., and Pasin Manurangsi. Parameterized in- tractability of even set and shortest vector problem from gap-eth. In ICALP, pp. 17:1-17:15, 2018.
Avrim L Blum and Ronald L Rivest. Training 3-node neural network is np-complete. Neural Networks, 511):117-127, 1992.
Jan van den Brand, Binghui Peng, Zhao Song, and Omri Weinstein. Training (overparametrizeed) neural networks in near-linear time. arXiv preprint arXiv:2006. //648, 2020.

---

Under review as a conference paper at ICLR 2021
Parinya Chalermsook, Marek Cygan, Guy Kortsarz, Bundit Laekhanukit, Pasin Manurangsi, Danupon Nanongkai, and Luca Trevisan. From gap-eth to in-iippppoximmbiit::: Clique, domi- nating set, and more. In 2017 IEEE 58th Annual Symposium on Foundations of Computer Science (FOCS), pp. 743-754. IEEE, 2017.
Sitan Chen, Adam R. Klivans, and Raghu Meka. Learning deep relu networks is fixed-parameter tractable. arXiv preprini orXiv:2009.135/1 2020.
Rajesh Chitnis, Andreas Emil Feldmann, and Pasin Manurangsi. Parameterized approximation al- gorithms for bidirected steiner network problems. In ESA, pp. 20: 20:1-20:16, 2018.
Amit Daniely. Complexity theoretic limitations on learning halfippaces. In Proceedings of the forty- eighth annual ACM symposium on Theory of Computing (STOC), pp. 105-117, 2016.
Amit Daniely and Shai Shalev-Shwartz. Complexity theoretic limitations on learning dnf's. In Conference on Learning Theory (COLT), pp. 815-830, 2016.
Amit Daniely and Gal Vardi. Hardness of learning neural networks with natural weights. arXiv preprint XXi:22006003177 2020.
Irit Dinur. Mildly exponential reduction from gap 3sat to polynomial-ges label-cover. In Electronic Colloquium on Computational Complexity (ECCC), volume 23, 2016.
Irit Dinur. Personal communication. 2017.
Irit Dinur and Pasin Manurangsi. Eth-hardness of approximating 2-csps and directed steiner net- work. ITCS, 2018.
Simon Du, Xiyu Zhai, Barnabas Poczos, and Aarti Singh. Gradient descent provably optimizes over-parameterized neural networks. In ICLR, 2019.
Cynthia Dwork. Differential privacy: A survey of results. In International conference theory and applications of models of computation, pp. 1-19. Springer, 2008
Amir Erfan Eshratifar, Mohammad Saeed Abrishami, and Massoud Pedram. Jointdnn: an efficient training and inference engine for intelligent mobile cloud computing services. IEEE Transactions on Mobile Computing, 2019.
Uriel Feige. Relations between average case complexity and approximation complexity. In Proceed- ings of the thiry-fourth annual ACM symposium on Theory of computing (STOC), pp. 534-543. ACM, 2002.
Matt Fredrikson, Somesh Jha, and Thomas Ristenpart. Model inversion attacks that exploit confi- dence information and basic countermeasures. Proceedings of the 22nd ACM SIGSAC Confer- ence on Computer and Communications Security (CCS), pp. 1322- 1333, 2015.
Rong Ge, Jason D. Lee, and Tengyu Ma. Learning one-hidden-laye neural networks with landscape design. In ICLR, 2018.
Surbhi Goel, Varum Kanade, Adam Klivans, and Justin Thaler. Reliably learning the relu in polyno- mial time. In Conference Learning Theory (COLTT), pp. 1004-1042. PMLR, 2017.
Johan Håstad. On bounded occurrence constraint satisfaction. Information Processing Letters, 74 (1-2):1-6, 2000.
Johan Hästad. Some optimal inapproximability results. Journal tfe 2001.
the ACM (JACM), 48(4):798-853
Johann Hauswald, Thomas Manville, Qi Zheng, Ronald Dreslinski. Chaitali Chakrabarti, and Trevor Mudge. A hybrid approach to offloading mobile image classification. 2014 IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP), pp. 8375-8379. IEEE, 2014.
Zecheng He, Tianwei Zhang, and Ruby B Lee. Model inversion attacks against collaborative in- ference. In Proceedings of the 35th Annual Computer Security Applications Conference, pp- 148-162, 2019.
10

---

Under review as a conference paper at ICLR 2021
Yangsibo Huang. Zhao Song, Danqi Chen, Kai Li, and Sanjeev Arora. Texthide: Tackling data privacy in language understanding tasks. In The Conference on Empirical Methods in Natural Language Processing (Findings of EMNLP), 2020a.
Yangsibo Huang, Zhao Song, Kai Li, and Sanjeev Arora. Instahide: Instance-hiding schemes for private distributed learning. In Internation Conference on Machine Learning (ICML), 2020b.
Russell Impagliazzo, Ramamohan Paturi, and Francis Zane. Which problems have strongly ex- ponential complexity? In Proceedings. 39th Annual Symposium on Foundations of Computer Science (FOCS), pp. 653-662. IEEE, 1998.
Arthur Jacot, Franck Gabriel, and Clément Hongler. Neural tangent kemel: Convergence and gen- eralization in neural networks. In Advances in neural information processing systems (NeurIPS), pp. 8571-8580, 2018.
Justin Johnson, Alexandre Alahi, and Li Fei-Fei. Perceptual losses for real-time style transfer and super-resolution. In European Conference on Computer Vision (ECCV), pp. 694-711. Springer, 2016.
Peter Kairouz, H. Brendan McMahan, et al. Advances and open problems in federated learning, 2019.
Yiping Kang, Johann Hauswald, Cao Gao, Austin Rovinski, Trevor Mudge, Jason Mars, and Lingjia Tang. Neurosurgeon: Collaborative intelligence between the cloud and mobile edge. ACM SIGARCH Computer Architecture News, 45 45(1):615-622 2017.
B Laekhanukit KCS and P Manurangsi. On the parameterized complexity of approximating domi- nating set. In Proceedings of the 50th Annual ACM SIGACT Symposium on Theory of Computing (STOC), pp. 1283-1296, 2018.
Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. Supervised contrastive learning. arXiv preprint arXiv:2004. 11300 2020.
Adam Klivans and Alexander Sherstov. Cryptographic hardness for learning intersections of halfspaces. Journal of Computer and System Sciences, 75(1):2-12, 2009.
Jong Hwan Ko, Taesik Na, Mohammad Faisal Amir, and Saibal Mukhopadhyay. Edge-host parti- tioning of deep neural networks with feature space encoding for resource-constraine internet- of-things platforms. In 2018 15th IEEE International Conference Advanced Video and Signal Based Surveillance (AVSS), pp. 1-6. IEEE, 2018.
Jakub Konečný, Brendan McMahan, and Daniel Ramage. Federated optimization: Distributed opti- mization beyond the datacenter. arXiv preprint xrXiv:551..0355,, 2015.
Yann LeCun. Lenet-5, convolutional neural networks. http:///ann, lecun. con/exdb/ Jenet, 20( 2015.
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 96(11):2278-233 1998.
Jason D Lee, Ruoqi Shen, Zhao Song, Mengdi Wang, and Zheng Yu. Generalized leverage score sampling for neural networks. In NeurlPS, 2020.
Qi Lei, Ajil Jalal, Inderjit S Dhillon, and Alexandros G Dimakis. Inverting deep generative models, one layer at a time. In Advances in Neural Information Processing Systems (NeurIPS), pp. 13910- 13919, 2019.
Yuanzhi Li and Yingyu Liang. Learning overparameterized neural networks via stochastic gradient descent on structured data. In Advances in Neural Information Processing Systems (NeurIPS), pp. 8157-8166, 2018
Yuanzhi Li and Yang Yuan. Convergence analysis of two-layer neural networks with ReLU activa- tion. In Advances neural information processing systems (NIPS), pp. 597-607, 2017.
11

---

Under review as a conference paper at ICLR 2021
Roi Livni, Shai Chalev-Shwarde and Ohad Shamir. On the computational efficiency of training neural networks. In Advances in neural information processing systems (NeurIPS), pp. 855-863, 2014.
Alice Lucas, Santiago Lopez-Tapia, Rafael Molina, and Aggelos K Katsaggelos. Generative adver- sarial networks and perceptual losses for video super-resolution. IEEE Transactions on Image Processing, 28(7):3312-332) 2019.
Pasin Manurangsi. Almost-polynomin ratio eth-hardness of approximating densest k-subgraph. In STOC, pp. 954 961. ACM, 2017.
Pasin Manurangsi and Prasad Raghavendra. A birthday repetition theorem and complexity of ap- proximating dense csps. In ICALP, pp. 78:1-78:15. 2017.
Pasin Manurangsi and Daniel Reichman. The computational complexity of training relu (8). arXiv preprint arXiv: /8/iv:////... 0//0..0020... 2018.
Dana Moshkovitz and Ran Raz. Two-query pep with subconstant error. In Journal of the ACM (JACM), volume 57(5), pp. 29. A preliminary version appeared in the Proceedings of The 49th Annual IEEE Symposium on Foundations of Computer Science (FOCS 2008), 2010.
Karthik Nandakumar, Nalini Ratha, Sharath Pankanti, and Shai Halevi. Towards deep neural net- work training encrypted data. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition Workshops, pp. 0-0, 2019.
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Ng. Reading digits in natural images with unsupervised feature learning. 2011.
Samet Oymak and Mahdi Soltanolkotabi. Towards moderate overparameterivaaion:: global conver- gence guarantees for training shallow neural networks. In IEEE Journal on Selected Areas in Information Theory. IEEE, 2020.
Ilya Razenshteyn, Zhao Song, and David Woodruff. Weighted low rank approximations with provable guarantees. In Proceedings of the 48th Annual Symposium on the Theory of Computing (STOC), 2016.
Reza Shokri and Vitaly Shmatikov, ACM SIGSAC Conference Computer and Communications Security (CCS), pp. 1310-1321 2015.
deep learning. In Proceedings of the 22nd
Zhao Song and Xin Yang. Quadratic suffices for over-parametrization via matrix chernoff bound. arXiv preprint arXiv:1906.03355 2019.
Zhao Song. David Woodruff, and Peilin Zhong. Low rank approximation with entrywise 11-norm error. In Proceedings of the 49th Annual ACM SIGACT Symposium on Theory of Computing (STOC), pp. 688-701, 2017.
Zhao Song, David P Woodruff, and Peilin Zhong. Relative error tensor low rank approximation. In SODA, 2019.
Surat Teerapittayanon, Bradley McDanel, and Hsiang-Tsung Kung. Distributed deep neural net- works over the cloud, the edge and end devices. In 2017 IEEE 37th International Conference on Distributed Computing Systems (ICDCS), pp. 328-339. IEEE, 2017.
Luca Trevisan. Non-approximmbility results for optimization problems on bounded degree instances. In Proceedings of the thirty-third annual ACM symposium on Theory of computing (STOC), pp- 453-461, 2001.
Paul Vanhaesebrouck, Aurélien Bellet, and Mare Tommasi. Decentralized collaborative learning of personalized models over networks. arXiv preprint arXiv:1610.05202. 2016.
Praneeth Vepakomma, Otkrist Gupta, Tristan Swedish, and Ramesh Raskar. Split learning for health: Distributed deep learning without sharing raw patient data. arXiv preprint 10X2::8812.000 2018.
12

---

Under review as a conference paper at ICLR 2021
Chaoyue Wang. Chang Xu, Chaohui Wang, and Dacheng Tao. Perceptual adversarial networks for image-to- tomiaag transformation. IEEE Transactions on Image Processing, 778)):00664007) 2018.
Tianhao Wang, Yuheng Zhang, and Ruoxi Jia. Improving robustness to model inversion attacks via mutual information regularization. arXiv preprint wrXiv:2009.05241, 2020.
Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli. Image quality assessment: from error visibility structural similarity. IEEE transactions on image processing, 13(4):600- 612, 2004.
Tsui-Wei Wei Weng, Huan Zhang, Hongge Chen, Zhao Song, Cho-Jui Hsieh, Duane Boning, Inderjit Dhillon, and Luca Daniel. Towards fast computation of certified robustmess for relu networks. In ICML, 2018
Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mniss a novel image dataset for benchmark- ing machine learning algorithms. arXiv preprint arXiv: 2tt::1708.0770 778.077477 2017.
Ziqi Yang. Bin Shao, Bohan Xuan, Ee-Chien Chang, and Fan Zhang. Defending model inversion and membership inference attacks via prediction purification. arXiv preprint arXiv:2005.039/5 2020.
Yi Zhang, Orestis Plevrakis, Simon S Du, Xingguo Li, Zhao Song, and Sanjeev Arora. Over- parameterized adversarial training: An analysis overcoming the curse of dimensionality. In NeurlPS, 2020a.
Yuheng Zhang, Ruoxi Jia, Hengzhi Pei, Wenxiao Wang, Bo Li, and Dawn Song. The secret revealer: generative model-inversion attacks against deep neural networks. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition (CVPR), pp. 253-261, 2020b.
Kai Zhong, Zhao Song, and Inderjit S. Dhillon. Learning non-overlapping convolutional neural networks with multiple kernels. arXiv preprint arXiv:1711.03440, 2017a.
Kai Zhong, Zhao Song, Prateek Jain, Peter L. Bartlett, and Inderjit S. Dhillon. Recovery guarantees for one-hidden-layer neural networks. In ICML, 2017b.
Ligeng Zhu, Zhijian Liu, and Song Han. Deep leakage from gradients. In NewrIPS, 2019.
13

---

Under review as a conference paper at ICLR 2021
Roadmap of Appendix The Appendix is organized follows. We discuss related work in Sec- tion A. We provide theoretical analysis in Section B. The details of data recovery experiment are in Section C and additional experiment details are in Section D.
A RELATED WORK
A.1 HARDNESS AND NEURAL NETWORKS
When there are no further assumptions, neural networks have been shown hard in several different perspectives. Blum & Rivest (1992) first proved that learning the neural network is NP-complete. Different variant hardness results have been developed over past decades Klivans & Sherstov (2009); Daniely (2016); Daniely & Shalev-Shward (2016); Goel et al. (2017); Livni et al. (2014); Weng et al. (2018); Manurangsi & Reichman (2018); Lei et al. (2019); Daniely & Vardi (2020); Huang et al. (2020b;a). The work of Lei et al. (2019) is most relevant to They consider the neural network inversion problem in generative models and prove the exact inversion problem is NP-complete.
A.2 DATA SEPARABILITY AND NEURAL NETWORK TRAINING
One popular distributional assumption, in theory, is to assume the input data points to be the Gaus- sian distributions Zhong et al. (2017b); Li & Yuan (2017); Zhong al. (2017a); Ge al. (2018); Bakshi et al. (2019); Chen et al. (2020) to show the convergence of training deep neural networks. Later, convergence analysis using weaker assumptions are proposed, i.e., input data points are sep- arable Li & Liang (2018). Following Li & Liang (2018); Allen-Zhu et al. (2019b;c;a;; Zhang et al. (2020a), data separability plays a crucial role in deep learning theory, especially in showing the convergence result of over-parameterized neural network training. Denote ō is the minimum gap between all pairs data points. Data separability theory says as long as the width (m) of neural net- work is at least polynomial factor of all the parameters (m 2 poly(n, d, 1/))), i.c., n is the number of data points, d is the dimension of data, and delta is data separability. Another line of work Du et al. (2019); Arora al. 20;);;;); Song & Yang (2019); Brand et al. (2020); Lee al. (2020) builds on neural tangent kernel Jacot et al. (2018). It requires the minimum eigenvalue (A) of neural tangent kernel lower bounded. Recent work Oymak & Soltanolkotabi (2020) finds the connection between data-separyyiblity g and minimum eigenvalue X, i.e. 8 22/ A/n².
A.3 DISTRIBUTED DEEP LEARNING SYSTEM
Collaboration between the edge device and cloud server achieves higher inference speed and lowers power consumption than running the task solely on the local or remote platform. Typically there are two collaborative modes. The first is collaborative training, for which training task is distributed to multiple participants Konečný et al. (2015); Vanhaesebrouck et al. (2016); Kairouz et al. (2019). The other model is collaborative inference. In such a distributed system setting, the neural network can be divided into two parts. The first few layers of the network are stored in the local edge device, while the rest offloaded to remote cloud server. Given input, the edge device calculates the output of the first few layers and sends it to the cloud. Then cloud perform the rest of computation and sends the final results each edge device Eshratifar al. (2019); Hauswald al. (2014); Kang et al. (2017): Tecrapittayano et al. (2017). 7). In our work, we focus on tackling data recovery problem under collaborative inference mode.
A.4 MODEL. INVERSION ATTACK AND DEFENSE
The neural network inversion problem has been extensively investigated in recent years Fredrikson et al. (2015); He al. (2019); Lei et al. (2019); Zhang al. (2020b). As used in this paper, the general approach is to cast the network inversion as an optimization problem and uses a problem specified objective. In particular, Fredrikson al. (2015) proposes use confidence in prediction as to the optimized objective. He et al. (2019) uses a regularized maximum likelihood estimation. Recent work Zhang et al. (2020b) also proposes to GAN to do the model inversion.
There are very few studies about defenses against model inversion attack. Existing data privacy pro- tection mechanisms mainly rely noise injection Fredrikson al. (2015); Dwork (2008); Abadi
14

---

Under review as a conference paper at ICLR 2021
et al. (2016) or Homomorphic Encryption Nandakumar et al. (2019). While being able to mitigate at- tacks, existing methods significantly hinder model performance. Recently MID Wang et al. (2020) was proposed to limit the information about the model input contained in the prediction, thereby limiting the ability of an adversary to infer data information from the model prediction. Yang et al. (2020) proposed to add purification block following by prediction output, so that the confidence score vectors predicted by the target classifier are less sensitivity of the prediction to the change of input data. However, the above two methods target the logit output layer (i.e., performing argmax). They either require auxiliary information (i.e., knowing attack model) or modifying network struc- ture (i.e., building variational autoencoder structure for mutual information calculation). In contrast, our proposed method MixConcan easily and efficiently serve as plug-in loss to the middle layers of arbitrarily network structures to defend inversion attack during inference.
B HARDNESS OF NEURAL NETWORK INVERSION
B.1 PRELIMINARIES
We first provide the definitions for 3SAT, ETH, MAX3SAT, MAXE3SAT and then state some fun- damental results related those definitions. For more details, we refer the reader to the textbook Arora & Barak (2009).
Definition B.1 (3SAT problem). Given n variables and m clauses in a conjunctive normal form CNF formula with the size of each clause most 3. the goal is decide whether there exists an assignment to the n. Boolean variables to make the CNF formula be satisfied.
Hypothesis B.2 (Exponential Time Hypothesis (ETH) Impagliazzo et al. (1998)). There is a 8 > 0 such that the 3SAT problem defined in Definition B.1 cannot be solved in O(2")) time.
ETH is stronger notion than NP P, and is well acceptable the computational complexity com- munity. Over the few years, there has been work proving hardness result under ETH for theoretical computer science problems Chalermsook et al. (2017); Manurangsi (2017); Chitnis al. (2018); Bhattacharyya et al. (2018); Dinur & Manurangsi (2018); KCS & Manurangsi (2018) and machine learning problems, e.g. matrix factorizations Arora al. (2012); Razenshteyn et al. (2016): Song et al. (2017); Ban et al. (2019), tensor decomposition Song et al. (2019). There are also variations of ETH, e.g. Gap-ETH Dinur (2016; 2017); Manurangsi & Raghavendra (2017) and random-ETH Feige (2002); Razenshteyn et al. (2016), which are also believable in the computational complexity community.
Definition B.3 (MAX3SAT). Given n variables and m clauses, a conjunctive normal form CNF formula with the size of each clause at most 3, the goal is to find an assignment that satisfies the largest number of clauses.
We use MAXE3SAT to denote the version of MAX3SAT where each clause contains exactly literals.
Theorem B.4 (Hástad (2001)). For every 8 >> 0, it NP-hard to distinguish satisfiable instance of MAXE3SAT from an instance where at most a 7/8 + 5 fruction of the clauses can be simultaneously satisfied.
Theorem B.5 (Hástad (2001); Moshkovitz & Raz (2010)), Assume ETH holds. For every ō > O. there is no 20(n)-o() time algorithm distinguish satisfiable instance of MAXE3SAT from an instance where at most fraction 7/8 + 8 of the clauses can be simultaneously satisfied.
We use (B) to denote the restricted special case of MAX3SAT where every variable occurs in at most B clauses. Håstad Håstad (2000) proved that the problem is approximable to within a factor 7/8 + 1/(64B) in polynomial time, and that it is hard to approximate within factor 7/8 + (log ()((), In 2001, Trevisan improved the hardness result,
Theorem B.6 (Trevisan (2001)). Unless RP=NP, there no polynomial time (7/8 + 5/V approximate algorithm for MAXE3SAT(B).
Theorem B.7 (Hástad (2001); Trevisan (2001); Moshkovitz & Raz (2010)). Unless ETH there no 20(n -ocil) time (7/8 + 5/ 5////////////p//iaiiii/ aapproixxe algorithm for MAXE3SAT(B).
15

---

Under review as a conference paper at ICLR 2021
B.2 OUR RESULTS
We provide a hardness of approximation result for the neural network inversion problem. In partic- ular, we prove unless RP=NP, there no polynomial time that can approximately recover the input of a two-layer neural network with ReLU activation function". Formally, consider the inversion problem
h(z) =2, x [-[,111]
(4)
R is the hidden layer representation, h is two neural network with ReLU gates, specified as
where
h(x) = (Wix b), W2 E Rmax"" W1 E R xd b Rm
We want recover the input data E [-11, 1]d given hidden layer representation and all parameters of the neural network (i.e., W (22)) b). It is known the decision version of neural network inversion problem is NP-hard Lei et al. (2019). It is an open question whether approximation version also hard. We show stronger result which is, it is hard give to constant approximation factor. Two notions approximation could be consider here, one we called solution approximation
Definition B.8 (Solution approximation), Given a neural network h and hidden layer representation
2, we say E [-1,11 1jd an e approximation solution for Eq. (4), if there exists x E [1-1, R
such that
I r'112 S evd and h(x) = 2.
Roughly speaking, solution approximation means recovery an approximate solution. The Vd factor in the above definition is a normalization factor and it is not essential.
One can also consider weaker notion, which called function value approximation
Definition B.9 (Function value approximation). Given a neural network h and hidden layer repre- sentation 2, we say x' [-1, 1] is e-approximate of value to Eq. (4), if
[1((x') vll2 e\/22
Again, the mmz factor is only for normalization. Suppose the neural network is G-Lipschitz con- tinuous for constant (which is the case our proof), then an e-approximate solution implies Ge-approximation of value. For the purpose of this paper, we focus on the second notion (i.e., function value approximation). Given our neural network (constant)-Lipschitz continuous, this immediately implies hardness result for the first one.
Our theorem formally stated below. In the proof, we reduce from MAX3SAT(B) and utilize Theorem B.6
Theorem B.10. There exists a constant B > 1, unless RP NP, it is hard to eğn approximmae Eq. (4). Furthermore, the neural network is O continuous, and therefore, is hard to find an Q²)/B²) approximate solution to the neural network.
Using the above theorem, we can see that by taking a suitable constant B > 1, the neural network inversion problem is hard approximate within some constant factor under both definitions. In particular, we conclude
Theorem B.11 (Formal statement Theorem 3.3). Assume NP + RP, there exists a constant > O, such that there is polynomial time algorithm that is able to give an e-approximation to neural network inversion problem.
Proof of Theorem 10 Given an 3SAT instance & with n variables and m clause, where each vari- able appears in at most B clauses, we construct a two layer neural network h2 and output represen- tation satisfy the following:
Completeness. If o is satisfiable, then there exists E [0, 1]d such that hs(a) = z.
We remark there polynomial time algorithm for layer ReLU neural network recovery
16

---

Under review as a conference paper at ICLR 2021
Soundness. For any such that [1b((*) 2|12 I 60B /m,,
assignment
satisfies at least (3 vB m clauses
Lipschitz continuous. The neural network is O
We set d = n,, denote the j-th clause and use (x) to denote output of the j-th neuron in the first layer, i.e.
+ 200B5n and m2 + 100B For any E [m]], we use Φ to
(1) (W x b.), where Wr()) is the j-th row of WT)), For any i [ni] use X; to denote the i-th variable.
Intuitively, use the imput vector [-1, 1]" to denote the variable, and the first neurons in the first layer to denote the m clauses. By taking
X, X Oj; X, X, Øj: and bj 2
w
1,
0,
otherwise.
for any i E [n],j E [m.. and viewing 2; = 1 as X; to be false and æ; = -1 as X, to true. One can verify that (x)) if the clause is satisfied, and (x) 1 if the clause is unsatisfied. We simply copy the value in the second layer hr(x) = hı for j E [m]]
For other neurons, intuitively, we make 100B² copies for each |>s| E n) the output layer. This can be achieved by taking
hm+ (0=4(G-1))1008=44(())
and set
h (Gm= = max 0}
10 max -a;; 0}
for any i E [m], E [100 Finally, we set the target output as
(0,
100
We are left to prove the three claims we made about the neural network h and the target output z. For the first claim, suppose satisfiable and X (X1, Xx,) is the assignment. Then as argued before, we can simply take 1 if X; is false and = -1 is X; is true. One can check that h(x))= 2.
For second claim, suppose we are given E [-1, 1]dd such that
[h(z) - 2|12 -
60B m2
We start from the simple case when is binary, i.c.. E 1-1, Again, by taking X; to be true Ir - and X, to be false when r; 0. One can check that the number of unsatisfied clause is at most
[lb(x) =113 <
3600B m2
(m + 100B²n)
3600B
171 +
(5)
12
3600B
5
< m
VB m
The third step follows from < 3m, and the last step follows from B 2 15000.
Next, to the general case that E [-1, We would round x; -1 or +1 based on the
sign. Define F E {-1,1) 11.1)am
T arg min It xz1
(E(-1.1)
17

---

Under review as a conference paper at ICLR 2021
We prove that E induces an assignment that satisfies (3 vB
5 Im clauses. It suffices to prove
[ =112 - [1((x) - =12 - 100 m
(6)
since this implies the number of unsatisfied clause is bounded by
[|{[z) =113 - [1(()))
=11 (I)((T) -=12 - l1h(x)) =211)
1
-
m
+
36B² m) +
100 m
12
1
< m
5/B" m
8
where the second step follow from Eq. (5)(6, and the last step follows from B 2 10?. We define A;
|x; zi| |arel
E [0, 1] and T + 128B2n. Then we have
[|h(z) =112 ||{(T) 2113
(h)(z) 2,) - ((h((z)) 23)²
j=1
m
Σ j=1
(h2 (z) - 22)² - (12(()) - =2,²
Σ (ly(T) 2,,² -1g(()) 22)²
j=++1
m
n
Σ hz)) h,(x)) 100 Σ A
m
2
|115 (T) h2,(,)) - 100
A?
j=1
2
Σ A 1
Σ A
j=1 iEd
i=1
22B
A 100
i=1
n
100
3m
100
The third step follow from 2j 0 for j E [m] and for j E {m 1, m 100 n). 2j 1,
(z) 2, O and (x) 2, 112 = A. given im + (i - 1) 100 + 1, 100 The fourth step follows from that h, (x) = h1, E [0. for j E [m]. The fifth step follows from the I-Lipschitz continuity of the ReLU. The sixth step follows from each variable appears in at most B clause. This concludes the second claim.
For the last claim, by the Lipschitz continuity of ReLU, we have for any 21, 22
h(*1) h(x2) (2) (w) 21 + +b) ( b)
< || || |||= 22l12
easy to see that
| 2
and
|1
<
v 200 +3B 203B2 < 15B,
18

---

Under review as a conference paper at ICLR 2021
where the second step follows from B 2 1. Thus concluding the proof.
By assuming ETH and using Theorem B.7, we can conclude
Corollary B.12 (Formal statement of Corollary 3.5). Unless fails,,s, there exists constant O, such that there is no 20(n* o0)) time algorithm that is able to give an c-approximation to neural network inversion problem.
The proof is similar to Theorem B.10, omit here.
C DETAILS OF DATA RECOVERY EXPERIMENTS
C.I INVERSION MODEL DETAILS FOR SYNTHETIC DATASET
In synthetic experiment, malicious attacker recover original input data E IRd by solving the the following optimization:
x²'
arg min [lb(s) 2//2 SER"
To estimate the optimal, we run an SGD optimizer with a learning rate of 0.01 and decayed weight 10- for 500 iterations. We test data recovery results on all the 200 testing samples. Namely, we solve the above optimization problems 200 times. Each time for testing data point.
C.2 INVERSION MODEL DETAILS FOR BENCHMARK DATASET
In benchmark experiment, malicious attacker recover original input data Rd by solving the the following optimization:
x= = arg min [h(ss) - 22/2 + (5 + (sijt1 - 81/)2)2/22 sERd
where i, j are the indexes of pixels in image.
To estimate the optimal, we run an SGD optimizer with learning rate of 10 and decayed weight 10-4 4 for 500 iterations. We used grid searching on the space of Ç. We find that the best data recovery comes from ( = 0.01 for SVHN dataset and ( 10-5 for MNIST and FashionMNIST by grid search.
C.3 QUANTITATIVE METRICS FOR IMAGE SIMILARITY MEASUREMENT
We adopt the following two known metries to measure the similarity between and æ::
Normalized structural similarity index metric (SSIM), perception based metric that con- siders the similarity between images in structural information, luminance and contrast. is widely used in image and video compression research to quantify the difference between the original and compressed images. The detailed calculation can found Wang et al. (2004). We normalize SSIM to take value range 00, (original SSIM takes value range [)).
Perceptual similarity (PSIM). Perceptual loss Johnson et (2016) has been widely used for training image generation and style transferring models Johnson et al. (2016); Lucas al. (2019); Wang et al. (2018). emerges as a novel measurement for evaluating the discrepancy between high-level perceptual features that extracted by deep learning model the reconstructed image and ground-truth image. We define PSIM as perceptual loss.
19

---

Under review as a conference paper at ICLR 2021
D ADDITIONAL EXPERIMENTAL RESULTS
D.1 COMPARE PENALTY STRATEGIES
natural approach arise reduce data separability could be adding penalty on the pair-wise distance for the data representations within a class. We name this approach as UniCon. Its loss function denoted as Lunicon can be written as:
Lunicon
ΣEΣ CEC VEC JEC.
(1(x;)) - M
C Ccl. (1Cal -
The final objective function : Lelass X Lunicon- This approach is similar to contrastive learn- ing Khosla et al. (2020). However, we observed that the approach not as ideal as our proposed MixCon, in the sense of defending inversion attack. The intuition is that MixCon can induce con- fusing patterns to fool the neural network learning typical patterns from a class. Here we show the visualization for the three benchmark datasets Figure 5. We select À ionMNIST and À = 0.5 for SVHN in both UniCon and MixCon. Then we choose the 8 = 1e for MixCon match the accuracy to Vanilla and UniCon. We use the same training and testing of MixCon for UniCon experiment. From the representative samples (while typical to the rest of the data samples), we observe worse data recovery quality of MixCon. Notably, the recovered results from UniCon keep the pattern of their class. While MixCon results in more blurred and indistin- guishable patterns across classes. We compare the quantitative evaluation results between MixCon and UniCon in Table 4. We metric SSIM and PSIM to evaluate the similarity between the recovered image and the original image. Lower scores indicate worse data recovery results. The data recovery experiment performed 100 testing samples, and report the mean + std and worst case (the best-recovered data) results. Except for the PSIM scores evaluated on MNIST, we get conformable evidence showing MixCon training is apt to defend inversion.
for MNIST and Fash-
Original Image
Original Image
Vanillla
MINIST
q
Fashion MNIST
5
5 5
SVIN
Figure 5: Qualitative evaluation for image inversion results.
MNIST
MaSII
UniCon
MixCon
UniCon
MixCon
UniCon
MixCon
(A 1.0.8 10-4)
- 10-*)
0.5, 10
Aoc (%)
10.2
98.8
80.6
88.9
88.3
88.2
SSIM
(1.009
0.112.
0.09(0.58)
0.09(0.52)
0.67 11(099911
0.15
PSIM
0.41
(0.60)
0.44 0.07
0.45 1 (0.64)
0.42 1 (.00800.6))
0.62
0.05(0.75)
0.59 1 0.07
Table 4: Quantitative evaluations image recovery results. fair evaluation. match the data utility (accuracy) for Vanilla and MixCon. SSIM and PSIM are measured on 100 testing samples. Those scores are presented in mean + std and worst-case (in parentheses) format. The smaller scores indicate harder data recovery.
We have presented the comparison between MixCon and vanilla training in Table
20

---

Under review as a conference paper at ICLR 2021
D.2 EFFECTS ON THE SELECTION OF MIDDLE LAYERS
The trade-off between data separability and data utility can be different for adding MixConon the different layers. In our benchmark experiment, we LeNet5 LeCun (2015) five-layer CNN. Thus there are four split methods, namely four intermediate outputs. In our collaborative inference setting we visit the all four possible middle layers to apply MixCon loss. We plot the accuracy and data separability plots over the different combinations of (3, A), together with SSIM and PSIM scores (mean and the worst-case results), for each layer on our three benchmark datasets. The results are shown Figure 6 to Figure 17. There is a clear trend that the shallower the ((x) it is, the easier to recover original x on average with respect to the mean SSIM and PSIM scores. The worst-case measurement may suffer from some outliers and imperfectness of the evaluation metrics. In most cases, distance and recovery similarity score for the first three layers shows a positive relationship, Le. in Figure 6 Figure 8. Usually, inversion from the deeper layers is not stable. Also, splitting network at a deeper layer in the collaborative inference setting is not common or realistic because clients, such edge-end devices, do not have powerful computational resources. Notably, the relationship between accuracy and similarity is highly non-linear. The sweet spot for a trade-off between accuracy and difficulty of recovery in the space where the accuracy degradation curve is slow, while recovery similarity is low. Users can search the best parameters and "cut layer" to meet certain accuracy and data recovery defending requirements in practice.
10There is no necessity to add MixCon loss for the layers hefore "cut layer", because the attacker is not able get to the original data hidden representations form those layers
21

---

Under review as a conference paper at ICLR 2021
Accuracy
Distance
1.0
0.29
accurac
distance
0.10
0.4
0.2
0.6
0.05
0.0
0.00
100.0
100.0
5.0
5.0
0.01
1.0
1.0
0.0001
0.1
0.0001
0.1
1e-08
1e-0
(a) Accuracy vs. Distance
Mean SSIM
Worst SSIM
0.9
0.8
0.75
0.9
0.7
0.8
0.5
0.6
0.50
0.7
0.2
0.4
0.5
0.25
0.6
0.3
0.5
960.0
0.2
9,8.00
0.4
0.1
1.0
1.0
0.01
0.0001
0.1
>>
0.01
0.0001
0.1
1e-06
8
1e:06
PSIM
1.00
1.00
0.75
0.75
0.95
0.50
0.50
0.85
0.25
0.70
0.25
0.80
4,0.0
0.60
0.65
9,80.0
0.75
0.70
5.0
0.01
0.0001
0.1
>>
0.0001
0.1
1e:06
1e:08
/
1e
(b) Quantitative evaluations on data recovery
Figure 6: Adding MixCon to the 1st layer of CNN on MNIST dataset. (a) The trade-off between data separa- bility and data utility. We show testing accuracy and mean pairwise distance (data separability) with different A and B. À and 3 show complementary effort on adjusting data separability. (b) Quantitative evaluation of data recovery results. We show SSIM and PSIM scores with different À and B3.
22

---

Under review as a conference paper at ICLR 2021
Accuracy
Distance
0.11
accurac
distance
0.075
0.6
0.75
0.050
0.4
0.2
0.025
0.0
0.25
0.00
100.0
100.0
5.0
5.0
0.01
1.0
0,0001
0.1
0.0001
0.1
8
1e-08
1e-08
(a) Accuracy vs. Distance
Mean SSIM
Worst SSIM
0.45
0.40
0.75
0.35
0.5
0.30
0.50
0.25
0.2
0.20
0.25
0.15
96b.0
0.10
9,8.0
0.3
0.05
1.0
1.0
0.01
0.0001
0.1 >>
0.01
0.0001
0.1
1e-06
8
1e:06
PSIM
PSIM
1.00
1.00
0.75
0.75
0.8
0.50
0.60
0.55
0.50
0.7
0.25
0.50
0.45
0.25
0.6
9,0.0
0.40
9,0.0
0.35
5.0
0.4
0.01
0.0001
0.1
>>
0.0001
0.1
1e:06
1e
1e:08
1e 18
(b) Quantitative evaluations on data recovery
Figure 7: Adding MixCon to the 2nd layer of CNN on MNIST dataset. (a) The trade-off between data separability and data utility We show testing accuracy and mean pairwise distance (data separability) with different À and B. and 3 show complementary effort on adjusting data separability. (b) Quantitative evaluation of data recovery results. We show SSIM and PSIM scores with different and B.
23

---

Under review as a conference paper at ICLR 2021
Accuracy
Distance
1.28
accuracy
distance
0.6
0.75
0.4
0.50
0.2
0.0
0.25
0.00
100.0
100.0
0.0
5.0
5.0
0.01
1.0
0,0001
0.1
0.0001
0.1
8
1e-08
1e-08
(a) Accuracy vs. Distance
Mean SSIM
Worst SSIM
1.0
1.00
0.30
0.8
0.25
0.75
0.6
0.50
0.25
0.10
0.3
9d0.0
0,80.0
0.05
0.2
5.0
1.0
0.01
0.0001
0.1
0.01
0.0001
0.1
>>
8
1e-06
1e-06
1.00
1.00
0.50
0.75
-0.450
-0.475
0.500
0.525
0.50
0.75
0.60
0.65
0.25
-0.425
0.25
0.55
9,80.0
0.400
0.375
9,8.0
0.50
0.45
5.0
0.0001
0.1
0.01
0.0001
0.1
8
1e 06
1e
8
1e:06
1e:88
(b) Quantitative evaluations on data recovery
Figure 8: Adding MixCon to 3rd layer CNN on MNIST dataset. (a) The tradc-off between data separa- bility and data utility. We show testing accuracy and mean pairwise distance (data separability) with different A and B. A and 9 show complementary effort on adjusting data separability. (b) Quantitative evaluation of data recovery results. We show SSIM and PSIM scores with different À and B3.
24

---

Under review as a conference paper at ICLR 2021
Accuracy
Distance
1.0
141
distance
0.8
0.4
0.5
0.2
0.0
0.00
100.0
100.0
5.0
5.0
0.01
1.0
1.0
0.0001
0.1
0.0001
0.1
1e-08
1e-08
(a) Accuracy vs. Distance
Mean Mean_SSIM SSIM
Worst SSIM
1.0
0.085
0.30
0.8
0.080
0.75
0.075
0.50
0.26
-0.070
0.065
0.25
0.24
0.060
0.22
980.0
0.055
0,88.0
5.0
0.20
1.0
0.01
0.0001
0.1
0.01
0.0001
0.1
8
1e-06
1e-06
Worst
1.00
0.525
H
1.00
0.72
0.75
0.500
0.75
0.70
0.68
0.50
0.450
0.50
0.66
0.64
0.25
0.425
0.25
0.62
0.400
0.60
9,80.0
0.375
0,08.0
5.0
0.0001
0.1
0.01
0.0001
0.1
8
1e 06
1e
8
1e:06
18:88
(b) Quantitative evaluations on data recovery
Figure 9: Adding MixCon to 4th layer CNN on MNIST dataset. (a) The trade-off between data separa- bility and data utility. We show testing accuracy and mean pairwise distance (data separability) with different A and B. A and 3 show complementary effort on adjusting data separability. (b) Quantitative evaluation of data recovery results. We show SSIM and PSIM scores with different À and B3.
25

---

Under review as a conference paper at ICLR 2021
Accuracy
Distance
1.0
0.49
0.20
0.8
distance
0.15
0.4
0.6
0.10
0.2
0.0
0.00
0.4
100.0
100.0
5.0
5.0
0.01
1.0
1.0
0.0001
0.1
0.0001
0.1
1e-08
1e-0
(a) Accuracy vs. Distance
Mean SSIM
Worst SSIM
1.0
1.00
0.8
0.9
0.75
0.975
0.8
0.950
0.50
0.7
0.900
0.6
0.25
0.875
0.850
980.0
0.5
0,88.0
0.825
5.0
1.0
0.01
0.0001
0.1
0.01
0.0001
0.1
8 1e:06
1e-06
1.00
1.00
0.75
0.900
0.925
0.75
0.98
0.97
0.50
0.850
0.875
0.50
0.95
0.96
0.25
-0825
0.25
0.800
0.93
9,80.0
0.775
0,08.0
0.92
5.0
0.750
0.91
0.0001 8
0.1
0.01
0.0001
0.1
1e 06
1e
8
1e:06
1e:08
(b) Quantitative evaluations on data recovery
Figure 10: Adding MixCon to the 1st layer CNN on MNSSI dataset. (a) trade-off between data separability and data utility. We show testing accuracy and mean pairwise distance (data separability) with different À and B. X and 3 show complementary effort adjusting data separability. (b) Quantitative evaluation of data recovery results. We show SSIM and PSIM scores with different À and B.
26

---

Under review as a conference paper at ICLR 2021
Accuracy
Distance
1.0
0.29
distance
0.15
0.4
0.2
0.05
0.0
0.00
100.0
0.2
100.0
5.0
5.0
0.01
1.0
1.0
0.0001
0.1
0.0001
0.1
1e-08
1e-00
(a) Accuracy vs. Distance
Mean_ SSMM
Worst SSIM
0.75
0.7
0.5
0.50
0.6
0.5
0.2
0.25
98b.0
9,8.0
0.3
1.0
1.0
0.01
0.0001
0.1
>>
0.01
0.0001
0.1
1e-06
8
1e:06
PSIM
PSIM
1.00
1.00
0.65
0.85
0.75
0.75
0.80
0.75
0.50
0.50
0.70
0.50
0.25
0.25
0.65
4,0.0
0.45
0.40
9,00.
0.60
5.0
0.01
0.0001
0.1
>>
0.0001
0.1
1e:06
1e:08
/
1e
1c 18-08
(b) Quantitative evaluations on data recovery
Figure 11: Adding MixCon the 2nd layer of CNN on FashionMNIST dataset. (a) The trade-off between data separability and data utility. We show testing accuracy and mean pairwise distance (data separability) with different À and B. X and 3 show complementary effort adjusting data separability. (b) Quantitative evaluation of data recovery results. We show SSIM and PSIM scores with different À and B.
27

---

Under review as a conference paper at ICLR 2021
Accuracy
Distance
1.0
1.9
0.8
distance
0.6
0.4
0.2
0.4
0.5
0.0
0.2
0.00
100.0
100.0
0.0
5.0
5.0
0.01
1.0
1.0
0.0001
0.1
0.0001
0.1
1e-08
1e-0
(a) Accuracy vs. Distance
Mean_ SSIM
Worst SSIM
0.18
0.55
0.16
0.75
0.50
0.5
0.14
0.50
0.2
0.12
0.25
0.40
0.35
96b.0
9,80.0
0.30
1.0
1.0
0.01
0.0001
0.1
>>
0.01
0.0001
0.1
1e-06
8
1e:06
PSIM
PSIM
1.00
1.00
0.48
0.675
0.75
0.46
0.75
0.650
0.44
0.625
0.50
0.50
0.600
-0.75
0.25
0.40
0.25
0.550
4,8.0
0..8
9,0.0
0.500
0.525
0.36
5.0
0.01
0.0001
0.1
>>
0.0001
0.1
1e:06 06
1e
1e:08
1c 18-08
(b) Quantitative evaluations on data recovery
Figure 12: Adding MixCon to 3rd layer of CNN on FashionMNIST dataset. (a) trade-off between data separability and data utility We show testing accuracy and mean pairwise distance (data separability) with different À and B. X and 3 show complementary effort adjusting data separability. (b) Quantitative evaluation of data recovery results. We show SSIM and PSIM scores with different À and B.
28

---

Under review as a conference paper at ICLR 2021
Accuracy
Distance
1.0
1.39
accurag
0.8
distance
1.0
0.4
0.6
0.5
0.2
0.0
0.00
100.0
100.0
5.0
5.0
0.01
1.0
1.0
0.0001
0.1
0.0001
0.1
1e-08
1e-08
(a) Accuracy vs. Distance
Mean SSIM
Worst SSIM
1.0
1.00
0.8
0.17
0.16
0.75
0.75
0.15
0.50
0.70
0.65
0.13
0.25
0.60
0.55
980.0
0,88.0
0.50
5.0
0.11
1.0
0.01
0.0001
0.1
0.01
0.0001
0.1
8
1e:06
1e-06
Worst
1.00
1.00
0.75
0.405
0.410
0.75
0.750
0,775
0.50
0.400
0.50
0.700
0.725
0.25
0.390
0.25
0.675
0.385
9,80.0
0.380
0,88.0
0.625
5.0
0.375
0.600
0.0001 1e
0.1
0.01
0.0001
0.1
8
1e 06
8
1e:06
1e:08
(b) Quantitative evaluations on data recovery
Figure 13: Adding MixCon to 4th layer of CNN on FashionMNIST dataset. (a) trade-off between data separability and data utility. We show testing accuracy and mean pairwise distance (data separability) with different À and B. À and 3 show complementary effort adjusting data separability. (b) Quantitative evaluation of data recovery results. We show SSIM and PSIM scores with different À and B.
29

---

Under review as a conference paper at ICLR 2021
Accuracy
Distance
1.0
0.90
0.8
0.3
distance
0.6
0.2
0.4
0.2
0.4
0.0
0.00
100.0
100.0
5.0
5.0
0.01
1.0
1.0
0.0001
0.1
0.0001
0.1
1e-08
1e 08
(a) Accuracy vs. Distance
Mean SSIM
Worst SSIM
0.75
0.5
0.66
0.50
0.2
0.5
0.25
0.4
960.0
0.3
9,8.0
1.0
1.0
0.01
0.0001
0.1
>>
0.01
0.0001
0.1
1e-06
8
1e:06
PSIM
1.00
1.00
0.75
0.70
0.75
0.75
0.80
0.85
0.50
0.65
0.50
0.60
0.25
0.25
0.70
0.65
4,0.0
0.45
9,0.0
0.60
0.40
5.0
0.01
0.0001
0.1
>>
0.0001
0.1
1e:06
1e
1e:08
(b) Quantitative evaluations on data recovery
Figure 14: Adding MixCon to the 1st layer CNN on SVHN dataset. (a) The tradc-off between data separa- bility and data utility. We show testing accuracy and mean pairwise distance (data separability) with different A and B. A and 9 show complementary effort on adjusting data separability. (b) Quantitative evaluation of data recovery results. We show SSIM and PSIM scores with different À and B3.
30

---

Under review as a conference paper at ICLR 2021
Accuracy
Distance
1.0
107
0.8
distance
0.4
0.4
0.6
0.2
0.2
0.0
0.00
100.0
100.0
5.0
5.0
0.01
1.0
1.0
0.0001
0.1
0.0001
0.1
1e-08
1e 08
(a) Accuracy vs. Distance
Mean SSIM
Worst SSIM
0.9
0.7
0.75
0.8
0.5
0.50
0.7
0.2
0.25
0.6
0.5
96b.0
0.3
9,86.0
0.4
1.0
1.0
0.01
0.0001
0.1
>>
0.01
0.0001
0.1
1e-06
8
1e:06
1.00
1.00
0.68
0.800
0.75
0.66
0.75
0.775
0.750
0.50
0.62
0.50
0.725
0.7000
0.25
0.25
0.675
0.650
4,0.0
9,0.0
0.625
0.52
5.0
0.01
0.0001
0.1
>>
0.0001
0.1
1e:06
1e
1e:08
lc 08
(b) Quantitative evaluations on data recovery
Figure 15: Adding MixCon to the 2nd layer of CNN on SVHN dataset. (a) The trade-off between data separability and data utility. We show testing accuracy and mean pairwise distance (data separability) with different À and B. and 3 show complementary effort adjusting data separability. (b) Quantitative evaluation of data recovery results. We show SSIM and PSIM scores with different and B.
31

---

Under review as a conference paper at ICLR 2021
Accuracy
Distance
1.0
144
distance
0.4
0.2
0.4
0.0
0.00
100.0
0.2
100.0
5.0
5.0
0.01
1.0
1.0
0.0001
0.1
0.0001
0.1
1e-08
1e-08
(a) Accuracy vs. Distance
Mean_ SSIM
Worst. SSIM
0.5
0.6
0.75
0.7
0.5
0.4
0.50
0.5
0.4
0.2
0.2
0.25
0.3
0.2
980.0
9,8.00
0.1
0.0
1.0
1.0
0.01
0.0001
0.1
>>
0.01
0.0001
0.1
1e-06
8
1e:06
PSIM
PSIM
1.00
1.00
0.75
0.60
0.75
0.70
0.50
0.50
0.65
0.25
0.50
0.25
0.60
4,8.0
9,0.0
0.55
0.40
5.0
0.50
0.01
0.0001
0.1 >>
0.0001
0.1
1e:06
1e
1c 18-08
(b) Quantitative evaluations on data recovery
Figure 16: Adding MixCon to the 3rd layer of CNN on SVHN dataset. (a) The trade-off between data separability and data utility We show testing accuracy and mean pairwise distance (data separability) with different À and B. and 3 show complementary effort on adjusting data separability. (b) Quantitative evaluation of data recovery results. We show SSIM and PSIM scores with different and B.
32

---

Under review as a conference paper at ICLR 2021
Accuracy
Distance
1.0
1446
distance
1.0
0.6
0.4
0.2
0.4
0.0
0.2
0.00
100.0
100.0
5.0
5.0
0.01
1.0
1.0
0.0001
0.1
0.0001
0.1
1e-08
1e-08
(a) Accuracy vs. Distance
Mean_ SSMM
Worst SSIM
0.75
0.20
0.5
0.15
0.50
0.2
0.10
0.25
0.3
0.2
980.0
0.05
9,8.00
0.1
1.0
1.0
0.01
0.0001
0.1
>>
0.01
0.0001
0.1
1e-06
8
1e:06
PSIM
PSIM
1.00
1.00
0.700
0.75
0.75
0.675
0.650
0.50
0.50
0.625
0.600
0.25
0.46
0.25
0.575
0.550
4,0.0
0.44
9,0.0
5.0
0.01
0.0001
0.1
>>
0.0001
0.1
1e:06
1e
1e:08
1c 18
(b) Quantitative evaluations on data recovery
Figure 17: Adding MixCon to the 4th layer of CNN on SVHN dataset. (a) The trade-off between data separability and data utility We show testing accuracy and mean pairwise distance (data scparability) with different À and B. and 3 show complementary effort on adjusting data separability. (b) Quantitative evaluation of data recovery results. We show SSIM and PSIM scores with different and B.
33


Reviews:
[{"review_id": "ttLRPm_iWhM", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Post response update\n\nThanks for the response. However, my major concern is still that the technical contribution of this paper is limited.\n\n-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------\n\nThis paper studies an important problem, i.e., the vulnerability of deep neural networks to model inversion attacks. To address this problem, the authors design an objective function to adjust the separability of the hidden data representations as a way to control the trade-off between data utility and vulnerability to inversion attacks. Specifically, the authors propose a consistency loss (MixCon) to train data feature extractor to help protect original data to be inverted by attacker during inference. The authors also conduct experiments on synthetic and benchmark datasets to evaluate the performance of the proposed method. However, I still have the following concerns.\n\n[-] The proposed consistency penalties (i.e., Eq. (1)) are quite straightforward and intuitional. Additionally, it lacks theoretical analysis of the proposed loss terms. For example, it would be better if the authors could analyze how much the attackers’ attacking ability are reduced by using the proposed loss terms and the approximation error before and after incorporating the consistency penalties.\n\n[-] For the threat model, the authors assume that the attacker not only has access to extracted features but also all network parameters of the trained model. However, these assumptions are strong and unrealistic in the real-world applications.\n\n[-] More details are needed. Firstly, in Theorem 3.2, the authors claim that under certain cases, (stochastic) gradient descent algorithm can find the global minimum of neural network function $f$. It would be better if the authors provide the proof for this theorem. Additionally, the authors mention that lower bound and upper bound imply better accuracy and hardness of inversion, respectively. However, the definitions of both lower bound and upper bound are not clearly. It would be better if the authors provide the formal (mathematical) definitions.\n\n[-] The authors fail to cite existing state-of-the-art works on defending model inversion attacks, e.g., [1,2,3,4]. Additionally, in experiments, state-of-the-art baselines are not adopted. Currently, there are some existing defenses against model inversion attacks, e.g., [1,2,3,4]. It would be better if the authors give further discussion and compare the performance of the proposed method with that of these existing works.\n\n[-] Some typos should be corrected. Just list some of them.\n* “by proposing and a self-supervised learning based feature extractor” in Page 1.\n* “force for the data representation” in Page 4.\n\n\n[1] “Improving Robustness to Model Inversion Attacks via Mutual Information Regularization”, 2020.\n\n[2] “Defending Model Inversion and Membership Inference Attacks via Prediction Purification”, 2020.\n\n[3] “Model Inversion Attacks that Exploit Confidence Information and Basic Countermeasures”, 2015.\n\n[4] “Adversarial neural network inversion via auxiliary knowledge alignment”, 2019.", "extra_scores": {}, "extra_text": {}}, {"review_id": "c0AdqW6FMAW", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Post response update\n\nI would like to thank the authors for their detailed response. The response addresses my confusion around the use of the terminology of model inversion (I would further suggest that the author use the term data reconstruction rather than than model inversion to avoid readers misunderstanding model inversion as referring to [1]). I still have concerns around the fact that differential privacy is not used as a baseline here, which would strengthen the argument made in the response that it provides orthogonal guarantees. \n\n-----------\n\nThe paper looks at the tradeoff between data separability in an embedding space and vulnerability to model inversion. The paper hypothesizes that increasing data separability improves accuracy but exposes the model to model inversion attacks. \n\n- At a high level it is not clear why the model inversion is a well-motivated attack vector given that model inversion extracts an average representation of the points from a class, not specific training points. The introduction talks about data recovery but it is not clear what it means to “recover” a data point in the context of this work.\n- Grammar in page 1: “ The central question here is how to better protect the data from being reconstruct while keeping useful information to the classification task.”\n- Grammar in page 1: “ by proposing and a self-supervised learning- based feature extractor”\n- Unclear what the following means: “  At a high level, we focus on the pipelines combining local data representation learning with global model learning manner.”\n- Section 3.1 proposes data separability as a measure of privacy, but it is not clear why it is necessary to introduce a new definition for privacy. Why not consider well-established definitions such as differential privacy? \n- Section 3.1 proposes informal statements tying data separability but the statements are not demonstrated, and the analysis does not outline how the results would be proved (There is also not proof in supplementary materials). \n- Section 3.2 introduces concepts such as “confusion” in embedding space without defining them. Moving forward, making the claims more precise would help make the paper more readable.\n- How does the approach from Section 3.2 relate to other losses like the triplet loss which compare distances between different points in the embedding space?\n- Section 4.1 does not specify whether the input x is part of the training and test set. Is the goal of the model inversion to invert a training point or instead a test point which the model is inferring on? \n- It is not clear why the experimental setup from Section 4.2 is well-motivated to study the model inversion problem given that the model architecture being fully connected, and the dataset being synthesized, but not used in prior work. \n- In Section 4.2, it is unclear how the quality of a reconstruction is evaluated. Is a successful model inversion evaluated based on human perception and a similarity metric? Does the lack of similarity mean that there is no privacy leakage? Here it seems that this comes back to the definition of privacy used, which looks at average case rather than worst case.\n- The introduction discussed a distributed setting scenario but this does not seem to be considered in the experimental setup itself. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "0EL8H8igZwk", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper aims at strengthening DNN against inversion attack. It proposes to utilize an extra term in NN training objective function, called L_mixcon, to play with separability of  hidden representation of data in different classes. Following is my concern:\n-\tReducing hidden representation separability of data points in different classes equates to more confusion for classification. All the results shown in the paper, e.g. Table 3 or Figure 4, confirm that and show that there is a trade off between accuracy and robustness against the inversion attack. Thus, this proposed method could be helpful if the user is willing to give up on some accuracy in the hope of getting a more robust model. However, the paper does not highlight this fact and presents the method as if it provides the same accuracy with higher robustness. For example in page 8 it is stated that : “We select (lambda, beta) to match the accuracy results of MixCon to be as good as Vanilla training (see Accuracy in Table 3),” whereas in Table 3 the accuracy is not as good as Vanilla, and it is misleading.\n-\tIt is also obvious from formula (1) that minimizing L_mixcon with lower beta or higher lambda degrades the accuracy, and the provided results show the same thing. But it is not shown that the sweet spot exists as there is always a trade-off.\n-\tRegarding formula 1:\no\tWhy do we only look at p data points per class and not all?\no\tThe formula says i-th data point form class c_1 would be compared against i_th data point from class c_2. How do you do this one-to-one mapping between points of different classes? Or did you mean all pairs and the formula is not written properly?\no\tAlso, as a suggestion, I think designing beta as a function of classe (c) would be more appropriate.\n-\tRegarding the layer h, I am wondering if there is any recommendation how to choose that and how many layers to choose. In this paper only one layer is considered. I can also imagine the best choice of beta and lambda depends on the choice of layer.\n-\tIt is mentioned that the local feature extractor is a shallow NN in the setting, is it one layer CNN with nonlinearity or a linear model?\n-\tThe paper is understandable but there are multiple typos and the English could be improved.\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: In this paper, the authors change the loss function of NNs to reduce the separability of the different classes in one of the hidden layers. The rationale for this assumption that the trained network will be more robust against white-box model inversion attack. The reviewers all concur that the paper had some merit, but that the paper is not well presented and believe the paper is not ready to be presented at ICLR.

Also, the separability issue is not totally explained, because a reduced L2 norm might not be the whole story that explains why a white-box model inversion would rely on for leaking information. This might need to be proof further and a couple of experiments in which there is still leakage of information shows the additional robustness from the new penalty.

Decision: Reject