Information Laundering for Model Privacy
Authors: Xinran Wang
Yu Xiang
Jun Gao
Jie Ding
Venue: ICLR 2021 Spotlight (2021)
Keywords: Adversarial Attack
Machine Learning
Model privacy
Privacy-utility tradeoff
Security

Abstract: In this work, we propose information laundering, a novel framework for enhancing model privacy. Unlike data privacy that concerns the protection of raw data information, model privacy aims to protect an already-learned model that is to be deployed for public use. The private model can be obtained from general learning methods, and its deployment means that it will return a deterministic or random response for a given input query. An information-laundered model consists of probabilistic components that deliberately maneuver the intended input and output for queries of the model, so the model's adversarial acquisition is less likely. Under the proposed framework, we develop an information-theoretic principle to quantify the fundamental tradeoffs between model utility and privacy leakage and derive the optimal design.

Published as a conference paper at ICLR 2021
INFORMATION LAUNDERING FOR MODEL PRIVACY
Xinran Wang
Yu Xiang
School of Statistics
Electrical and Computer Engineering
University of Minnesota- Twin Cities Minneapolis, MN 55455, USA wang87 4 06 umn. edu
University of Utah
Salt Lake City, UT 84112, USA yu. xiangeut edu
Jun Gao
Jie Ding
Department of Mathematics Stanford University Stanford, CA 94305, USA
School of Statisties
University of Minnesota- Twin Cities Minneapolis, MN 55455, USA ding jumnn edu
jung28stanFord.edd edu
ABSTRACT
In this work, we propose information laundering, novel framework for enhane- ing model privacy. Unlike data privacy that concerns the protection of raw data information, model privacy aims to protect an already-learned model that to be deployed for public methods, and its deployment means that will return a deterministic or random response for given input query. An information irformatioo--landereed model consists of probabilistic components that deliberately maneuver the intended input and out- put for queries of the model, the model's adversarial acquisition is less likely. Under the proposed framework, we develop an anformation-theoree principle to quantify the fundamental tradeoff's between model utility and privacy leakage, and derive the optimal design.
The private model can be obtained from general learning
INTRODUCTION
An emerging number of applications involve the following user-scenario. Alice developed model that takes a specific query as input and calculates a response as output. The model is a stochastic black-box that may represent novel type of ensemble models, known deep neural network chitecture with sophisticated parameter tuning, or physical law descrihed by stochastic differential equations. Bob is a user that sends a query to Alice and obtains the corresponding response for his specific purposes, whether benign or adversarial. Examples of the above scenario include many re- cent Laatiine-Laanin-aa---- et al., 2020) and artificial intelligence chips, where Alice represents learning service provider, and Bob represents users.
(MLaaS) services (Alabbadi, 2011; Ribeiro et al., 2015; Xian
Suppose that Bob obtains sufficient paired input-output data as generated from Alice's black-box model, it is conceivable that Bob could treat as supervised data and reconstruct Alice's model to some extent. From the view of Alice, her model may be treated as valuable and private. As Bob who queries the model may be benign or adversarial, Alice may intend to offer limited utility for the return of enhanced privacy. The above concern naturally motivates the following problem.
(Q1) How to enhance the privacy for an already-learnee model? Note that the above problem is not about data privacy, where the typical goal is prevent adversarial inference of the data infor- mation during data transmission or model training. contrast, model privacy concerns an already- established model. We propose study general approach to jointly maneuver the original query's input and output so that Bob finds it challenging to guess Alice's core model. As illustrated in Figure 1a, Alice's s model is treated transition kernel (or communication channel) that produces y conditional on any given X. Compared with an honest service Alice would have provided (Fig- 1b), the input X is maneuvered version of Bob's 's original input X; Moreover, Alice may choose

---

Published as a conference paper at ICLR 2021
(a)
(b)
Query
Pseudo-query
resuoo-resoonse
Response
Query
Response
K₁
2
K.
K2
X
K.
Y
K
Figure I: Illustration of (a) Alice's effective system for public use, and (b) Alice's idealistic system not for public use. In the figure, K, denotes the already-learned model/API, K₃ denotes the kernel that perturbs the input data query by potential adversaries, and K₂ denotes the kemel that perturbs the output response from K. to publish the final response Y.
to return a perturbed outcome Y instead of Y to Bob. Consequently, the apparent kernel from Bob's input query X the output response Y a cascade of three kernels, denoted by K Figure la. The above perspective provides a natural and general framework to study model privacy. Admittedly, if Alice produces a (nearly) random response, adversaries will find it difficult to steal the model, while benign users will find it useless. Consequently, raise another problem.
(Q2) How to formulate the model privacy-utility tradeoff. and what is the optimal way of imposing privacy? To address this question, we formulate model privacy framework from information- theoretic perspective, named information laundering. We briefly describe the idea below. The gen- eral goal is jointly design the input and output kernels (Kı and K2 in Figure Ia) that deliberately maneuver the intended input and output for queries of the model so that 1) the effective kernel (K in Figure la) for Bob is not too far away from the original kernel (K, in Figure la), and 2) adver- sarial acquisition of the model becomes difficult. Alternatively, Alice 'launders' the input-output information maximally given a fixed utility loss. To find the optimal way of information launder- ing, we propose an objective function that involves two components: the first being the information (KL)
shared between X, X and between Y, Y, and the second being the average Kullback-Leine divergence between the conditional distribution describing K and K.. Intuitively, the first compo- nent controls the difficulty of guessing K. sandwiched between two artificial kemels K₃ and K2, while the second component ensures that overall utility is maximized under the same privacy con- straints. By optimizing the objective for varying weights between the components, can quantify the fundamental tradeoffs between model utility and privacy.
1.1 RELATED WORK
We introduce some closely related literature below. Section 3.3 will incorporate more technical discussions some related but different frameworks, including information bottleneck, local data privacy, information privacy, and adversarial model attack.
A closely related subject of study is data privacy, which has received extensive attention in cent years due to socictal concerns (Voigt & Von dem Bussche, 2017; Evans et al., 2015; Cross & Cavallaro, 2020; Google, 2019; Facebook, 2020). Data privacy concerns the protection of (usu- ally personal) data information from different perspectives, including lossless cryptography (Yao, 1982; Chaum et al., 1988), randomized data collection (Evfimievski et al., 2003: Kasiviswanathan et al., 2011; Ding & Ding, 2020), statistical database query (Dwork & Nissim, 2004; Dwork, 2011), membership inference (Shokri al., 2017), and Federated learning (Shokri & Shmatikov, 2015; Koneveny et al., 2016; McMahan et al., 2017: Yang et al., 2019; Diao et al., 2020). A common goal in data privacy learning. In contrast, the subject of model privacy focuses on protecting a single learned model ready to deploy. For example, we want to privatize a classifier to deploy on the cloud for public use, whether the model is previously trained from raw image data or a data-private procedure.
obfuscate individual-level data values while still enabling population
Another closely related subject is the model extraction in (Tramèr et al., 2016; Papernot et al., 2016b), where Bob's goal is to reconstruct Alice's model from several queries' inputs and outputs, knowing what specific model Alice uses. For example, suppose that Alice's 's model is a generalized linear regression with features. In that case, it is likely to be reconstructed using p queries of the ex- pected mean (a known function of X 3) by solving equations (Tramèr et al., 2016). In the supervised classification scenario, when only labels returned to any given input, model extraction could be
2

---

Published as a conference paper at ICLR 2021
cast active learning problem where the goal is to query most efficiently (Chandrasckaran al., 2018). Model extraction was also studied in contexts beyond the prediction API, e.g., when am ad- versary utilizes the gradient information (Milli al., 2019). From Alice's perspective, there exist several solutions to safeguard against model leakage. A warning-based method was developed in (Kesarwani et al., 2018), where Alice continuously monitors the information gain and raise alarms when they become unusual. Another warning method was developed in (Juuti et al., 2019), where the detection of an adversary based on testing whether the pairwise distances among queried data approximately follow the Gaussian distribution. The work in (Lee et al., 2018) developed a defense strategy for the setting where the target classifier, and the adversary queries each class's prob- ability. The probabilities are maximally perturbed under the constraint that the most-likely class label) remains the same. The work (Orekondy et al., 2019) studied similar setting but from a different perspective. The main idea is to perturb the probabilities within an lg di-dtataece constraint to poison the adversary's 's gradient signals.
1.2 CONTRIBUTIONS AND OUTLINE
The main contributions of this work are three folds. First, we develop novel concept. theory, and method, generally referred to as information laundering, to study model privacy. Unlike data privacy that concerns the protection of raw data information, model privacy aims to privatize an already-learned model for public use. To the best of the authors' knowledge, this work is the first framework study model privacy in principled manner. Second, under the developed information- theoretic framework, we cast the tradeoffs between model privacy and utility as general optimiza- tion problem. We derive the optimal solution using the calculus of variations and provide extensive discussions the solution's 's insights from different angles. Third, we develop a concrete algorithm. prove its convergence, and elaborate on some specific cases. We also provide some experimental studies to illustrate the concepts, and discuss several future problems at the end.
The paper is organized as follows. In Section 2, we describe the problem formulation and a gen- eral approach to protect the model. In Section 3, propose the information laundering method that casts the model privacy-utility tradeoff as an optimization problem and derives a general solution. In Section 3.3, provide some additional discussions of the related frameworks, including informa- tion bottleneck, local data privacy, information privacy, and adversarial model attack. In Section 5, conclude the paper with some potential future work. In the Appendix, provide the proofs of the main results and experimental studies.
2 FORMULATION
2.1 BACKGROUND
The private model can be obtained from general learning methods, and its deployment means that will return a response for given input query. Suppose that X' and y are the input and output alphahets (data space), respectively.
Definition 1 (Learned model) A learned model is kernel p X y + 10, 11. which induces class of conditional distributions {p(- x) E X}.
model in the above definition is also referred to as a communication channel in information theory. A model can be regarded as the input-output (or Alice's application programming interface, API) offered to Bob. Examples include regression/clasiiicati model that outputs predicted labels, clustering model that outputs the probabilities of belonging to specific groups, and a stochastic differential equation system that outputs the likely paths for various inputs variables. It does not matter where the model comes from since are only concerned about the privacy of fixed given model. The (authentic) model of Alice is denoted by Pk.
An adversary Bob is user that can access the above model's API, providing arbitrary input, X, and obtaining an output, Y. Bob aims to use as few queries as possible to construct a model that closely matches Alice's 's model Px.- - We will formalize the 'closeness' using the KL divergence.
What is model privacy? Our perspective is that privacy is not an intrinsic quantity associated with model; instead, it is measure of information that arises from interactions between the model and
3

---

Published as a conference paper at ICLR 2021
its queries. In our context, the interactions are through X (offered by Bob) and Y (offered by Alice). The key idea of enhancing Alice's 's model privacy is to let Alice output noisy predictions Y for any input X so that Bob cannot casily infer Alice's original model. Similarly, Alice may choose to manipulate X well before passing it through K,. Alternatively, Alice intends to 1) impose some ambiguity between X, X, and between Y, Y,, which conceivably will produce response deviating from the original one, and 2) seek the K closest to K. under the same amount of ambiguity imposed. Motivated by the above concepts, introduce the following notion.
Definition 2 (Information-laundeeed model) A information-laundeee model with respect to given model K, is model K that consists of three internal kernels K = K1 o K, a K2 (illus- trated in Figure 1).
Naturally, the informatioon-laundeee model of Alice is denoted by pK.
2.2 NOTATION
We let Pk_ (- -), (1).). Pk, 1.). P.(( denote the kernels that represent the authentic model, input kernel, output kernel, and the iiformattionllunddeee model, respectively, We let (-) denote the marginal distribution of X. Similar notation is for p& (-), ppppp)).. and py(-). Note that the py implicitly depends on the above conditional distributions. We use oK, (- an.pp.... I.) to denote cascade conditional distributions Y X and Y X, respectively.
Throughout the paper, random variables are denoted by capital letters. Suppose that X E X E L, Y E V, and Y E V. For technical convenience, we will assume that X, X, ", are finite alphabets unless otherwise stated. We will discuss some special cases when some of them are the same. Our theoretical results apply to continuous alphabets as well under suitable conditions. For notational convenience, we write the sum ΣzEx u(x) Σ. u(x) for any function u.
With a slight abuse of notation, we will use to denote distribution, density function, or transition kernel, depending on the context.
INFORMATION LAUNDERING
3.1 THE INFORMATION LAUNDERING PRINCIPLEE
The information laundering method is an optimization problem formulated from the concept of Kl.-divergence between the (designed) effective kernel and the original kernel, with constraints of the privacy leakage during the model-data interaction. In particular, we propose to minimize the following objective function over (P.,,KK.).
[PK,, (ipK,PKK, .K2)) _ Ex-px (PK (. X). (X) 8II(X; 3,1(Y;))
(1)
In the above, Ki and K₂ are implicitly involved in each additive term of I., and 31 2 0,82 0 are constants that determine the utility-privacy tradcoffs. Small values of 31 and 33 (c.g.. zeros) pushes the K to be the same as K., while large values B, pushes X to be nearly-independent with X (similarly for B2). It is worth mentioning that the principle presumes given alphabet (or representation) for X and Y. The variables to optimize over is the transition laws X + X and Y.
The objective in (1) may be interpreted in the following way. On the one hand, Alice aims to develop an effective system of K that resembles the authentic one K, for the utility of benign users. This goal is realized through the first term in (1), which is the average divergence between two system dynamics. On the other hand, Alice's 's model privacy leakage is through interactions with Bob, which in turn is through the input X (publicly offered by Bob) and output Y (publicly offered by Alice). Thus, we control the information propagated through both the input-interfaces and out-interfaces, leading to the second and third terms in (I).
We note that the above objective function may also be formulated in alternative ways from different perspectives. For example, we may change the third term be 3I | X, X), interpreted
4

---

Published as a conference paper at ICLR 2021
in the way that Alice will design K1 first, and then design K2 conditional K1. Likewise, may change the second term to be ( Y,Y), meaning that K₂ is designed first. From Bob's perspective, may also change the third term to ByI(Y; X), interpreted for the scenario where Bob conditions on the input information X during model extraction. Additionally, from the perspective of adaptive interactions between Alice and Bob, may consider px as part of the optimization and solve the max-min problem mäxpx minpxy L(pK,,KK)) We leave these alternative views for future work.
3.2 THE OPTIMAL SOLUTION
We derive the solution that corresponds to the optimal tradeoff's and point out some nice interpre- tations of the results. The derivation nontrivial as the functional involves several nonlinear terms of the variables to optimize over. Note that for the notation defined in Subsection 2.2, only px and РK. are known and others are (implicitly) determined by PK; PKa
Theorem The optimal solution (1) satisfies the following equations.
PK.oK, (Y ())
32
By 1 log (Y 18)
Pk] (o) x) Kx (T) exp
By ByIx==xpp.
B1
PK (Y |x) 31
Pr
PK. (y X) PK,c.. (U X)
pKo (y 0) Typy exp
E Xx-px
(3)
Bapy (y)
pk pK X)
where Kx and Ty are normalizing constants implicitly defined that the conditional density function integrates to one.
Note that the distributions of X, Y,Y, and Ý Y X, implicitly depend рk, and PK,- The above
theorem naturally leads to an iterative algorithm to estimate the unknown conditional distributions
(2) PK, and PKa- In particular, we may alternate Equations (2) and (3) to obtain PK, (ø x), Pk, (u Ø) from PK₁ (e-1) (i x). PKa (e-1) (y y) at step i = 1.2.... with random initial values at e = 0. The
(8)
pseudocode summarized in Algorithm 1.
In the next theorem, we show that the convergence of the algorithm. The sketch of the proof is described below. First, we treat the original objective variables, PK, h1, f2, evaluated at hą Px and h2 py. Using technique historically used to prove the convergence
another functional J of four independent
the Blahut-Arimoto algorithm for calculating rate-distortion functions in information theory, we show that J 2 L. We also show that L is convex in each variable so that the objective function is non-increasing in each alternation between four equations. Since L 2 O, the convergence is implied by the monotone convergence theorem.
Theorem Algorithm converges to a minimum that satisfies equations (2) and (3).
Note that the minimum is possibly a local minimum. We will later show the convergence to a global minimum in particular case. Next, provide interpretations of the parameters and how they affect the final solution.
A large 31 in the optimization of (1) indicates a higher weight the term I(X; X). In the extreme case when 31 oo, minimizing T(X; X) is attained when X is independent with X. Consequently, the effective model of Alice produces a fixed distribution of responses for whatever Bob queries. The above observation is in line with the derived equation (2), which will become PK, x) R x (7) (and thus K2 & 1) for large 31 0.
Similar to the effect of B1, larger 3g imposes more independence between Y and Y. In the case 32 = oo, Alice may pass the imput to her internal model K, but output random results. This can be seen from either the Formulation (1) Equation (3).
For the first expectation in equation (2), the term may be interpreted as the average likelihood ratio of y conditional on 0 against From Equation (2), it is more likely to transit from to ă in the presence of a larger likelihood ratio. This result is intuitively appealing because a large likelihood ratio indicates that may be replaced with without harming the overall likelihood of observing Y.
5

---

Published as a conference paper at ICLR 2021
Algorithm Optimized Information Laundering (OIL)
input Input distribution private model px., alphabets X, ,,, output Transition kernels pk, and PK₃
) for X, X, Y, Y, respectively.
I: Let and p denote the uniform distribution on X' and respectively.
2: for - - - do
3: Calculate
10+ (o x)) = KaP> Pki
(0)
exp
31
1
Ey
lmmpK.
Ko. (Y (3)
PK
(1) (Y 1x)
$2 E
Pko
Y
log
31 8, y,y
Mooks
py (Y)
Pk. (y) ()) (t+1)
PKiok.. (Ø |
(4 (y () = T.px (v) exp
Pk2
B
Ex-px
14+1.1 (y X)
Pk
(
1+ (8) = Σ
PKi (i (x),
(1+1)
py (v) = Σ Pk2 (y Dp, (g),
where ((+1)
PKicK. PK and Pk
(1+1) (,+1,) denote the kernels cascaded from (PK, PP..). ). (Px..Pk. and (e+1)
(PK) PP..PKk/ respectively, and p
((+1) (t)
(t+1) is the marginal from (P& .PK.,PKg (
4: for
5:
Return pk; = Pk, PK2 PK2
(T)
(T)
3.3 FURTHER DISCUSSIONS ON RELATED WORK
Information Bottleneck: extracting instead of privatizing information. The information bottle- neck method (Tishby et al., 2000) is an information--hheortiicc approach that aims to find a parsi- monious representation of raw data X, denoted by X, that contains the maximal information of a variable Y of interest. The method has been applied to various learning problems such as clustering, dimension reduction, and theoretical interpretations for deep neural networks (Tishby & Zaslavsky, 2015). Formally, the information bottleneck method assumes the Markov chain
X x X +Y,
(4)
and seeks the the optimal transition law from X to X by minimizing the functional
L(xx)x) (xxxx) X) (XX;
with 3 being is tuning parameter that controls the tradeoffs between compression rate (the first term) and amount of meaningful information (second term). The alphabet of the above X needs to be pre-selected and often much smaller in size compared the alphabet of X to meet the purpose of compression. In other words, the information that X provides about Y is passed through 'bottleneck' formed by the parsimonious alphabet of X.
similarity between the information bottleneck method and the particular case of information laun- dering in Subsection A.2 is that they both optimize a functional of the transition law of - X. Nevertheless, their objective and formulation are fundamentally different. First, the objective of information bottleneck is to compress the representation while preserving meaningful information, under the assumption of (4); Our goal is to distort X while minimizing the gap between the (random) functionality of X Y. under a different Markov chain X + X +Y
Data Privacy and Information Privacy: protecting data instead model. The tradeoffs be- tween individual-level data privacy and population-level learning utility have motivated active search on what is generally referred to as 'local data privacy' across multiple fields such as data mining (Evfimievski et al., 2003), security (Kasiviswanathan et al., 2011), statistics (Duchi et al., 2018), and information theory (du Pin Calmon & Fawaz, 2012; Sun et al., 2016). For example, a popular framework is the local differential privacy (Eyfimievski et al., 2003; Dwork et al., 2006;

---

Published as a conference paper at ICLR 2021
Kasiviswanathan et al., 2011), where raw data X is suitably randomized (often by adding Laplace noises) into Y so that the ratio of conditional densities
PY Prix (y 21)
S e
(5)
PY (y
(22)
for any V, 22 E X, where o > 0 is pre-determined value that quantities the level of privacy. In the above, X and Y represent the private data and the processed data to be collected or publicly distributed. The requirement (5) guarantees that the KL-divergence between py|z, and [y|e, is universally upper-bounded by a known function of a (see, e.g., Duchi et al., 2018), meaning that Ti and 22 are barely distinguishable from the observed y. Note that the above comparison is made between two conditional distributions, while the comparison in information laundering (recall the first term in (I)) is made between two transition kernels.
The local differential privacy framework does not need to specify probability space for X, since the notion of data privacy is only built on conditional distributions. Another related framework is the information privacy (du Pin Calmon & Fawaz, 2012), which assumes a probabilistic structure on X and a Markov chain X + Y + Y. In the above chain, X is the private raw data, Y is a set of measurement points to transmit or publicize, and Y is a distortion of Ý that is eventually collected or publicized. We deliberately chose the above notation of X, X,Y,Y. so that the Markov chain appears similar the special case of information laundering in Subsection 4. Nevertheless, the objective of information privacy is to minimize 1 [(X; Y) over pyiı subject to utility constraints,
assuming that the joint distribution of X, Y is known. In other words, the goal is to maximally hide the information of X. In the context of information laundering, the system input X is provided by users and is known.
Adversarial Model Attack: rendering harm instead of utility to a model. The adversarial model attack literature concerns the adversarial use of specially crafted input data to cause a machine learning model, often a deep neural network, to malfunction (Papernot et al., 2016a; Narodytska & Kasiviswanathan, 2017; Papernot et al., 2017). For example, an adversarial may inject noise into an image so that a well-trained classifier produces an unexpected output, even if the noise is perceptually close to the original one. A standard attack is the so-called (Adaptive) Black-Box Attack against classifiers hosted by a model owner, e.g., Amazon and Google (Rosenberg et al., 2017; Chakraborty et al., 2018). For target model K., a black-box adversary has no information about the training process of K, but can access the target model through query-response interfaces. The adversary issues (adaptive) queries and record the returned labels to train a local surrogate model. The surrogate model is then used to craft adversarial samples to maximize the target model's s prediction error.
If we let X, Y denote the model input, adversarially perturbed input, and output, respectively, then we may draw a similarity between adversarial model attack and the particular case of informa- tion laundering in Subsection A.2 since they both look for the law X + X. The main difference is in the objective. While the model attack aims to find input domain that maximally distorts the model, information laundering aims to maintain a small model discrepancy. Under our notation, a possible formulation for the model attack is to seek maxp Ex-px Dxu (PK.( (: X), (- X)) under a constraint on PX|X
4 SPECIAL CASE: INFORMATION LAUNDERING OF THE OUTPUT (Y) ONLY
Two special cases of an Citormation-lannderre system are illustrated in Figure 2. Here, we elaborate on one case and include the other special case in the Appendix. Suppose that K₁ is an identity map and let 81 O. In other words, we alter the output data only (Figure 2b). Furthermore, suppose that for each given ă, the conditional distribution px Ø) assigns all the mass at ÿ. In other words, K. reduces to a deterministic function mapping from each ă E X to unique ÿ Y, which is denoted by ÿ = f(Ø). For example, Alice's model is a classifier that takes input features and returns hard-thresholded classification labels. Then the optimization problem (1) reduces to minimizing
L(pks) Exxpx (KK (. X)) (X)) + ())(;
(6)
7

---

Published as a conference paper at ICLR 2021
(a)
(b)
Query
Pseudo-quee
Response
Query
Pseudo Pseudo-respone response
Response
K1
K2
K
Figure 2: Illustration of Alice's information-laundentt system for public use, by (a) alternating input only, and (b) alternating output only. The notations are similar to those Figure 1.
Corollary The solution to the optimization problem (6) satisfies
PK. (v X) PK. (8 X
PK, (y () -T:PY(y) exp
Ex~ex
B2pg (7) where Ty is normalizing constant. In particular, if K2 is deterministic, equation (7) becomes
pk X)
PKg - Typ exp
E px(x)
1
(2pp(() (T)
PK x
m=f(n))
Ly=8
= (g) exp
B2 v)
To exemplify the proposed methodology, we study a specific case with the following conditions.
1) X' may be large or continuously-valued. - = y is a moderately- alphabet,
2) 5 = so that and Y are in the same space,
3) K, is deterministie.
Under the above scenario, we can apply Algorithm and Corollary to obtain a simplified procedure below (denoted by OIL-Y). At each time step
......... for each 7, y E V, we calculate
(++1) (y ū) =Tpy (v) exp
1 1y=ū
where TØ T Σ
Py (v) exp
1 y
B2 Pk, (y (8)
B2 Pk
(t- (w) Kg By
(++1) (y) 7), where ry
Σ Px()).
Note that the above ry is the probability that Alice observes output of K,* if Bob inputs X E px. Therefore, rg can be easily estimated to be the empirical frequency of observing at the end of Alice.
Note that since y' is a finite alphabet, we can use a matrix representation for easy implementation. In particular, we represent the elements of y by 1...., a, where card We then represent PK2 by P E Raxa and py by q E Ra where Pg.g = Px, (y Ù). Such representation will lead to a matrix form the above procedure, summarized in Algorithm
Algorithm OIL-Y (a special case of Algorithm 1, in the matrix form)
input Input distribution px- private model pk.
output Transition kernels PK, ) x y [0, 1] represented by P Raxe where card()))
I: Estimate [r....., Tul from px and pk. as in equation (9)
2: Initialize the entries of (()) and g (respectively representing PKg PY) to be 1/a
3: for 0 T' do
Caleulate 1(++] (+)) x 1" diig()) where [l,..., 1] denote a vector.
5: Update diag( ((te))))+))))))))))))))))))())))
diag(P(t- (pptt=))-exxp[111 [1/(8z eepp diag(P())) where the operations element-wiss
6: Scale cach column (conditional distribution) of p(++1) so that it sums to one
7: Calculate q (++1) p((+))
=(t+)) x
8: end for
Return Pk2 that is represented by P(T).
Moreover, proved the convergence to the global minimum for the alternating equations in the above scenario. The same technique can be emulated to show a similar result when we employ Ki (instead of K2) only. The result is summarized in Theorem 3.
8

---

Published as a conference paper at ICLR 2021
Theorem Suppose that K, is deterministic. The alternating equation (9), or its matrix form in Algorithm 1, converges to a global minimum of the problem (6).
5 CONCLUSION AND FURTHER REMARKS
Despite extensive studies data privacy, little has been studied for enhancing model privacy. Mo- tivated by the emerging concern of model privacy from the perspective of machine learning service providers, we develop a plug-and-play methodology "information laundering" to enhance the pri- vacy of any given model of interest. The information laundering is model-agnostic as it applies to general API models, including classification and regression models that output aabels/robaabilities,, and black-box models generating probabilistic outputs. We believe that the developed principles, theories, and insights can lead to new resilient machine learning algorithms and services.
An interesting problem is to integrate information laundering with various application scenarios on case-by-case basis. Another problem to adapt the developed principle to specific API models to incorporate side information (also mentioned in Section 3.1). Taking into account the potential side information available to an adversary can be essential in some pathetic situations. Consider an example where the adversary knows that the output associated with extremely large input is fixed constant. Then, the adversary may strategically send the same input with extreme values to accurately identify that constant. An information-lauee laundered model may be vulnerable in the above scenario since the current information laundering concerns the average over data distributions.
Theoretically, there are three open problems left from the work that deserves further research. First, how does the imposed constraint of mutual information affect the rate of convergence from the adversary perspective for specific models (e.g., generalized linear models, decision trees, neural networks)? Second, we focused on finite alphabets for technical convenience. How to emulate the current methods for continuously-vain would be the relative importance of laundering X versus Y, and will this depend on specific learning problems?
alphabets (especially with large dimensions)? Third, what
Appendix. In Appendices A.1 and A.2, we first include two particular cases of information laun- dering that were not included in the main part of the paper. We then include the proofs of the theorems in Appendix A.3. Experimental results are included in Appendices A.4, A.5, A.6, and A.7 demonstrate the algorithm convergence, model privacy-utility tradeoffs, how tradeoff parameters and unbalanced samples may influence the optimized information laundering, and how the informa- tion laundering effectively mitigates adversarial attacks.
ACKNOWLEDGMENTS
The last author was supported by the Army Research Office (ARO) under grant number W91INF. 20-1-0222.
REFERENCES
Mohssen M Alabbadi. Mobile learning (mlearning) based on cloud computing: mlearning as a service (mlaas). In UbiComp, 2011.
Anirban Chakraborty, Manaar Alam, Vishal Dey, Anupam Chattopadhyay, and Debdeep Mukhopad- hyay. Adversarial attacks and defences: A survey, arXiv preprint arXiv: 18 18/0000069 2018.
Varun Chandrasekaran, Kamalika Chaudhuri, Irene Giacomelli, Somesh Jha, and Songbai Yan.
Model extraction and active learning. arXiv preprint arXiv:1811.0200 2018.
David Chaum, Claude Crépeau, and Ivan Damgard. Multiparty unconditionally secure protocols. In Proc. STOC, pp. 11-19, 1988.
Thomas M Cover. Elements of information theory. John Wiley & Sons, 1999.
Maria S Cross and Andrea Cavallaro. Privacy as a feature for body-worn cameras [in the spotlight]. IEEE Signal Processing Magazine, 37(4):145-149 2020.

---

Published as a conference paper at ICLR 2021
Enmao Diao, Jie Ding, and Vahid Tarokh. HeteroFL: Computation and communication efficient federated learning for heterogeneous clients. arxiv preprint arxiv:2010.01264, 2020.
Jie Ding and Bangjun Ding. "To tell you the truth' by cnterval-private data. In Proc. IEEE BigData, 2020.
Flavio du Pin Calmon and Nadia Fawaz. Privacy against statistical inference. Proc. Allerton Conf. Commun., Control and Computing, pp. 1401-1408, 2012.
John Duchi, Michael Jordan, and Martin Wainwright. Minimax optimal procedures for locally private estimation, J. Am. Stat. Assoc., 113(521):182-211 2018.
Cynthia Dwork. Differential privacy. Encyclopedia Cryptography and Security, pp. 338-340, 2011.
Cynthia Dwork and Kobbi Nissim. Privacy-preserving datamining on vertically partitioned databases. In Proc. CRYPTO, pp. 528-544. Springer, 2004.
Cynthia Dwork, Frank McSherry, Kobbi Nissim, and Adam Smith. Calibrating noise to sensitivity in private data analysis. In Theory of cryptography conference, pp. 265-284. Springer, 2006.
Nicholas Evans, Sebastien Marcel, Arun Ross, and Andrew Beng Jin Teoh. Biometrics security and privacy protection. IEEE Signal Processing Magazine, 32(5):17-18, 2015.
Alexandre Evfimievski, Johannes Gehrke, and Ramakrishnan Srikant. Limiting privacy breaches in privacy preserving data mining. In Proc. SIGMOD/PODSO3, pp. 211-222, 2003.
Facebook.
Communicating about privacy: Towards people-centered and account-
able design. https://about: fb. comm wp content//upooas///00000/ rrtjacyyy-aaapprreeess--ttttt-pp pdf,, July 2020.
Google.
Google security whitepaper. /top:::///eriiessss https//eeeiieeoglllllees/eee google. com cow/ff/filef/////////////
Jan 2019.
Mika Juuti, Sehastian Szyller, Samuel Marchal, and N Asokan. Prada: protecting against dnn model stealing attacks. In Proc. IEEE EumS&& pp. 512-527. IEEE, 2019.
Kaggle. Life expectancy dataset. https: / //tinurri//rr http:///wwo.lccoo///r/aaa....pp 2020. 2020.
Shiva Prasad Kasiviswanathan, Homin K. Lee, Kobbi Nissim, Sofya Raskhodnikova, and Adam Smith. What can we learn privately? SIAM J. Comput., 40(3):793-826, 2011.
Manish Kesarwani, Bhaskar Mukhoty, Vijay Arya, and Sameep Mehta. Model extraction warning in mlaas paradigm. Proc. ACSAC, pp. 371-380, 2018.
Jakub Koneveny, Brendan McMahan, Felix X Yu, Peter Richtárik, Ananda Theertha Suresh, and Dave Bacon. Federated learning: Strategies for improving communication efficiency. arXiv preprint urXiv:1610.054p0) 2016.
Taesung Lee, Benjamin Edwards, Ian Molloy, and Dong Su. Defending against machine learning model stealing attacks using deceptive perturbations. arXiv preprint arXiv:1806.0005: 2018.
Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. Communication-effiieee learning of deep networks from decentralized data. In Proc. AISTATS, pp. 1273-1288 PMLR, 2017.
Smitha Milli, Ludwig Schmidt, Anca D Dragan, and Moritz Hardt. Model reconstruction from model explanations. In Proc. ACM FAceT, pp. 1 1-9,2019
Nina Narodytska and Shiva Kasiviswanathan. Simple black-box adversarial attacks on deep neural networks. In Proc. CVPRW, pp. 1310-1318. IEBE, 2017.
Tribhuvanesh Orekondy, Bernt Schiele, and Mario Fritz. Prediction poisoning: Towards defenses against dnn model stealing attacks. In Proc. ICLR, 2019.
10

---

Published as a conference paper at ICLR 2021
Nicolas Papernot, Patrick McDaniel, and lan Goodfellow. Transferability in machine learning: from phenomena to black-box attacks using adversarial samples. arXiv preprint arXiv:1605.07277,, 2016a.
Nicolas Papernot, Patrick McDaniel, Ian Goodfellow, Somesh Jha, Z Berkay Celik, and Ananthram Swami. Practical black-box attacks against deep learning systems using adversarial examples. arXiv preprint arXi:: 1(2)3, 2016b.
Nicolas Papernot, Patrick McDaniel, lan Goodfellow, Somesh Jha, Z Berkay Celik, and Ananthram Swami. Practical black-box attacks against machine learning. In Proc. ASIA CCS, pp. 506-519, 2017.
Anand Rajaraman and Jeffrey David Ullman. Mining of massive datasets. Cambridge University
Press, 2011.
Mauro Ribeiro, Katarina Grolinger, and Miriam AM Capretz. Mlaas: Machine learning as a service. In ICMLA, pp. 896-902. IEEE, 2015.
Ishai Rosenberg, Asaf Shabtai, Lior Rokach, and Yuval Elovici. Generic black-box end-to-end attack against mns and other API calls based malware classifiers arXiv preprint 17X,v:7777.057000
2017.
Scikit-learn. The R2 score. https://ttnnyuulccom//moo
2020a.
Scikit-learn. Breast cancer wisconsin dataset.
https://sc.kkt--eam.orr//////// modules/ hatrrettce//llaannccttttsseseeer
2020b.
Scikit-leam. Moons dataset.
generateed/skeeam. datasets https///eckkt-lleem.rrrttttlllet
2020c.
Scikit-learn. The 20 newsgroups text dataset. https: pttpps://tinuzzl. com.y2mmmdv..,.....mmmmmmmmmmm 2020d.
Reza Shokri and Vitaly Shmatikov. Privacy-preserving deep learning. In Proc. CCS, pp. 1310-1321.
ACM, 2015.
Reza Shokri, Marco Stronati, Congzheng Song, and Vitaly Shmatikov. Membership inference tacks against machine learning models. In IEEE Symp. Sec. Priv., pp. 3-18. IEEE, 2017.
Meng Sun, Wee Peng Tay, and Xin He. Towards information privacy for the internet of things. arXiv preprint IrXiv:I611.0422 2016.
Naftali Tishby and Noga Zaslavsky. Deep learning and the information bottleneck principle. In Proc. ITW, pp. 1-5. IEEE, 2015.
Naftali Tishby, Fernando Pereira, and William Bialek. The information bottleneck method. arXiv preprint physics/00044057, 2000.
Florian Tramèr, Fan Zhang, Ari Juels, Michael K Reiter, and Thomas Ristenpart. Stealing machine learning models via prediction apis. In USENIX, pp. 601-618, 2016.
Paul Voigt and Axel Von dem Bussche. The EU general data protection regulation (GDPR). A Practical Guide, 1st Ed., Cham: Springer International Publishing, 2017.
Xun Xian, Xinran Wang, Jie Ding, and Reza Ghanadan. Assisted learning: framework for multi- organization learning. NeurIPS 2020 (spotlight), 2020.
Qiang Yang, Yang Liu, Tianjian Chen, and Yongxin Tong. Federated machine learning: Concept and applications. ACM Trans. Intell. Syst. Technol., 10(2):119,, 2019.
Andrew Yao. Protocols for secure computations. In Proc. SFCS, pp. 160-164. IEEE, 1982.
11

---

Published as a conference paper at ICLR 2021
A APPENDIX
A.1 SPECIAL CASES: DETERMINISTIC MODEL Kx
In this case, Theorem implies the following corollary. We will use this result in later sections.
Corollary 2 The optimal solution of (1) satisfies the following equations.
Px,(f(z) ((t))
Pk. (f xx) KxPx (T)
exp
31
(f(((()) ((')))K (*' x))
B2
PK, (f
E (((=)
log
31
PY
PKioK. (ú x)))
PK,(yl (0) Typy exp
E px
Bepy (V)
PK (
where Ka and Ty are normalizing constants implicitly defined so that the conditional density function integrates to one.
A.2 INFORMATION LAUNDERING OF THE INPUT (X) ONLY
Suppose that K2 an identity map and let B2 so that we only maneuver the input data (Fig- ure 2a). Then the optimization problem (1) reduces to minimizing
L(K.) e Ex-px (PK (-1 x).Px( x)) + (XX(
(10)
Corollary The optimal solution of (10) satisfies the following equations.
PK. (Y X = 8) Ey By|x=x<px Px X x)) S
Pk, a K (
exp
(11)
B1
where Kg is an implicitly defined normalizing constant. In particular, if K, is deterministic, equation (1I) becomes
1 f(x)=((()
PK. (T x)) KaPx
exp
(12)
31 Eat PK, ( x)
As we can see from Corollaries and 3, for a deterministic (represented by f), the simplified equation of (8) is similar to that of (12). The subtle difference that one has sum while the other does not is because f may not one-to-one mapping.
A.3 PROOFS
Proof (Proof of Theorem 1) Introducing Lagrange multipliers, A1(x) for the normalization of the conditional distributions PK; (- x)) each x, \z) for the normalization of the conditional distributions (K 8) at each ÿ. The Lagrangian of (1) can be written as
PK. (5 x)
L= Σ
(xx(x)px (y ( log (y x) +.3, Σ
(Ø)
x)) log
px (i)
F.M
2.7
PKo (y 1)
32 Σ Py (ū)P log
+ Σ
\1
( ②) + Σ
10) +
py
A A2 Ag + As
As c
(13)
up additive constant that is determined by the known px and pk.-
12

---

Published as a conference paper at ICLR 2021
It can be verified that
Opk (y 2)
PK.CKs 8)
(14)
PK, (Ø x))
Öp (i)
= -xx(x)
(15)
PK, (i |x)
Opy (i)
-xx(x))x (8 a)
(16)
PK, (i |x)
Opy (v)
px (xxxx))X_ooa) 8).
(17)
PKi (I x)
Using (14)-(17), for given and i, we calculate the derivatives of each term in (13) with respect to PK, (i 2) to be
0A1
PK. KK (y x))
PK. PK. x)) (x)
Σ
(18)
PK. (i x)
pK (y x)
y
AA2 = -Bixx (2))
PR. (5 x)
log
(19)
PK, (+ xx)
Px (i)
0Ag
PK. (v 9)
B2px (x)
Σ
PK.oK, (Ø. v X = I) log
PK₁ (f (x)
pv()))
D.Y
Epr(o)pss (y 0) PK. Kg (y (t)
-xxx(x) (x)
Σ
py(1))
8.8
PKa (y 1) pr (V)
= =2pxx (x)
Σ PKK.oK,0. 8 2) log
B2px (x)
(20)
0A4 (x))
(21)
РK, (+ x))
AA5 0
(22)
PK, (i [x)
Taking equations (18)-(2)) into (13), obtain the first-order equation
0L
Bylx= Pk.aK, (Y X = 5)
PKi (7 x)
=Px(x)
B1 log
Öpk, (ō xx)
PY X= x)
Px (x
+ By log PY, 18) { =0,
(23)
PY
where i(x) = (x(xx//xx(x)) - B2. Reurranging the terms in Equation (23), we obtain
PK. (i x) 1 -x( Eyxx=
PK.oKa (Y X =
32By x1x== log PKa PK,(Y 18)
(Y
log
(+) B1
PY X x)
PY (Y)
which implies Equation (2).
Similarly, taking derivatives with respect to pK₂ (y 17) for given ūj and y, it can be verified that
0pk (y x) = PK.cK. (ū x)
Opk, (y) 5))
ØL
Σ P (y x)) PKicK. (7 x)
PK. (y 9)
+ 32pp (0) log
+ >²
(y (8)
(x x))
Py (v(
Pk. (y) X) PK,o.. (Ú X) + Bapy (0) log PK₃ (v ()) py (3)
-Exxppx
+ >z (24)
Pk X)
Letting Equation (24) be and rearranging obtain Equation (3).
13

---

Published as a conference paper at ICLR 2021
Proof 2 (Proof of Theorem 2) We PK, PKay h1, h2.
define
the
following functional of four variables:
J(K,, PKg h1, h2)
Σ px (x PK. (y x) log px x))
(25)
x.y
px(x)pK (7 ( 1) log PK, (5 x)
+ 31
Σ
h1 (T)
Σ py (ū)pxx (v 1)) log PK₃ (y (5)
+
82
(26)
h2(y)
We will use the following known result (Cover, 1999, Lemma 10.8.1) 1). Suppose that X and Y have joint distribution with density pxy, and the marginal densities are px Py, respectively. Then a den- sity function ry of y that minimizes the KL-divergence D(pxy, pxrry is the marginal distribution py- This result implies that minimizing the objective function (1) can be written as quadruple minimization
min PK Js.
JPK. PK2 11,111)) 1t2).
(27)
can be from (23) and its preceding identities that 02.J
PK.oKa (Y ()) Opk (Y x))
Px
px By
+ 31
Öpk, (o 2)²
PY x)² 0pk, ( (x)
PK, ( xx)
Pk. K2 (Y |3)²
PK (Y |x)² 1 px(x)
= px X TOPK
(28)
PK, (ø x))
0J
PK. (y X)) PK,oKK (ū)
()) Opk (y) X)
Py (0)
Ex-px
+ +3
(29)
OpKs(v (v D)2
PK X)
Opk, (y ())
PKa 9)
PK. (y (X) РK₁c,, (ū) |x2
+ B2 py
py(1) (ī)
Exapx
(30)
PK(y) (3/ X)2
Pka (y l
02J
(x(x) (7 |x)
B1 Σ
(31)
0h, (8)²
h1 (
m
JJJ
Py (ū)PK, (y D)
B2
(32)
022
h22y)²
Thus, J[K PK,,, PK2, hn h2) is convex in each of the variables.
We begin with a choice of initial pK,, h1, b2, and calculate the Pk, that minimizes the objective. Using the method of Lagrange multipliers for this minimization (in way similar to (13)), obtain the solution of pk, shown in the first equation of Line 3, Algorithm 1. Similarly, we obtain the second equation in Algorithm 1. For the conditional distributions pK, and pK2 we then calculate the marginal distributions hy (of X)) that minimizes (26). Note that the terms of (26) involving hı may be rewritten as
31 Ep(a, log p(2,
(T)
which, by the aforementioned lemma, is minimized by the third equation of Line 3, Algorithm I. Similar arguments apply for h2z Consequently, each iteration step in Algorithm reduces J. By the non-negativeness of Kl-divergence, J + > L, 2 O, where L is in (1) and is introduced in (13). Therefore, J has a lower bound, and the algorithm will converge to a minimum. Note that J(K,, PK2, h1, h2) is convex in each of the variables independently but not in the variables' product space. The current proof does not imply the convergence to a global minimum.
Proof 3 (Proof of Theorem 3) Similar the technique used in the above proof of Theorem we cast the optimization problem in (6) as a double minimization with respect to (PKa: lz2).
JPK,, 122) A Px (y)
logpk (y x) + B2 Σ py (T)P..y ( log PK, (y 7) U.
hz(4)
2.9
14

---

Published as a conference paper at ICLR 2021
We only need check that J is strongly convex in its arguments. Direct calculations show that
82
PK.
(ū
x))
py (T)
px (x))) (y
x)
82
0pk,(U (u 5)
PK
x))
PK2 (y (g)
py Pv
E
Px (z)
+ 13
P (y x)
PKa (y
m: (
Py Py(ū)
Py (ï) PKa (y 1
11 y== B2
Pk. (y (1)
8J
82 PY (v)
0hg (v)²
122 (v)²
82 L
B2 py (0)
0pk, (y y)
h2(v))
The above equations indicate that the determinant of the Hessian satisfies
J2
JJJ
{
8
Öpk, 0) ahz(y)?
@pkx( (/] 1))h2
py (0)py (V)
py (8)pY (v)
1 py. Pg (ū.1) py (y)
= = PK 0))2(y)² Lu=8 + $2 PK, h2)
which further further
implies the convexity of J in the product space of PK2 PK2, and l22.
A.4 VISUALIZATION OF ALGORITHM 2
We provide a toy example to visualize Algorithm 2. In the simulation, we choose an alphabet of size 100, and py as described by r E [0, 1]ah is uniform-rande randomly generated from the probability simplex. We independently replicate the experiment 50 times, each time running Algorithm 2 for 30 iterations, and calculate the average of the following results. First, we record p(t+1) p(() I1/a at cach iteration f, which traces the convergence of the estimated transition probabilities. Second, we record the final transition probability matrix into a heat-map where Py. means the estimated PK, (y 5). The experiments are performed for 3 100, 10, 1, corresponding columns 1-3. The plots indicate the convergence of the algorithm, though the rate of convergence depends on B. They also imply the expected result that small induces identity transition while large 3 induces y that nearly independent with Y.
A.5 DATA STUDY: NEWS TEXT CLASSIFICATION
the '20 '20-newsgroups' dataset provided by seikit-learn open- source library (Scikit-leam, 2020d), which comprises news texts on various topics. The ex- periment is intended to illustrate the urility-privacy tradeoff and the optimality of our proposed solution compared with other methods. For better visualization we pick up the first four top- ies (in alphabetic order), which are "alt.atheism', (comp.graphicss ccomp.sy.ibb..p. pe..hrrware'. Suppose that the service Alice provides is to perform text-based clus- tering, which takes text data as input and returns one of the four categories (denoted by 0, 1,2,3) as output. The texts are transformed into vectors of numerical values using the technique of term reequency-ivverss document frequency (TF-IDF) (Rajaraman & Ullman, 2011). In the transforma- tion, metadata such as headers, signature blocks, and quotation blocks are removed. To evaluate the out-sample utility, we split the data into two parts using the default option provided in (Scikit-learn, 2020d), which results in training part (2245 samples, 49914 features) and a testing part (1494 samples, 49914 features). The above split between the training and testing is based upon messages posted before and after specific date.
In this experimental study,
'comp.os counp.ss.mswwnndww. mns-windows.mise
Alice trains a classifier using the Naive Bayes method and records the frequency of observing each category (0.220.2700 270.200..20.3 (r in Algorithm 2). Then, Alice runs the OIL-Y Algorithm (under given 52) to obtain the transition probability matrix P E [0, 1]4x4, In other words, the effective system provided by Alice is the cascade of the learned classifier, and P determines the Markov
15

---

Published as a conference paper at ICLR 2021
(a1)
(b1)
(c1)
-1.5
0-
0.0
-1.6
-2.5
-1.7
Error (in log scale)
Eroor (in log scale)
-2
-5.0
-1.8
Error (in log
-3
-7.5
-2.0
-4-
-10.0
-2.1
-5-
1225
-2.2
-6
-15.0
-2.3
-7
-17.5
10
20
30
10
20
30
10
20
30
Iteration
Iteration
Iteration
(a2)
(b2)
(c2)
1.0
1.0
1.0
20
0.8
20-
0.8
20-
0.8
40-
0.6
40-
0.6
60-
0.4
0.4
60-
0.4
80-
0,2 80-
0.2
80-
0.2
20
60
80
20
40
80
20
60
80
Figure 3: Visualization of Algorithm 2 in terms of the convergence (row 1) and the final transition probabilities (row 2), for 3 = 100, 10, 1 (corresponding to three columns).
transition. Alice's resulting out-sample performance from the testing data is recorded in Figure 4a, where we considered different 3e's summarized in Table 1. As we expected, larger value of 82 cuts off more information propagated from y to Y, resulting in a degraded out-sample performance of Alice's effective system.
We also visualize the model privacy-utility tradeoff by the following procedure. First, we approxi- mate the utility that quantifies the useful information conveyed by Alice. With Alice's 's trained model and the optimally laundered Y (from training data), we retrain another Naive Bayes classifier and generate predictions on the testing data, denoted by yk Meanwhile, we apply Alice's ' authentic model to generate predictions on the testing data, denoted by Уи. We approximate the model util- ity as the accuracy measure between yk and yx The model utility can be approximated by other measures. We also considered retraining methods such tree-based classifiers and average F1- score in computing the model utility, and the results are consistent in the data experiments. Second, approximate the privacy leakage Alice's prediction accuracy the testing data. Intuitively speaking, for a given utility, larger out-sample prediction accuracy indicates less information laun- dered, indicating higher privacy leakage Alice's internal model. We plot the model leakage against utility obtained from our proposed solution in Figure 4b.
For comparison, we considered a benchmark method described below. The conditional probability mass function Px,(- (- Ø) given each ğ is independently drawn from Dirichlet distribution with parameters [b. .... b, a, .......b, b), where a the ýith entry. An interpretation of the parameter that larger a/b favors larger probability mass at y (and thus less noise). We consider different pairs of (a, b) so that the tradeoff curve matches the counterpart curve from our proposed method. The curve is averaged over 50 independent replications. As shown in Figure 4b, the results indicate that our proposed solution produces less leakage (and thus better privacy) for a given utility.
We also plot heatmaps illustrating the transition laws Pk2 (y D)) obtained from the proposed infor- mation laundering in Figure 5. We considered two cases, where there are 20% class-O labels, and where there are 1% class-O labels (by removing related samples from the original dataset). Intu- itively, once we reduce the size class-O data in (b), the transition probabilities Px, (0 Ø) for each ÿ should be smaller compared with those in (a) class-O is no longer 'important'. Our expectation
16

---

Published as a conference paper at ICLR 2021
Table 1: Summary of the tradeoff parameters used for the OIL.-Y algorithm and random bench- mark from Dirichlet distributions (averaged over 50 independent replications), and the correspond- ing model utility (as evaluated by the closeness of Alice's authentic and effective systems), well as the model privacy leakage (as evaluated by Alice's out-sample accuracy).
B
20
50
Proposed
Utility
1.00
0.86
0.78
0.68
0.46
0.30
Leakage
0.79
0.64
0.53
0.45
0.35
0.30
a, b
100,
20,
10, 1
5,
5,3
10, 10
Random
Utility
0.96
0.88
0.79
0.49
0.39
0.23
Benchmark
Leakage
0.77
0.70
0.62
0.40
0.34
0.27
(a) Performance curve (of Alice)
(b) Tradeoff curve
0.9
0.8
Proposed
0.8
0.7
Random
Model leakage
Accuracy
0.6
0.6
0.4
0.3
0.3
0.2
10
20
0.6
0.8
1.0
B2
Utility
Figure 4: Visualization of (a) Alice's out-sample performance against the tradeoff parameter B2 in Information Laundering, and (b) Alice's model utility-privacy tradeoffs under the information laundering technique and the random benchmark using Dirichlet-gen birichlet-generated transition laws. Detailed parameters summarized in Table 1.
is aligned with Figure 5, where the first row in (b) are indicated by darker colors compared with that in (a), meaning that the class-O is less likely to observed.
A.6 DATA STUDY: LIFE EXPECTANCY REGRESSION
In this experimental study, we the 'life expectancy' dataset provided by kaggle open-source data (Kaggle, 2020), originally collected from the World Health Organization (WHO). The data was collected from 193 countries from 2000 2015, and Alice's model is linear regression that predicts life expectancy using potential factors such as demographic variables. immunization factors, and mortality rates. This experiment is intended illustrate the utility-privacy tradeoff and our proposed solution in regression contexts.
In the regression model, we quantize the output alphabet y by 30 points equally-spacee in between + 3o, where represent the mean and the standard deviation of Y in the training data. We then applied similar procedure as in Subsection A.6, except that we use the empirical R² score as the underlying measure of utility and leakage. The empirical R² score has been commonly used for evaluating regression performance, and it can be negative, meaning that the predictive performance is worse than sample mean-based prediction (Scikit-learn, 2020a). In particular, obtain tradeoff curves in Figure 6, where we compared the information laundering results based on the proposed technique and Dirichlet-based technique (similar that in Subsection A.6). The different I's s and Dirichlet parameters are summarized Table 2. The detailed performance values also summarized in Table 2.
17

---

Published as a conference paper at ICLR 2021
(a) 20% class-0
(b) 1% class-O
1.0
0.8
0.8
0.6
0.6
0.4
0.4
0.0
0.0
Figure 5: Heatmap showing the transition law pKg (y ()) for information laundering, under (a) 20% of class-O labels, and (b) 1% class-O labels. In contrast with the case (a), the class-O negligible in (b) and thus the transition probabilities PK, (0 y) for each ý becomes smaller (as indicated by darker colors).
Table 2: Summary of the tradeoff parameters used for the OIL-Y algorithm and random bench- mark from Dirichlet distributions (averaged over 50 independent replications), and the correspond- ing model utility (as evaluated by the closeness of Alice's authentic and effective systems), well as the model privacy leakage (as evaluated by Alice's out-sample accuracy). The underlying metric used is the empirical R², which can be less than zero.
B
0
2
5
8
20
Proposed
Utility
0.99
0.92
0.84
0.62
0.48
0.35
Leakage
0.79
0.42
0.09
00.66
-0.45
00.51
(a, b)
10000,
200, 5
100, 5
100, 8
100, 10
100, 20
Random
Utility
0.99
0.77
0.58
0.42
0.36
0.15
Benchmark
Leakage
0.78
0.10
00.07
0.15
0..1
-0.22
To illustrate the impact of tradeoffs, we considered two cases corresponding to /3 = and 8g 20. We compute the transition laws РK,(v) y) obtained from Algorithm and illustrate them in the first row Figure 5. We also take the snapshot the year Ý 66 and plot the conditional density function PK. (- Y 69) (as approximated by the quantizers) in the second row of Figure 5. The visualized results aligned with expectation that larger penalty of model leakage will cause a more dispersed transition law.
A.7 DATA STUDY: MITIGATION OF ADVERSARIAL ATTACKS
In this experimental study, we demonstrate the use of information laundering in mitigating two model extraction attacks studied in (Tramèr al., 2016). We consider settings where the target model is classifier. The first attack is the Retraining attack. The adversary Bob sends random queries and receives class labels train local model. Bob does not need to know Alice's model architecture. The second attack is the equation-solving attack. Alice's model is assumed to be Logistic classifier, and Bob knows about it. Bob sends random queries in this attack and receives class probabilities (a vector of Alice's Logistic model. We note that there exist more types attacks suitable for various specific settings (see., e.g., Tramèr et al., 2016; Juuti et al., 2019).
simplex). Bob then solves linear equation obtain the coefficients
The experiments this section indicate the following two points. First, an adversary needs more samples to achieve the same utility under the information laundering (Figure 8). Second, the ef- fects of attack and laundering depend on the particular models specified by the adversary and target (Figure 9).
18

---

Published as a conference paper at ICLR 2021
(a) Performance curve (of Alice)
(b) Tradeoff
0.8
0.8
Proposed
0.6
0.6
Random
Model leakage
R²
0.2
0.2
0.0
0.0
-0.4
-0.4
B2
Utility
Figure 6: Visualization of (a) Alice's out-sample performance against the tradeoff parameter 8 in Information Laundering, and (b) Alice's model utiiity-privacy tradcoff's under the information laundering technique and the random benchmark using Dirichlet- srichlee-generated transition laws. Detailed parameters summarized in Table 2.
(a) B₂
(b) B2 20
40
40
60
60
60
0.6
0.6
P(Y Y 69)
PLY Y 69)
0.5
0.5
0.4
0.4
0.3
0.3
0.0
0.0
40
40
Figure 7: Heatmap (row 1) showing the transition laws optimized from information laundering.
under (a) 32 = 1, and (b) 82 20. The snapshots of probability mass functions of Y conditional on Y = 69 are also visualized (row 2).
The experimental details are given below. In Figure 8(a), Alice uses half of the Breast Cancer dataset (Scikit-leamn, 2020b) (standardized) to train a Logistic classification model. Bob queries the class labels with standard Gaussian random input and locally trains another Logistic classifier. In responding to Bob, Alice employs different levels of laundering. The utility is defined as the
19

---

Published as a conference paper at ICLR 2021
(a) Retraining attacks
(b) Equation Solving attacks
1.00
0.90
100
100
300
0.95
=00
0.85
n= 1000
0.90
-.. n= 1000
0.80
0.85
0.75
0.75
0.70
0.70
0.65
0.60
0.55
40
50
10
40
5D
Figure 8: An adversary's utility against the laundering parameter 32 in the contexts of (a) Retraining attack with random queries, and (b) Equation-Son -oliinn attack. each plot, the three curves represent query sizes = 100, 300, 1000, respectively.
(a) RandonFooooo orest against RandomForest
(b) Logistic agalinst RandomForest
1.0
n=
1.0
100
300
300
1000
... n= 1000
0.9
0.9
Utityy
0.8
Uuitly
0.8
0.7-
0.7
0.6
0.6-
0.5
0.5
10
50
40
8z
Figure 9: An adversary's utility against the laundering parameter B2 in the contexts of Retraining attacks, where (a) both the adversary and the target use the Random Forest model, and (b) the adversary uses the Logistic model while the target uses the Random Forest. In each plot, the three curves represent query sizes n 100, 300, 1000, respectively.
percentage of agreement of Alice's and Bob's models tested on the other half of the dataset. In Figure 8(b), Bob sends the same random queries and solves linear equation to estimate Alice's Logistic coefficients. From the results, we can see that Bob's performance better than that in Figure 8(a). It is mainly due to the substantial side information and the least-squares estimate of Bob.
In Figure 9, Alice used half of the simulated Moons dataset (Scikit-learn, 2020c) (with 1000 sam- ples, 0.1 standard deviation for the noise) to train Random Forest model. The other half of the data are reserved to evaluate Bob's utility. Suppose that Bob uses the above Retraining attack, but with two different models. In Figure 9a)) Bob uses the Random Forest classifier, which has the expressive power to extract Alice's 's model. In Figure 9(b), Bob uses the Logistic classifier, which has linear decision boundary. Bob's model in (b) is inadequate because the Moons data is not linearly separable. Figure 9 shows that the influence of information laundering the adversarial query size needed to maintain the same utility largely depends on Bob's model choices. Also, Bob's inadequate model may show more robustness against laundered information even though it does perform satisfactorily on cleaned data.
20


Reviews:
[{"review_id": "ltNNB3v7s42", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Paper summary:\nThis paper aims to tackle the problem of model stealing/extraction, as in stealing a model that is deployed remotely, through API access. The threat model that they are aiming to protect against is not well-defined. The proposed method is information theoretic. They propose adding two modules (kernels) before and after the main deployed model, to \"launder\" information.  The loss function for achieving the desired modules consists of two main terms, for utility, and privacy of the model. The utility term tries to keep the expected value of the output being accurate high, while the privacy term (which are actually two terms for the two modules) try to decrease the MI between the true output/input and the laundered ones. They then offer an iterative approach for minimizing the loss. \n\npros:\n+ I like how the approach is model-agnostic, you don't seem to need to change the model, you just add some kernels. And it can be applied to any model.\n\n+ The suggested approach is intuitive and sound. \n\ncons:\n- The paper is not well-situated, given prior work. I am not completely familiar with the model stealing literature either, but with a search I found numerous papers since 2016 that either propose model stealing attacks, or mitigations. The paper does not qualitatively nor quantitatively compare their approach to any other prior protection approaches, such as [1], [2] or [3].   \n\n- The paper does not conduct any ablation studies or experiments into the way their approach affects the model accuracy, and how effective it is to existing attacks, such as the Tramer et al. attack. This makes at really hard to evaluate the claims made about privacy and utility, in action.\n\n- Mutual Information and Expected Value (used in the loss function) are both average notion. I wonder what this means for the provided protection. Would the model perform really badly on some few examples? would there be some sample inputs that could extract a big part of the model? I think a study of this, or at least a discussion would be very helpful.\n\nOne final note, it would be nice to have the threat model well-defined, it would really help better communicate what the protection is supposed to do, and what we should be expecting/not expecting of it. \n\nReferences\n[1] Juuti M, Szyller S, Marchal S, Asokan N. PRADA: protecting against DNN model stealing attacks. In2019 IEEE European Symposium on Security and Privacy (EuroS&P) 2019 Jun 17 (pp. 512-527). IEEE.\n\n[2] Orekondy T, Schiele B, Fritz M. Prediction Poisoning: Towards Defenses Against DNN Model Stealing Attacks. InInternational Conference on Learning Representations 2019 Sep 25.\n\n[3] Lee T, Edwards B, Molloy I, Su D. Defending against machine learning model stealing attacks using deceptive perturbations. arXiv preprint arXiv:1806.00054. 2018 May 31.\n\n\n--------- Comments after reading the author response:\nI thank the authors for adding the experiments and applying the suggested modifications! I have updated my score based on the changes and the clarifications made on the related work, and also the results of the mounted attacks. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "dTRPE5BgxRu", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary: This paper studies model privacy and its privacy and utility tradeoff. In particular, the authors proposed information laundered model where the input and output of the true model are perturbed.  The objective is to minimize the KL divergence between the true model and laundered model with mutual information between input and output as constraints. Theoretically, they show the optimal condition of the above optimization problem and provides an iterative algorithm.\n\n\nPros: 1. Model privacy is indeed an interesting problem. This paper is well written.\n\n\nQuestions: 1. The authors claim the proposed framework and algorithm can be also applied to continuous space. However, it is not clear to me how algorithm 1 would work in continuous space. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "Dcri7I0q0zx", "reviewer": "AnonReviewer5", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Theoretically:\nThe paper is both conceptually and formally descriptive.  Combined with appendix details, the authors present a very full picture of their work, which is very much appreciated.  To the best of my knowledge, this appears to be a novel approach to the challenge of adversarial model reconstruction.\n\n\nExperimentally:\nSome experiments reside in the appendices that confirm utility of the theoretical basis of information laundering, however, I believe further experimentation would strengthen the paper significantly.\n\nThere appear to be multiple works (Tramer / Papernot / etc.) that provide different strategies for reconstructing private learning models.  A high-value experiment would be to test those methods on a model protected with the information laundering framework.\n\nAdditionally, there appear to be other techniques for safeguarding against model reconstruction efforts (some papers shared below).  A comparison between those and information laundering would help inform design decisions that practitioners/users are making about their machine-learning-as-a-service systems.  I’m left wanting to know how information laundering compares to other model-privacy protective measures.  A clear answer to that question would make the paper very strong, in my opinion.\n\n\nRelated work:\nWhile submission length is always a limiting factor, I believe it may be worth mentioning a few other efforts.\n\n‘Federated machine learning: concept and applications’ ACM 2019\nData privacy can be related to the topic of ‘Federated Learning’.  Mention in related work would provide an extra connection to the machine learning community for a security related problem setting.\n \nThere appear to be a few more model extraction techniques in literature than are mentioned in related work:\n\n“Model reconstruction from model explanations” FAT* 2019\nSwaps out prediction API for ‘explanation API’\n\n“Membership inference attacks against machine learning models” IEEE 2017.\nUnder the assumption that the adversary has access to data, infer which samples make up training data, and use this information to reconstruct their own model.\n\n\nDirectly relevant related work:\nPhilosophically speaking, there are some direct competitors that are not addressed, in the sense that they are methods to safeguard against leakage / maintain model privacy.  I only argue that these should be addressed/discussed by the authors, not that these papers invalidate the authors' novelty claims.\n\n“Model extraction warning in mlaas paradigm” ACM 2018 \n“PRADA: protecting against DNN model stealing attacks” IEEE 2019\n\nBoth papers monitor queries and raise alarms when query distributions become adversarially-suspect.  Functionally, preventing adversarial queries but allowing benign queries solves the same problem as information laundering but without reducing utility for all users.  The question of interest would be whether these alarms can be sounded before meaningful information is recovered from the model.  I suspect information laundering provides a stronger safeguard against adversarially model reconstruction efforts than these warning systems, but experimental evidence would make the case even stronger.\n\nOverall:\nThe paper reads very well.  Further experimentation would help relate information laundering to other related efforts, however, I believe the paper can stand on its own as more of a theoretical investigation rather than an empirical one.  For that reason, I believe the paper is strong enough for acceptance as-is.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This clearly written paper has been constructively evaluated by three expert reviewers who provided at least two very detailed and informative summaries. The authors have addressed the inquiries raised by the reviewers in a comprehensive fashion, and at least one reviewer has updated their score as a result of those detailed rebuttals. In spite of some outstanding limitations, including a somewhat limited view of the relation of the proposed approach to existing alternatives, the reviewers are consistent in suggesting that this work is sufficiently mature to be considered for the inclusion in the program of ICLR 2021. I concur with that and recommend accepting this paper.
Decision: Accept (Spotlight)