Marich: A Query-efficient & Online Model Extraction Attack using Public Data
Authors: Pratik Karmakar
Debabrota Basu
Venue: Submitted to ICLR 2023 (2023)
Keywords: Privacy attacks
Model extraction
Membership inference
Black-box attack
Query efficiency
Active learning

Abstract: In this paper, we study black-box model stealing attacks where the attacker is only able to query a machine learning model through publicly available APIs. Specifically, our aim is to design a black-box model stealing attack that uses a minimal number of queries to create an informative replica of the target model. First, we reduce this problem to an online variational optimization problem. At every step, the attacker solves this problem to select the most informative query that maximizes the entropy of the selected queries and simultaneously reduces the mismatch between the target and the stolen models. We propose an online and adaptive algorithm, Marich, that leverages active learning to select the queries. We instantiate efficiency of our attack against different models, including logistic regression, BERT and ResNet18, trained on different text and image datasets. Marich is able to steal a model that can achieve 70-96$\%$ of true model's accuracy using 0.8-10$\%$ samples from the attack datasets which are publicly available and different from the training datasets. Our stolen models also achieve 75-98$\%$ accuracy of membership inference and also show 70-90$\%$ agreement of membership inference with direct membership inference on the target models. Our experiments validate that Marich is query-efficient and capable of creating an informative replica of the target model.

Under review as a conference paper at ICLR 2023
MARICH: A QUERY-EFFICIENT MAX-INFORMATION MODEL EXTRACTION ATTACK USING PUBLIC DATA
Anonymous authors
Paper under double-blind review
ABSTRACT
In this paper, we study black-box model stealing attacks where the attacker is only able to query machine learning model only through publicly available APIs. Specifically our aim is to design a black- Bbxck-box model stealing attack that uses min- imal number of queries to create an informative replica of the target model. First, we reduce this problem into an online variational optimisation problem. The at- tacker solves this problem to select the most informative queries that maximise the entropy of the selected queries and simultancously reduce the mismatch between the target and the stolen models. We propose an online and adaptive algorithm, MARICH that leverages active learning to select the queries. We instantiate ef- ficiency of our attack on different text and image data sets and different models including BERT and ResNet18. Marich is able steal model that can achieve 69-96% of true model's accuracy using 1,070 6,950 samples from the attack datasets which are completely different from the training data sets. Our stolen models also achieve 85-95% accuracy of membership inference and also show 77-94% agreement of membership inference with direct membership inference on the target models. Our experiments validate that Marich is query efficient and also capable of creating an informative replica of the target model.
INTRODUCTION
In recent years, Machine Learning a Service (MLaaS) are widely deployed and used in industries. In MLaaS (Ribeiro al. 2015), an ML model trained remotely on a private dataset, deployed in Cloud, and offered for public access through prediction API, such as Amazon AWS] Google API, Microsoft Azure This API allows an user, including a potential adversary, to send queries to the MI. model and fetch corresponding predictions. Recent works have shown such models with public APIs can be stolen or extracted by designing black-box model extraction attacks (Tramèr et al. 2016). In model extraction attacks, adversary queries the target model with query dataset, which might be same or different than the private dataset, collects the corresponding predictions from the target model, and builds replica model of the target model. The goal is to construct model which is nearly-equivalent to the target model over the input space of interest (Jagielski et al.) 2020).
Often, MI. models proprietary, guarded by IP rights, and expensive build. These models might be trained on datasets which are expensive to obtain (Yang et al.) 2019) and consist of private data of individuals (Lowd & Meek 2005). Also, extracted models can be used to perform other privacy attacks on the private dataset used for training, such as membership inference (Nasr et al.) 2019). Thus, understanding susceptibility of models accessible through MLaaS presents an important co- nundrum. This motivates us to investigate black-box model extraction attacks while the adversary has no access the private data or a perturbed version of it (Papernot et al. 2017). Instead, the adversary uses public dataset to query the target model (Orekondy et al. 2019 Pal et al. 2020).
Black-box model extraction attacks pose tension between the number of queries sent to the target ML model and the accuracy of extracted model (Pal et al. (020). With more number of queries and predictions, adversary can build better replica. But querying an API too much can be expensive, as each query incurs monetary cost in MLaaS. Also, researchers have developed algorithms that can detect adversarial queries, when they are not well-crafted or sent to the API in large numbers (Juuti et al.) 2019; Pal et al.) 2021). Thus, designing query- query-efficient attack is paramount for practical deployment. Also, it indicates how more information can be leaked from a target model with less number of interactions.

---

Under review as a conference paper at ICLR 2023
Private tmiiting Conaact
Predicted label
Black box model
Y,
Individuals
(Q,Y)
Querics Q
Trai oxtrastd model
Marich
Public Query dataset
D
Send D to f
Figure 1: A schematic for black-box model extraction attack with MARICH.
In this paper, investigate effective definitions of efficiency of model extraction and corresponding algorithm design for query-efficient black-box model extraction attack with public data, which is oblivious deployed model and applicable for any datatype.
Our contributions. Our investigation yields three-fold contributions.
1. Formalism: Distribution Equivalence and Max-Information Extraction. Often, the ML models, specifically classifiers, are stochastic algorithms. They also include different elements of random- ness during training. Thus, rather than focusing terms of a fixed dataset or accuracy on that dataset (Jagielski et al.) 2020), we propose a distribu- tional notion of equivalence. We propose that if the joint distributions induced by query generating distributions and corresponding prediction distributions due to the target and extracted models are same, they will be called distributionally equivalent Sec. 4 Another proposal is reinforce the objective of the attack, i.c. to extract as much information as possible from the target model. This allows us formulate the Max-mmofoontionn attack, where the adversary aims to maximise the mu- tual information between the extracted and target models' distributions. We further show that both of these attacks can be performed by optimising a variational objective (Staines Barber 2012).
equivalence of extracted and target models in
2. Algoritim: Adaptive Query Selection for Extraction with MARICH. We propose algorithm, MARICH (Sec.5) 5 that optimises the objective of the variational optimisation problem (Equation ( Given extracted model, target model, and previous queries, MARICH adaptively selects a batch of queries enforeing this objective. Then, it sends the queries to the target model, collect the predic- tions, and use them to further train the extracted model (Algorithm I In order to select the most informative set of queries, deploys three sampling strategies sequentially. These three strategies select: a) the most informative set of queries, b) the most diverse of queries in the first selection, and c) the set queries in the first selection where the target and extracted models mismatch the most. Together these strategies allow MARICH select small subset of queries, which maximise the information leakage and align the extracted model with the target (Figure D
3. Experimental Analysis. We perform extensive experimental evaluation with both image and text datasets, and diverse model classes, such as logistic regression, ResNet18, and BERT (Sec. (). Our experimental results validate that MARICH can extract accurate and informative replicas of the target models in comparison to random sampling. While MARICH uses a small number of queries (0.83 6. 1.%%) selected from publicly available query datasets, the models extracted by lead to comparable accuracy with the target model while encountering membership inference attack. This shows that MARICH can extract alarmingly informative models query-efficiently.
2 RELATED WORKS
Here, we claborate the questions in the model extraction literature that we aim to mitigate.
Taxonomy of Model Extraction. Black-box model extraction (or model stealing or model infer- ence) attacks aim to replicate of a target ML model, commonly classifiers, deployed in a remote service and accessible through public API (Tramer et al., 2016). The replication is done in such way that the extracted model achieves one the three goals: a) accuracy close to that of the target model on the private training data used train the target model, b) maximal agreement in predic-
2

---

Under review as a conference paper at ICLR 2023
tions with the target model on the private training data, and c) maximal agreement in prediction with the target model over the whole input domain. Depending on the objective, they are called task ac- curacy, fidelity, and functional equivalence model extractions. respectively (Jagielski et al.] 2020). Here, we generalise these three approaches using a novel definition of distributional equivalence and also introduce a novel inoomatiinnthhoree objective of model extraction which maximises the mutual information between the target and the extracted model over the whole data domain.
Framework of Attack Design. Following Tramèr et al. (2016), researchers have proposed multiple attack algorithms to perform one of the three types of model extraction. The attack algorithms are based on two main approaches: direct recovery (target model specific) (Milli et al., 2019 Batina et al., 2018: Jagielski et al.) (020)) and learning (target model specific/oblivious).. The learning- based approaches can also categorised into supervised learning strategies, where the adversary has access to both the true labels of queries and the labels predicted by the target model (Tramèr) jet al. 2016 Jagielski et al. (002). and online active learning strategies, where the adversary has only access to the predicted labels of the target model, and actively select the future queries depending on the previous queries and predicted labels (Papernot et al. 2017: Pal et al. 2020; Chandrasekaran let al. 2020). As query-effcciency is paramount for adversary while attacking an API to save the budget and to keep the attack hidden and also the assumption of access true label from the private data is restrictive, we we focus on designing an online and active learning- -eaningg-uuedddd atlack strategy which can be model oblivious.
Classes of Target Model. While (Milli et al. 2019 Chandrasekkaan et al. 2020) focus per- forming attacks against linear models, all others are specific to neural networks (Milli et al. 2019 Jagielski al., 2020; Pal et al.) 2020) and even a specific architecture (Correia-Sile et al. 2018) 8). In contrast, MARICH is based on active learning methodology and also capable to attack both linear models and neural networks.
Types of Query Feedback. Learning-based attack algorithms often assume access to either the probability vector of the target model over all the predicted labels (Tramer et al. 2016 Orekondy et al. 2019: Pal et al. 2020, Jagielski et al., 2020). or the gradient of the last layer of the target neural network (Milli al. al.][ 2019] Miura et al.) [02T], which are hardly available in a public API. In contrast, following (Papernot et al.) (017). we assume access to only the predicted labels of the target model for a given set of queries, which always available through a public API.
Type of Query Dataset. The adversary needs query dataset to select the queries from and to send it to the target model to obtain predicted labels. In literature, researchers assume three types of query datsets: synthetically generated samples (Tramèr et al., 20166), adversarially perturbed private (or problem domain) dataset (Papernot et al., 2017] Juuti et al. 2019), and publicly available (or out-of-problent domain) dataset (Orekondy al.] 2019 Pal et al. 2020 As we do not want to restrict MARICH to have access to the knowledge of the private dataset or any perturbed version of it, we use publicly available datasets, which are different than the private dataset.
In brief, we propose an online and active-learning based model extraction attack using MARICH, which is model-oblivious, assumes only access the predicted label for given query through public API, and uses only publicly available non-domain data to query the target model. This is a less restrictive setup than the ones considered in literature, while the models extracted by MARICH demonstrate significant accuracy and act as informative replica of the target model to conduct per- vasive membership inference on the private dataset.
3 BACKGROUND: CLASSIFIERS, MODEL EXTRACTION, AND MEMBERSHIP INFERENCE ATTACKS
Before proceeding to the details of our contribution, we present the fundamentals of a classifier in ML, and two types of inference attacks: Model Extraction (ME) and Membership Inference (MI).
Classifiers. A classifier in ML (Goodfellow et al. 2016) is function f X y that maps set of input features X E X to an output Y E The output space is a finite set of classes, i.e. {1...., ..... k). Specifically, a classifier f is parametric function, denoted fo. with parameters 0 E Rd, and is trained on a dataset D T i.e. a collection of n tuples ((x; 9:) 1 generated IID
We represent sets/vectors by hold letters, and the corresponding distributions by calligraphic letters. We express random variables in uppercase, and an assignment of random variable lowercase.
3

---

Under review as a conference paper at ICLR 2023
from an underlying distribution D. Training implies that given a model class F = {fold 8 E )), loss function 1 y x y + R20, and training dataset D we aim to find the optimal parameter or A arg mingee Di=1 ((ff(xx), v1). We use cross-entropy, i... ((fo(x;), yi) - y: log(fe(x;)), as the loss function for classification.
Model Extraction Attack. A model extraction attack is an inference attack where an adversary aims f" trained on a private dataset D" and create another replica of it f (Tramer et al. 2016). In the black-box setting that we are interested in, the adversary can only query the target model 1 by sending queries Q through a publicly available API and to use the corresponding predictions Y to construct f The goal of the adversary is to create a model which is either (a) as similar to the target model as possible for all input features, i.e. fx(xx)
to steal a target model t
fxx) Vx: E X'' (Song & Shmatikov 2020 Chandrasekaran et al., (222) or (b) predicts labels that has maximal agreement with that of the labels predicted by the target model for given data generating distribution. i...
f = arg min Prz=p| (x(f f' (x)) (Tramèr et al. 2016 Pal et al. 2020;] Jagielski et al. 2020). The first type of attacks are called the functionally equivalent attacks. The later family of attacks is referred as the fidelity extraction attacks. The third type of attacks aim to find an extracted model fE that achieves maximal classification accuracy for the underlying private dataset used train the f These are called task accuracy extraction attacks (Tramèr et al. 2016; Milli et al. 2019 Orekondy et. 2019). In this paper, we generalise the first two type of attacks by proposing the distributionally equivalent attacks and experimentally show that it achieves significant task accuracy.
Membership Inference Attack. Another popular family of inference attacks on ML models the
Membership Inference (MI) attacks (Shokri et al.) 2017: Yeom et al. 2018). In MI attacks, given private (or member) dataset D to train 1 and another non-member dataset S with JD" n * 0, the goal of the adversary is to infer whether any E X' is sampled from the member dataset D or the non-member dataset S. Effectiveness of an MI attacks can be measured by its accuracy of MI, i.e. the total fraction of times the MI adversary identifies the member and non-member data points correctly. Accuracy of MI attack on the private data using f rather than ff is considered measure of effectivvness of the extraction attack (Nasr et al. 2019). We show that the model f extracted using MARICH allows us to obtain similar MI accuracy as that obtained by directly attacking the target model fe using even larger number of queries. This validates that the model f by MARICH in a black-bo box setting acts as an information equivalent replica of the target model ft.
4 DISTRIBUTIONAL EQUIVALENCE & MAX-INFORMATION MODEL EXTRACTIONS: A VARIATIONAL OPTIMISATION FORMULATION
In this section, we introduce the notions of distributionally equivalent and max-information model extractions. We further reduce both the attacks into a single variational optimisation problem.
Definition (Distributionally Equivalent Model Extraction). For any query generating distribution DR over TRd x y, an extracted model , : Rd + Y is distributionally equivalent to target model f : Idd + Y if the joint distributions of input features Q E Rd ~ D2 and predicted labels induced by both the models are same almost surely. This means that for any divergence D, two distributionally equivalent models f and fr satisfy D(Pr(f" (Q), Q)1l (r(fF(())
= OAD DQ.
To ensure query- -nery-efficieny in distributionally equivalent model extraction, an ad adversary aims to choose a query generating distribution D& that minimises further. If we assume that the extracted model is also parametric function, i.e. 1 with parameters w E Ω, we can solve the query-efficient distributionally equivalent extraction by computing
(W
Do A arg min arg min min
wen
DQ
Equation T Tallows us to choose a different class of models with different parametrisation for extrac- tion till the joint distribution induced by it matches with that of the target model. For example, the extracted model can be logistic regression This formulation also enjoys the freedom to choose the data distribution D& for which we want to test the closeness. Rather the distributional equivalence pushes to find the best query distribution for which the mismatch between the posteriors reduces the most and to compute an extracted model f.E. that induces the joint distribution closest to that of the target model l7.
CNN if the target model is a logistic regression.
4

---

Under review as a conference paper at ICLR 2023
If we choose D D", distributional equivalence reduces to the fidelity extraction attack. we choose D
Remark 1.
win Uni distributional equivalent extraction coincides with functional equivalent extraction. Thuus, distributional equivalence attack can ensure both fidelity and functional equivalence extractions depending on the choice of query generating distribution DQ.
Theorem 1 (Upper Bounding Distributional Closeness). If we choose KL-divergence the diver- gence function D, then for a given query generating distribution Do
DKL (Q), Q)Il Pr(fupp, (Q), Q)) I min min (Q - (2)
By variational principle, Theorem | implies that minimising the upper bound on the RHS will lead to an extracted model which minimises the KL-divergence for a chosen query distribution.
Max-Information Model Extraction. The common objective of any inference attack is to leak as much information as possible from the target model f Specifically, in model extraction attacks, we want to create an informative replica f of the target model f such that it induces a joint distri- bution Pr(f (Q); Q)) which retains the most information regarding the target's joint distribution. As adversary can control the query distribution, we want to choose such a query distribution DQ for which the information leakage is maximised.
Definition 2 Max-Information Model Extraction). A model fE IRd Y and query distribution Do are called max-information extraction of a target model fr IRd -> Y and max-information query distribution, respectively, if they maximise the mutual information between the joint distribu- tions of imput features Q E R& DQ and predicted labels induced by f and that of the target model. Mathematically, (f=', D is max- max-information extraction of fg- such that
(Witxxnnf D A arg max arg max ((P((ff (Q), QIII
(Q)
(3)
W
Da
Similar to Definition Definition 2 also does not restrict us to choose a parametric model w dif- ferent than that of the target f. It also allows us compute the data distribution DQ for which the information leakage is maximum rather than relying on the private dataset D used for training f".
Theorem (Lower Bounding Information Leakage). For any given distribution D9, the informa- tion leaked by any max-information attack (Equation 3) is lower bounded as follows:
I(Pr(fT-()) (Q), Q)Il Pr Maaselnt (Q)
2 max -Eq (Q), 13 (Q)) + (())
(4)
By variational principle, Theorem 2 implies that maximising the lower bound in the RHS will lead to an extracted model which maximises the mutual information between target and extracted joint distributions for given query generating distribution.
Distributionally Equivalent and Max- Informmooontion Extractions: A Variational Optimisa- tion Formulation. From Theorem [| and 2 we observe that the lower and upper bounds of the objective functions of distribution equivalent and max-information attacks are negatives of each other. Specifically, - DDL (Pr(fT (Q), Q) Pr(fe Pr(seee (Q), Q)) 2 max_ - (F DQ) and
I(Pr(f7 (Q), Q)I Prff Wa
Maxini (Q).Q) 2 max F(w, D9), where
(5) Thus, following variational approach, aim to solve optimisation problem on F(w, DQ) in an online and frequentist manner. Specifically, we do not assume a parametric family of D& Instead, choose set of queries Q E Rd at each round t T. This leads to empirical counterpart of our problem, i.e.
F(w,D9 A ( (Q), f2 (QM) + ((E)(Q)).
max
max
F(w,Q Q1,TT)
A max max WEW Q ED (Z)
1
Σ
((). + Σ H (Q)).
wEW Quo ED
T
As we need to evaluate f for each Q1, refer Q''s queries, the dataset D Fd x y from where they are chosen as the query dataset, and the corresponding unobserved distribution D as the query generating distribution. Given the optimisation problem of Equation we propose an
algorithm MARICH to solve it effectively.
5

---

Under review as a conference paper at ICLR 2023
5 MARICH: A QUERY SELECTION ALGORITHM FOR MODEL EXTRACTION
In this section, we propose an algorithm, MARICH, to solve Equation in query-efficient manner.
Algorithm Design. In Equation 6 we observe that once the queries Qu.21 are selected the outer maximisation problem, equivalent to regualrised loss minimisation, can be solved using any standard empirical risk minimisation algorithm, such as Adam, SGD etc. Thus, to achieve query efficiency, we focus designing query selection algorithm that selects batch Q2 at round s T such that
Qi - arg max- QEDo
+ Σ (Q.
Here, fat is the model extracted by round 1. The objective of query selection using Equation 7 is two-fold. First, we want to select a query that maximises the entropy of predictions for the extracted model _ This allows us to select the queries which are most informative about the mapping between the input features and the prediction space. Secondly, Equation 7 also pushes the adversary to select queries where the target and extracted models mismatch the most. Thus, minimising the loss between target and extracted models for such query forces them to match over the whole domain of queries. In MARICH, we execute this query selection in two phases.
Initialisation Phase. the Phase 1, i.e. the initialisation phase, we select a set of ni queries, called Qr uniformly randomly from the query dataset D We send these queries to the target model and collect corresponding predicted classes Yttrin (Line 3 We use these ng samples of input-predicted label pairs to construct primary extracted model To
Adaptive Phase. Phase 2, i.e. the adaptive phase, we select B number queries at every round t. In order to be query-efficient, namely ENTROPY SAMRLOPISAMMPIGG
sequentially deploy three query selection mechanisms, GRADIENTRRADE
and LossS AMPLING These tech- niques together aim to select batch of Q: every step that optimises the objective of Equation]
ENTROPYSAMPLEE First, we aim to select the set of queries which unveil most information about the mapping between the input features and the prediction space. Thus, we deploy ENTROPYSAM, PLING. In ANTROPYSAMPLING
we compute the output probability vectors from f.E for all the query points in Do Qtaain and then select top B points with highest entropy (Line 18). This selects the queries Qentropy, about which f. is most confused and training these essentially would make the model a better classifier.
To be frugal about the number of queries, we refine Qentropy to compute the most diverse subset of it. First, we compute the gradients of entropy of f (x), i.e. (=)()E for all x E Qontropy, The gradient at point reflects the change at x in the prediction distribution induced by f. We use these gradients to embed the points x E Qenaropy Now, we deploy K-means clustering to find k (= #classes) clusters and sample ? B points (Line 25 namely Qaradd from these clusters. Selecting points from & different clusters ensures diversity of queries and reduces the number of queries by 7/1
LOSSSAMPLING. In this step, we select points from Qarad for which the predictions of fT and ff are most dissimilar. To identify these points, we compute the loss (z), (f)((=))) for all x E Qtaqin. Then, we select top-k points from Qt with the highest loss values (Line 29 and sample subset Qloss of size B from Qerad which closest to the k points selected from Q (Line 31). This ensures that f would align with f" better it trains on the points at which the mismatch in their predictions is the highest.
Remark 2. Eq. 7 dictates that the active sampling strategy should try to select queries that max- imise the entropy in the prediction distribution the extracted model, while decreases the mismatch in predictions of the target and the extracted models. We further use the EANTROPYGRADENTTEE AM- PLING to choose a smaller but most diverse subset. As Eq. 77) does not specify any ordering between these objectives, choose to use sampling strategies in the decreasing order of runtime complexity as the first strategy selects the queries from the whole query dataset, while the following strategies work only the already selected queries. We show in Appendix C that LossS incurs the highest runtime while ENTROPYSAMPLIIN
can argue about the sequence of using these three sampling stralegies. We
followed by
significantly cheaper.

---

Under review as a conference paper at ICLR 2023
Algorithm MARICI
Target model: f Query dataset: D # #laasses: k Parameter: #initial samples: rog, Training epochs: Emans # #Bathhes of queries: T, Query budget:
Input:
B. Subsampling ratios: 2/2 E (0, 1]
Output: Extracted model f
1: f/a Initialisation the extracted model*//
D Phase
2: Qiaain +- Rog datapoints randomly chosen from DQ
3: Yo + ( (Qh
D Query the target model f" with Qi Qtrain
4: for epoch +- to Emaz do
5: 1P +- Train f with (Q) Ygrain)
6: 1/a Adaptive query selection to build the extracted model%// do
Phase
7: for +
Qentropy < ENTROPY D Q B)
8:
Qorad + EATRROOGGR..DDENTSSAPPPNGGGG Qe ;/ B)
9:
10:
Qosss + LOSSSAMPLING(fFF. Qtra Qt Y 1)
Ynew f (Q(
Query the target model f with Q
12:
Qtrain + Q UQ
13:
Marain + Y U U.Yeow
for epoch <- to Emnax do
15:
f.2 -Taain f. with (Qtroin y!vain)
16:
return Extracted model f ff
17:
function EXTROPYSAAPLINNGEXXTAAN model: f input data points: Xin, budget: B)
18:
Qentropy < arg max (( Select points with maximum entropy
19:
return Qentropy
20:
function ENTROPY CAADOOGRRDDNSSSAPPNNNNttttttttEE model: f "iipuu: Xin, budget: yi B)
21:
E+ (X
22:
G E
23:
Cin + centres G computed using K-means
24:
Qgrad arg cx.. Ex,xx ExjeC. VV Va, F11
25:
Select points from Xin whose aE are closest to that of Cin dz
return Qgrad
27:
function LOSSS LPLIN model: f input data points: Xins previous queries:
Qtrain previous predictions: Ytrains budget: B)
28:
L - [(Ytrains f (Qtrain))
> Compute the mismatch vector
29:
Qmis f- ARGMAXSORRT
Select top-k mismatching points
30:
Qioss < arg min x EmEx Σz eQ || x D Select 71 points closest to Qmis
31:
return Qloss
At the end of Phase 2 in each round of sampling, Qloss is sent fT for fetching the labels Ytrain predicted by the target model. We use (Q)esa "oo=s, Y!los)) along with (Q Y to train f further. Thus, MARICH performs no BT number of queries through + number of inter- actions with the target model f to create the final extracted model f We describe pseudocode for MARICH in Algorithm I. We experimentally demonstrate effectiveness of the model extracted by MARICH to achieve high task accuracy and to act as an informative replica of the target model for extracting private information regarding the private training data D
EXPERIMENTAL ANALYSIS
In this section, perform experimental evaluation of models extracted by MARICH. We discuss the experimental setup, the objectives of experiments, and experimental results. We defer the source code, additional results, effects of defenses, and hyperparameter choices the appendix.
Experimental Setup. We have implemented prototype of MARICH using Python 3.9 and Py Torch 1.12, and run NVIDIA Quadro 10100 32 GB GPU. We perform attacks against three

---

Under review as a conference paper at ICLR 2023
310
Enirayy
Tagge zodid (llNNE11
Randon
Randen
Brason
Rairepy
Estrapy
Bramnes
Target modal
Tarpel aodal 11:)
200
305
1890
Querio
Qardo
(a) LR, EMNIST
(b) LR, CIFARIO
(c) ResNet18, STLI0
(d) BERT, AGNews
Figure 2: Accuracy of the extracted model (mean over 10 runs) in comparison to the target model using MARICH, entropy sampling, and random sampling. Each figure stands for a target model class and query dataset. Models extracted using queries selected by MARICII are closer target model.
target models (f (f)), namely Logistic Regression (LR), ResNet18 (He et al., 2016), and BERT (De- vlin al. 2018). trained on three private datasets (D"): MNIST handwritten digits (Deng 2012). CIFARTO (Krizhevsky et a and BBC News,, respectively. For model extraction, we use EMNIST letters dataset (Cohen et al. 2017), CIFARTO, (TTLIO, and AGNews (Zhang et al. 2015). as publicly available and mismatched query datasets D To instantiate task accuracy, we compare accuracy of the extracted models MARICH with the target model and models extracted by Random Sampling
(RS), fis. To instantiate informativeness of the extracted models (Nasr al. 2019). we compare the membership inference (MI), i.e. MI accuracy, AUC of MI, and MI agreements, performed on the target models, and the models extracted using MARICH and RS with same query budget. For MI, we use in-built membership attack from IBM ART (Nicolac et al. 2018). The objectives of our experimental studies are:
1. How do the accuracy of the model extracted using MARICH on the private dataset compare with that of the target model, and RS with same query budget?
2. How do the models extracted by MARICH behave under membership inference attack in compar- ison to the target model, and the models extracted by RS with same query budget?
Accuracy of Extracted Models. MARICH extract logistic regression models with 5,130 and 1,420 queries from EMNIST and CIFAR10 query datasets by attacking a target logistic regres- sion model, flogietsc trained MNIST. While the target model achieves 90.82% test accuracy, the models extracted using EMNIST and CIFARIO achieve test accuracies 82.37% (90.69% of eT logiette) and 89.48% (98.52% of fingietie), respectively (Figure 2a and 2b The models ex- tracted using RS show test accuracy 52.96% and 84.18%, and the models extracted by Entropy sampling achieve 80.81% (88.97% of slogietse) and 99.66% (98.72% of flogistic). MARICH at- tacks a ResNet18, f trained on CIFARIO (accuracy: 93.58 (3.58%) with 6,950 queries from STLIO dataset. The extracted ResNet18 shows 68. 28% (72.90% of ffeesNe) test accuracy. But the model extracted using RS and Entropy sampling achieve 9.99% and 9.39% accuracy (Fig. 2c). To verify MARICH'S effectiveness for text data, we attack a BERT, SHET trained on BBC- News (test accuracy: 98.65%) with queries from the AGNews dataset. By using only 1.070
queries, MARICH extracts a model with 87.01 87.01%(88..)) 20% of fT FBERT) test accuracy (Figure 2d The model extracted using RS and Entropy sampling shows test accuracy of 76.41% 41% and 62.57% Thus, for all the models and datasets, MARICH is able to extract models that achieve closer test accuracy with respect to the target models and are more accurate than the models extracted by both Entropy sampling and RS.
14
Target member
Target nonmember
12
Marich member
10
Marich nonmember
Denalty
Random member
Membership Inference with Extracted Models. In Ta- ble [ we report the statistics, i.c. accuracy, agreement in inference with target model, and agreement AUC, of membership attacks performed on different target mod- els and extracted models with different query datasets. The models extracted using MARICH demonstrate higher MI agreement with the target models than the models ex- tracted using RS (Figure 4. They also achieve compara- ble MI accuracy with respect to the target model. These results instantiate that the models extracted by MARICH act as informative replicas of the target.
Random nonmember
0.0
0.2
0.1
0.6
0.8
1.0
Probahilit
Figure 3: BERT with AGNews queries
8

---

Under review as a conference paper at ICLR 2023
Table Statistics of membership inference (MI) for different target models, datasets & attacks
Unery Alprriim
Non-member
Onerry
ICC.
ngreement agreement
MNKT
909
MNIST
EMNIST CIFARIO
MARICH MARKCH
EMNEST
5,130 (3.55)
55.58%
92.82%
92.73%
MNIST
1R
CIFARIO
1-200 (2375) 37%)
94,27%
93.97%
92.47%
MNIST
1R
EMNIST
RS
EMNEST
5,130 (3.59)
$9.619
91.01%
91. 115
MNNSTT
LR
CHAKIO
RS
CIEARIO
1,420 (2,37
92.69%
1935555
35.7916
CIPARTO
Restett
STETO
CIFARIO
Resrett9
STL.0 STL.0
MARKCH
STLJO
6.990 da 1895)
90.90%
75.52%
7649%%
CIFARIO
Resseti9
RS
STLIO
6,950 (6.15%)
92.32%
75.25%
75 13.92%
BBCNEWS
BERT
NONERS
1,200
98.6%%
BBCNews
BERT
AGNeos
MARICH
MGNews
1umu c0 K316)
94.42%
91.2%
82.63%
BBCNews
BEXT
AGN
AGTNow
1,070 (0.83%)
6466
100
Narth ovend
146
106
Mand Ovend
300
Apvezzat
Marih
Mond wordd
65
1.2
Nezterap tamutola
Nazberap tameboaa
Nazderap taritaaa
(a) LR with EMNIST
(b) LR with CIFARIO (c) ResNet18 with STL10 (d) BERT with AGNews
Figure 4: Membership agreements of models extracted by MARICH and by Random Sampling (mean over 10 runs). Each figure stands for target model class and a query dataset. Models extracted using queries selected by MARICH are higher.
Analysis of Membership Distributions. To analyse the outcomes of MI attacks further, we visu- alise the membership distributions for a BERT model trained on BBCNews dataset, and two BERT models, f MARICH and frondom extracted using MARICI and RS respectively. To check efficacy of MI, we use BBCNews the member dataset and AGNews as the non-member dataset. After train- ing 5 neural nets on them, we compute the distribution of membership probabilities for the member and non- mom-meber data points for f", f MMRRICH and f frandom From Figure 3 we observe that MI on MMaRiCH produces membership distributions closer to those of f than those of f random.
fE
Summary of Results. From the experimental results, we deduce the following conclusions.
Accuracy. Test accuracy (on the subsets of private datasets) of the models extracted using MARICH are higher than the models extracted with random queries and... of the target model (Figure 2 and Table D. This shows effectiveness of MARICH as a task accuracy extraction attack, while solving distributional equivalence and max-info extractions.
Effeettvv Membership Inference. The agreement MI achieved by attacking MMARICH and the target model is always higher than that of the f 4). Also, membership accuracy for M are 88.58% 94.42% (Table l). This shows that the models extracted by MARICH act as informative replicas of the target model.
f
Query-effciincy: Table I shows that MARICH uses only 1,070 6,950 queries from the public query datasets, which in most cases are lower than data used for training the target models. This shows MARICH is significantly query efficient whereas other known active learning attacks use at least 10,000 of the query dataset to begin (Pal et al. 2020 Table 2).
CONCLUSION AND FUTURE DIRECTIONS
In this paper, we investigate the design of a model extraction attack against a target ML model (specifically classifier) trained
private dataset and accessible through public API, which re- turns only a predicted label for a given query. We propose the notions of distributional equivalence extraction, which extends the existing notions task accuracy and funetionally equivalent model ex- tractions. We also propose another information-theone further propose a variational relaxation of these two types of extraction attacks, and solve it using an online and adaptive query selection algorithm, MARICH. MARICH uses publicly available query dataset different than the private dataset. We experimentally demonstrate that the models extracted by MARICH achieve 68.22 89.48% accuracy on the private dataset while using 1,070 6,950 queries. For both text and image data, we demonstrate that the models extracted by MARICH act as informative replicas of the target models leading to 79.35 94.42% accuracy and 75.52 93.97% agreement in MI on the target model. Typically, the functional equivalence attacks require model- specific techniques, while MARICH is model-oblivious while performing distributional equivalence attack. This poses an interesting question: is distributional equivalence extraction 'casier' than func- tional equivalence extraction, which is NP-hard (Jagielski et al. 2020
notion, i.e. Max-Info model extraction. We

---

Under review as a conference paper at ICLR 2023
REFERENCES
Lejla Batina, Shivam Bhasin, Dirmanto Jap, and Stjepan Picek. CSI neural network: Using side- channels to recover your artificial neural network information. CoRR, abs/1810.09077, 2018.
Varun Chandrasekaran, Kamalika Chaudhuri, Irene Giacomelli, Somesh Jha, and Songbai Yan. Ex- ploring connections between active learning and model extraction. In 29th USENIX Security Symposium (USENIX Security 20), pp. 1309-1326, 2020.
Gregory Cohen, Saeed Afshar, Jonathan Tapson, and André van Schaik. Emnist: Extending mnist to handwritten letters. In 2017 International Joint Conference Neural Networks (LJCNN), pp- 2921-2926 2017. doi: 10.1 109//ICNN.2017.76661111
Jacson Rodrigues Correia-Silva, Rodrigo F Berriel, Claudine Badue, Alberto F de Souza, and Thiago Slivira-Saatto Copycat enn: Stealing knowledge by persuading confession with random non- labeled data. In 2018 International Joint Conference on Neural Networks (IJCNN), pp. 1 1-8. IEEE, 2018.
Li Deng. The mnist database of handwritten digit images for machine learning research. JEEE Signal Processing Magazine, 29(6):141-142, 2012.
Jacob Devlin, Ming- Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 2018.
Ian Goodfellow, Yoshua Bengio, Aaron Courville, and Yoshua Bengio. Deep learning, volume 1.
MIT Press, 2016.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 770-778, 2016.
Matthew Jagielski, Nicholas Carlini, David Berthelot, Alex Kurakin, and Nicolas Papernot. High accuracy and high fidelity extraction of neural networks. In 29th USENIX security symposium (USENIX Security 20), pp- 1345-1362, 2020.
Mika Juuti, Sebastian Szyller, Samuel Marchal, and N Asokan. Prada: protecting against dnn model stealing attacks. In 2019 IEEE European Symposium on Security and Privacy (EuroS&P), pp- 512-527. IEEE, 2019.
Alex Krizhevsky, Vinod Nair, and Geoffrey Hinton. Cifar-10 (canadian institute for advanced re- search). hwt.t://...ttttt//.......... wtw.:ec.stttttoooooooooo toronto. edu/ catae//cifaaraa. html cs
Daniel Lowd and Christopher Meek. Good word attacks on statistical spam filters. CEAS, volume 2005, 2005.
Smitha Milli, Ludwig Schmidt, Anca Dragan, and Moritz Hardt. Model reconstruction from model explanations. In Proceedings of the Conference on Fairness, Accountability, and Trans- parency, pp. 1-9, 2019.
Takayuki Miura, Satoshi Hasegawa, and Toshiki Shibahara. Megex: Data-free model extraction attack against tradient-based explainable ai. arXiv preprint arX1v:2107.08909 2021.
Milad Nasr, Reza Shokri, and Amir Houmansadr. Comprehensive privacy analysis of deep learning: Passive and active white-box inference attacks against centralized and federated learning. In 2019 IEEE symposium on security and privacy (SP), pp. 739-753. IEEE, 2019.
Maria-Irine Nicolae, Mathieu Sinn, Minh Ngoc Tran, Beat Buesser, Ambrish Rawat, Martin Wis- tuba, Valentina Zantedeschi, Nathalie Baracaldo, Bryant Chen, Heiko Ludwig, Ian Molloy, and Ben Edwards. Adversarial robustness toolbox v1.2.0. CoRR, 1807.01069, 2018. URL.
atttp:://ariit
http:/df//800/ 01069
Tribhuvanesh Orekondy, Bernt Schiele, and Mario Fritz. Knockoff nets: Stealing functionality of black-box models. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 2019.
10

---

Under review as a conference paper at ICLR 2023
Soham Pal, Yash Gupta, Aditya Shukla, Aditya Kanade, Shirish Shevade, and Vinod Ganapathy- Activethief: Model extraction using active learning and unannotated public data. In Proceedings of the AAAI Conference Artificial Intelligence, volume 34, pp. 865-8722 2020.
Soham Pal, Yash Gupta, Aditya Kanade, and Shirish Shevade. Stateful detection of model extraction attacks. arXiv preprini urXiv:2107.05 2021.
Nicolas Papernot, Patrick McDaniel, Ian Goodfellow, Somesh Jha, Z Berkay Celik, and Ananthram Swami. Practical black-box attacks against machine learning. In Proceedings of the 2017 ACM on Asia conference computer and communications security, pp. 506-519, 2017.
Mauro Ribeiro, Katarina Grolinger, and Miriam AM Capretz. Mlaas: Machine learning service. In 2015 IEEE 14th International Conference on Machine Learning and Applications (ICMLA), pp. 896-902. IEEE, 2015.
Reza Shokri, Marco Stronati, Congzheng Song, and Vitaly Shmatikov. Membership inference tacks against machine learning models. In 2017 IEEE symposium on security and privacy (SP), pp. 3-18. IEEE, 2017.
Congzheng Song and Vitaly Shmatikov. Overlearning reveals sensitive attributes. In 8th Interna- tional Conference on Learning Representations, ICLR 2020, 2020.
Joe Staines and David Barber. Variational optimization. arXiv preprint
Florian Tramèr, Fan Zhang, Ari Juels, Michael K Reiter, and Thomas Ristenpart. Stealing machine learning models via prediction {APIs). In 25th USENIX security symposium (USENIX Security 16), pp. 601-618, 2016.
Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoc V Le. Xlnet: Generalized autoregressive pretraining for language understanding. Advances in neural information processing systems, 32, 2019.
Samuel Yeom, Irene Giacomelli, Matt Fredrikson, and Somesh Jha. Privacy risk in machine learn- ing: Analyzing the connection overfitting. In 2018 IEEE 3 computer security foundations symposium (CSF), pp. 268-282. IEEE, 2018.
Ashkan Yousefpour, Igor Shilov, Alexandre Sablayrolles, Davide Testuggine, Karthik Prasad, Mani Malek, John Nguyen, Sayan Ghosh, Akash Bharadwaj. Jessica Zhao, Graham Cormode, and Ilya Mironov. Opacus: User-friendly differential privacy library in PyTorch. arXiv preprint
wwXiv:2/09.12298, 2021.
Xiang Zhang. Junbo Jake Zhao, and Yann LeCun. Character-level convolutional networks for text classification. In NIPS, 2015.

---

Under review as a conference paper at ICLR 2023
A PROOFS OF SECTION
In this section, we elaborate the proofs for the Theorems and 2
Theorem | (Upper Bounding Distributional Closeness), If we choose KL-divergence as the diver- gence function D, we can show that
(KL(Pr(JE (Q), Q)II Pr (Q) & min EQ) -
E
Proof. Let us consider a query generating distribution D on R A target model fø. R2 + y induces a joint distribution over the query and the output (or label) space, denoted by Pr(fØr Q). Similarly, the extracted model f Rd - y also induces a joint distribution over the query and the output (or label) space, denoted by Pr( Q).
(Pr (Q), (Q), Pr (Q). Q))
= =I dPr(Je- (Q), Q) log Pr(Sz-
QERR
Pr(f
- (Q) g) Pr(Q log Pr( 9) dg
Pr(f2 (Q) 9)
J Pr(fT. 9) Pr(Q log Pr(sā. (Q)IQ g) dq
I (Q) = 4) Pr(Q = 4) log Pr. (Q)QQ = q) dg
| (T(F((Q))4 = 9) Pr(Q = 4) Pr(f (Q) = 9) dq + Eg (())
QERA
< -H( (Q) dq Eq (11 S -- (Q) dq Ey-oe (()
The last inequality holds true as the extracted model f is trained using the outputs of the target model f Thus, by data-processing inequality, its output distribution posses less information than that the target model. Specifically, we know that if Y f(X), HYY < H(X).
Now, by taking min, both sides, obtain
DKL (Pr (Q), Q)I
Pr[Juppp DDR (Q), min (f) (())
Here, Whg A arg min_ DKL (Q) Q)Il Pr(f LHS and RHS coincide.
Q)). The equality exists if minima of
Theorem
0 (Lower Bounding Information Leakage). The information leaked by any max- information attack (Equation 3) is lower bounded follows:
(Q), Pr(se Pr(Set (Q),Q)) 2 max - (Q) H(
Proof. Let us consider the same terminology as the previous proof. Then,
= -0(110000001T((30000-00000000000
= =H((((()).)+++(((0))) (1 ((
2 10110000001111000101110101101100
2 H(FE(Q)) -((E(())H((5001))))) (1
(10)
2 H(S (Q)) -EQIUE(())))
(11)
Throouhhut the proofs, we slightly abuse the notation to write I(Pr(X), as I(X, Y) for avoiding cumbersome equations.
12

---

Under review as a conference paper at ICLR 2023
The inequality of Equation i due the fact that entropy is always non-negative. Equation hollls true as (X, Y) 2 max ( H(Y} for two random variables X and Y. The last inequality is due to the fact that conditional entropy of two random variables X and Y, i... H(X))), is smaller than or equal to their cross entropy, i.e. 1(x, Y) (Lemma
Now, by taking max on both sides, we obtain
I(Pr(f) (Q), QIIl Pr(fu Mtnxtal (Q), S max Eol (Q), (2)) H(
Here, Maxinf - arg max I(Pr(ff (Q). Q) Pr(f Maxtat (Q), Q)) The equality exists if maxima of LHS and RHS coincide.
Lemma 1 (Relating Cross Entropy and Conditional Entropy). Given two random variables X and Y, conditional entropy
H(X S ((.
(12)
Proof. Here, H(X & P y) log - Pr(()) In Pr(y) denotes the cross-entropy.
Pr(n.y) Pr(y)
and ((X,Y) ((PI())
1(,YY (() + ((L(PP()))))(()))
H( I(X:) (Px(I))
2 H
The last inequality holds as both mutual information and KL-divergence DKL are non-negative functions for any X and Y.
13

---

Under review as a conference paper at ICLR 2023
B ADDITIONAL EXPERIMENTAL RESULTS
In this section, we elaborate further experimental setups that we skipped for the brevity of space in the main draft. We provide anonymised version of the code at: attpss://drive..ooggl. com com/dr http//vvveeffddrr///// 1mpM zE3w 2E3w_pIS0c uiR9Jw Uw wtt.:/WWeeerrssssseeseee
B.1 ACCURACY OF MODELS EXTRACTED BY MARICH AND OTHER SAMPLING STRATEGIES
To compare MARICH with other active learning algorithms, we attack the same target models using Entropy sampling, K-centre sampling, and Random sampling using the same number of queries as used for MARICH.
On the LR models we have shown performances of both Entropy sampling and K-centre sampling, while due to time and resource constraint, we could not present the K-centre sampling results for the BERT and ResNet18.
From the results in Figure we see that most cases MARICH outperforms other algorithms. In Figure 2 have not plotted the standard deviations for better visibility. We plot both mean + standard deviation over 10 runs in Figure
80
80
Accuracy %
%
60
60
MARICH
MARICH
40
Random
Random
Entroyy
Entropy
20
K-Centre
K-Centre
Target model (LR)
Target model (LR)
0
1000
2000
3000
4000
5000
250
500
750
1000
1250
Queries
Queries
Logistic regression with EMNIST queries
(b) Logistic regression with CIFAR queries
100
80
Accuracy %
MARICH
Accuracy %
60
Random
60
Entropy
40
Target model (ResNet18)
MARICH
40
Random
Entropy
20
Target model (BERT)
20
2000
4000
6000
260
400
600
800
1000
Queries
Queries
(c) ResNet18 with STL10 queries
(d) BERT with AG news queries
Figure 5: Comparison among different active sampling algorithms.
Extraction of CNN Trained on MNIST. Along with the other experiments mentioned in the paper, we trained CNN with MNIST handwritten digits (D" here), that shows a test accuracy of 97.53% on disjoint test set. Two datasets are used as y here, EMNIST letters and CIFAR10 to extract two CNN models from the target model. MARICH extracts two CNNs using queries from EMNIST letters and CIFAR10 10 which show test accuracies of 88.81% % and 87.16% respectively. On the other hand, models extracted using random sampling using EMNIST letters and CIFAR 10 show accuracies of 85.37% and 88.46% respectively. Table 2 contains queries used, and membership inference statistics for all the experiments.
14

---

Under review as a conference paper at ICLR 2023
Membersil spreementt AUC
02.3%%
11.29%
91 11.098
KS 53.505
02.2%
01.10
01.3%%
74. 18.006
12.000
O all Member hipp sgreement
01.018
02.08
51.22
92.353
1532%
23.259
86.958
Ovemal member ship axc
Table Model extraction and membershij inference statisties
87 R7.99%
72300
89.0%
02.6'5
RL.5%
91.7%
91.0%%
91
94.006
02.35%
91.0%
80.1%
02
Membershipp ace Nonnenbennt axx
67.
77,00
83168
71.72%8
34343
73603
78332
91.338
95.5%
98566
07.13%
07.08%
9537%
05.1 05.158
9L.2%
03.0%
93.753
92.558
25015
50,00 70 (1005)
30,000 (T0055
5, 20 1338
1,42 22 37%
50000 (
5,00 (3.0
35 (8218)
Saird (38)
SET SET0W2R1
6050 5%81
6U5H 6 19%)
1,490 (UNES))
1010 (0835))
Querles
Noe menaher dateset
ESNNSTT
CIFATT
EXINISY
CIFARTO
ESINSY
GRARRO
DAINST
CIARIO
EXINST
CIRACIO
EMINST
CIEARTO
STETO
$1L10
AONows
NO
Algurithm used
Kanaknn Sampling
Randon Sampling
Kana Samplng
Random Sampling
MARICT Kandkn vl Sampling
MARICH
MARICH
MARICH
MARICH
Attack Daiazet
EMNIST EMNISY CIAARI
EMNNT
CIFARTO
EMNIST
CIEARTO
STL10
STLIO
AONows
Target model
Resnc199
Lit
18
LR
CNN
ONN
ONN
CNN
BLK!
MOXI
Menber dataset
MNIST
KINIST
MINST
ANSST
MNIST
MNIST
MNIST
MINSST
MNNST
MNNST
CIFARIO
CIFARTO
BICNews
15

---

Under review as a conference paper at ICLR 2023
B.2 MEMBERSHIP INFERENCE WITH THE EXTRACTED MODELS
From Figure we see that in most cases the probability densities of the membership inference are closer to the target model when the model is extracted using MARICH, than using random sampling (RS).
6
Target member
Target member
5
Target nonmember
12.5
Target nonmember
Random member
10.0
Marich member
Density
Random nonmember
Density
Marich nonmember
Marich member
Random member
Marich nonmember
Random nonmember
2
2.5
0.0
0.0
0.2
0.4
0.6
0.8
1.0
0.00
0.25
0.50
0.75
1.00
Probability
Probability
(b) Logistic regression with CIFARIO queries
Logistic regression EMNIST queries
10
Target member
15.0
Target member
Target nonmember
Target nonmember
12.5
Marich member
Marich member
Density
Marich nonmember
Density
10.0
Marich nonmember
6
Random member
Random member
Random nonmember
7.5
Random nonmember
4
2.5
0
0.00
0.25
0.75
1.00
0.0
0.0
0.2
0.4
0.6
0.8
1.0
Probability
Probability
(c) CNN with EMNIST queries
(d) CNN with CIFAR 10 queries
5-
Target Member
14:
Target member
Target Nonmember
Target nonmember
4-
Marich Member
12
Marich member
Density
3-
Marich Nonmember
Random Member
Densily
Random member
Marich nonme mber
8-
Random nonmember
Random Nonmember
2-
1
0
0
0.00
0.25
0.50
0.75
1.00
0.0
0.2
1.0
Probability
Probability
(c) ResNer18 with STLI0 queries
BERT with AGNews queries
Figure 6: Comparison among membership vs. non-membership probability densities for member- ship attacks against models extracted by MARICH, Random sampling and the target model. Each figure represents the model class and query dataset. Memberships and non-memberships inferred from the model extracted by MARICH are significantly closer to the target model.
In Figure present the agreements from the member points, nonmember points and overall agreement curves for varying membership thresholds, along with the AUCs of the overall member- ship agreements. We see that in most cases, the agreement curves for the models extracted using MARICH are above those for the models extracted using random sampling, thus AUCs are higher for the models extracted using MARICH.
16

---

Under review as a conference paper at ICLR 2023
These observations support claim that model extraction using MARICH gives models accu- rate and informative replica of the target model.
Marich overall
Marich membership
Marich nonmembership
Random overall
Random membership
Random nonmembership
100
100
95
90
Agreement %
90
85
70
60
80
50
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
Membership thresholds
Membership thresholds
(a) Logistic regression with EMNIST queries
(b) Logistic regression with CIFAR 10 queries
100
100
95
Agreement %
90
90
85
80
80
70
75
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
Membership thresholds
Membership thresholds
(c) CNN with EMNIST queries
(d) CNN with CIFAR 10 queries
100
100
90
80
Agreement %
80
Agreement
60
70
60
20
50
40
0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
Membership thresholds
Membership thresholds
(c) ResNet18 with STL.0 queries
(f) BERT with AGNews queries
Figure 7: Comparison of membership, nonmembership and overall agreements of membership tacks against models extracted by MARICH and Random sampling and the target model trained with MNIST. Each figure represents the model class and query dataset. Membership agreement of the models extracted by MARICH are higher.
17

---

Under review as a conference paper at ICLR 2023
B.3 FIDELITY OF THE PREDICTION DISTRIBUTIONS THE EXTRACTED MODELS
We claim to achieve distributionally equivalent l from I using MARICH. To measure the perfor- mance of MARICH this objective, measure KL divergence of the output distributions of fz and f when the input is Datfte every round of training.
In Figure & we observe that MARICH and Entropy sampling show almost same performance for this particular case while K-centre sampling performs much worse.
3.0
Marich
Entropy
2.5
KL Divergence
K-Center
2.0
1.5-
1.0
0.5
0.0
1000
2000
3000
4000
5000
Samples
Figure 8: Comparison of fidelity of distributions for different active learning algorithms (LR trained on MNIST attacked with EMNIST).
18

---

Under review as a conference paper at ICLR 2023
C SIGNIFICANCE AND COMPARISON OF SAMPLING STRATEGIES
Given the bi-level optimization problem, we came up with MARICH in which three sampling meth- ods are used in the order: (i) ENTROPYSAMPLING, (ii) ENTROPYGRADDENTSSMPIING,,, and (ii) LOSSSAMPLING.
These three sampling techniques contribute different goals:
ENTROPYSAMPLING selects points about which the classifier at particular time step is most confused
uses gradients of entropy outputs of the extracted model w.r.t. the inputs embeddings and selects points behaving most diversely at ev- ery time step.
LOSSSAMPLING selects points which produce highest loss when loss is calculated between target model's output and extracted model's output.
One can argue that the order is immaterial for the optimization problem. But looking at the algorithm practically,
that ENTROPYGRADIENT AMPLING and LossSAMPLING incur much higher time complexity than ENTROPYSAMPLING set is more efficient than the others. This makes put ENTROPYSAMPLING as the first query selection strategy.
Thus, using ENTROPY AMPLIING on the entire query
As per the optimization problem in Equation 7 we are supposed find points that show highest mismatch between the target and the extracted models after choosing the query subset maximising the entropy. This leads us to the idea of LOSSSAMPLING. But as only focusing on loss between models may choose points from one particular region only, and thus, decreasing the diversity of the queries. We use ENTROPYGRAADIENTS.N AMPLINN before LOSsS This ensures selection of diverse points with high performance mismatch.
In Figure 9 we experimentally see the time complexities of the three components used. These are calculated by applying the sampling algorithms on logistic regression model, on mentioned slices of MNIST dataset.
Entropy sampling
Entropy Gradient sampling
Loss sampling
30
Time (s)
20
10
1000
2000
3000
4000
Size of entire query set
Figure Runtime comparison of three sampling strategies to select queries from 4500 datapoints.
Table 3: Time complexity of different sampling Strategies
Sampling Algorithm
Query space size
#Selected queries
Time (s)
Entropy Sampling
4500
100
1.82 0.04
Entropy- Entropy-Gradient Sampling
4500
100
10.56 + 0.07
Loss Sampling
4500
100
41.64 + 0.69
19

---

Under review as a conference paper at ICLR 2023
D PERFORMANCE AGAINST DIFFERENTIALLY PRIVATE TARGET MODELS
In this section, we aim to verify performance MARICH against privacy-preserving mechanisms.
Specifically, we apply (e, Diffffferlt Privacy (DP) inducing mechanism, namely DP-SGD (?), to train the target model on the member dataset. This mechanism adds noise to the gradients and clip them while training the target model. We use the default implementation of Opacus YYousefoour et al. 2021 to conduct the training in PyTorch.
Following that, attack the 6 target models using MARICH and compute the corresponding accuracy of the extracted models. In Figure I0 we show the effect of different privacy levels on the achieved accuracy of the extracted Logistic Regression model trained with MNIST dataset and queried with EMNIST dataset. Specifically, we assign 8 = 10 and vary in {0.2, 0.5, 1,2, oo). Here, ε oo corresponds the model extracted from the non con-private target model.
We observe that the accuracy of the models extracted from private target models are approximately 2.3 7.4% lower than the model extracted from the non-private target model. This shows that performance of MARICH decreases against DP defenses but not significantly.
80
Accuracy
60
Target model (LR)
3 inf
40
ε 0.2
3 = 0.5
20
Σ = 1
3 = 2
1000 2000
3000 Queries
4000
5000
Figure 10: Performance of models extracted by MARICH against (e, (effeeddiffrrentially private target models for different privacy levels e and 8 10-3
20

---

Under review as a conference paper at ICLR 2023
E EFFECT OF MODEL MISMATCH
we observe that functionality of MARICH is not constrained by select- ing the same model class for both the target model f and the extracted model f But in all the previous experiments, we have used the same model class for both the target and extracted models, le., we have used LR extract LR or CNN to extract CNN. In this section, we conduct experi- ments to show MARICH's capability to handle model mismatch and impact of model mismatch on performance of the extracted models.
From Equation equation 7
Specifically, we run experiments for two cases. We train an LR and CNN model MNIST dataset, and use them as target models. We further extract these two models with two other LR and CNN models using EMNIST the query datasets. We use MARICH without any modification for both the cases when the model classes match and mismatch. This shows universality of MARICU as model extraction attack.
we observe that model mismatch influences performance of the model extracted by MARICH. When extract the LR target model with LR and CNN, observe that both the extracted models achieve almost same accuracy and the extracted CNN model achieves even a bit accuracy than the extracted LR model. In contrast, when extract the CNN target model with LR and CNN, we observe that the extracted LR models achieves lower accuracy than the extracted CNN model.
From Figure TT|
From these observations, we conclude that if we use a less complex model to extract a more complex model, the accuracy drops significantly. But if extract low complexity model with higher complexity one, we obtain higher accuracy instead of model mismatch. This is intuitive as the low-complexity extracted model might have lower representation capacity to mimic the non-linear decision boundary of the high-complexity model but the opposite is not true.
Accuracy %
60
40
LR by LR
CNN by CNN
LR by CNN
CNN by LR
0
1000
2000
3000
4000
5000
1000
2000
3000
4000
5000
Samples
Samples
(a) LR extracted by LR vs. LR extracted by CNN
CNN extracted by CNN CNN extracted LR
Figure 11: Effect of model mismatches on MARICH.
Table 4: Effect of Model mismatch on Accuracy of The Extracted Models.
F
#samples
Accuracy
LR
LR
5130
82.37
5.7%
LR
CNN
5130
85.41 0.57%
CNN
LR
5440
57.81 3.64%
CNN
CNN
5440
91.63 + 0.42%
21

---

Under review as a conference paper at ICLR 2023
F CHOICES OF HYPERPARAMETERS
In this section, we list the choices the hyperparameters of Algorithm | for different experiments and also explain how do select them.
Hyperparameters " and 7/2 are kept constant, i.e., 0.8, for all the experiments. These two parameters the budget shrinking factors.
Instead of changing these two, we change the number of points vo, which are randomly selected in the beginning, and the budget B for every step. We obtain the optimal hyperparameters for each experiment by performing a line search in the interval 500).
We further change the budget over the rounds. At time step I, the budget, B, a x B The idea is to select more points as f goes on reaching the performance of f". Here, a >> 1 and needs be tuned. We use 1.02, which is obtained through line search in [1.01, 1.99).
For number of rounds T, we perform line search in [10, 20).
Table 5: Hyperparameters for different datasets and target models.
Member dataset
Target
Attack
Budget
Initial points
T:
T>
Epochs
Learning
MNIST
LR
EMNIST
500
400
0.8
0.8
14
20
2 10
LR
CIFARIO
150
100
0.8
2 10
CNN
EMNIST
500
350
0.8
0.8
20
10
CIFARIO
ResNorT8
STLIO
390
1500
0.8
0.8
20
4. 10
22


Reviews:
[{"review_id": "swMotw6exq", "reviewer": "Reviewer_LzQT", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper studies how to perform model extraction attacks in a query-efficient manner. Specifically, the authors design a new extraction method named MARICH that leverages active learning to select informative query data to improve the query efficiency of model extraction. Three active learning strategies, entropy-sampling, gradient-sampling, and loss-sampling, are deployed sequentially to select the most informative query data. Experiments are conducted on image and text datasets to verify the effectiveness of the MARICH, in which statistics of classification accuracies and membership inference statistics are employed to evaluate the performance of model extraction attacks.", "summary_of_the_review": "I think this paper should be rejected for now.", "strength_and_weaknesses": "In general, I think this paper is written badly and the proposed method seems to be trivial. Detailed comments are listed below:\n\n1. The ideas in Section 4 are very hard to follow. In this Section, the authors seem to propose two new definitions (Def. 1 and Def. 2) of model extraction performance indicators, but I could not figure out the relationship between these definitions and the proposed MARICH algorithm.\n\n2. In Eq.(1), the authors claim that \"an adversary should choose a query generating distribution $\\mathcal D^Q$ that can minimize the difference between the target model and extracted model\". However, I do not agree with that.\nOn the one hand, if we let $\\mathcal D^Q$ only generate zero values, it would probably result in a small distributional model difference but not help extract informative knowledge from the target model.\nOn the other hand, from my perspective, a query generating distribution that can result in large differences between extracted and target models may help extract more information from the victims, as larger model prediction disagreement may suggest more knowledge to be transferred by query data.\n\n3. The proposed MARICH algorithm seems to be a combination of several existing heuristic strategies that have been already studied in model stealing and other aspects of ML security. Specifically:\n    - The entropy-sampling is as same as that of the \"Uncertainty Strategy\" in ActiveTheif method [r1].\n    - For the gradient-sampling, a similar idea of leveraging gradient information has been used in [r2] for detecting backdoored data.\n    - The loss-sampling is an (slightly) improved \"DFAL + k-center strategy\" in ActiveTheif method [r1].\n\n    All these results may shrink the contribution of the proposed MARICH method.\n\n4. Too few model extraction attack baselines for comparison. The authors only compare MARICH with the random sampling-based method. However, I think they should at least include [r1], which is another existing active learning-based model extraction method, for comparison.\n\n\n[r1] Pal et al. \"ActiveThief: Model Extraction Using Active Learning and Unannotated Public Data\". AAAI 2020.\n\n[r2] Xu et al. \"Detecting AI Trojans Using Meta Neural Analysis\". IEEE SP 2021.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "pI7COp2ddt", "reviewer": "Reviewer_4xVs", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "- The paper addresses the problem of model extraction attacks (i.e., learning a surrogate model to mimic a black-box classifier under a budgeted query constraint)\n- The main contributions of the paper is two-fold: \n    1. formally posing model extraction as a bi-level optimization problem, thereby unifying some work around slightly different extraction notions (fidelity, functional equivalence, ..)\n    2. proposing a novel query-efficient approach to solve the above optimization problem. The key idea is to define a set of heuristics to selectively sample data from a query set.\n- The approach is evaluated on logistic regression, ResNet-18 (CIFAR dataset), BERT (BBC News) and compared a random sampling strategy. Results indicate significant improvements over random sampling strategy in model extraction and membership inference.", "summary_of_the_review": "I find the paper studies an interesting problem and also takes a step in the right direction (formal framework to unify recent works, novel query-efficient approach). The key issue I have is that implementation details are unclear (esp. choice of hyperparams) which leads to a concern that query-efficient claims hold only when the attacker is given an oracle test set to tune the attack hyperparameters. I also have a number of different concerns (e.g., attack evaluated against a defense, ablation studies), but I'm somewhat willing to down-weigh these.\n", "strength_and_weaknesses": "### Strengths\n**1. Formal framework**\n- I appreciate the paper's contribution on unifying a number of recent approaches within a reasonable framework (Eq. 6). While most papers inherently solve this bi-level optimization problem, this is the first paper I'm aware of presents the extraction formulation and further grounds it in a variational formulation.\n\n**2. Significant query-efficient improvements**\n- On the surface of it, the paper appears to significantly improve query-efficiency over a strong random sampling baseline e.g., 24.44 → 68.2% CIFAR accuracy.\n- The core idea is also insightful: leveraging a combination of entropy and diversity within query data.\n\n**3. Related works discussion**\n- I particularly liked the extent of the related work section, which faithfully covers most model extraction attacks (including many earlier works).\n\n### Concerns\n\n**1. (Major concern) Selecting hyperparameters / Implementation details**\n- A key issue I have with the paper is somewhat insuffient technical details to carefully judge effectiveness of the approach.\n- Specifically, I'm unclear on:\n    1. what are the hyperparameters for the sampling strategy (e.g., $\\gamma_1$, $\\gamma_2$, $B$, $k$). Moreover, does this change per dataset?\n    2. What is the architecture of the extracted model $f^E$? Is it the same as $f^T$? Does the architecture choice influence results? After all, some signals for the sampling approach is dependent on the architecture (e.g., gradients, losses)\n    3. Is the model $f^E$ retrained at each time-step $t$? Is it retrained from scratch? For how many epochs is it trained?\n- Importantly, I curious about (1) - how much do choices of the hyperparameters influence results? After all, an reasonable attacker does not have a set to carefully tune it. Can the authors clarify how they searched for the hyperparameters and how much they vary between different datasets?\n\n**2. vs. Defenses**\n- Alongside recent attack approaches, there are also a number of recent defenses (e.g., Kariyappa et al., CVPR '20). I'm somewhat surprised that the attack is not evaluated with a defense in place.\n- At the very least, I would be curious of the attack performance with a very simple defense strategy e.g., where the target model returns only the argmax prediction label.\n\n**3. Evaluation / Ablation**\n- While the experimental results indicate strong query efficiency, there are a few of gaps here:\n    1. how influential is each of the proposed sampling strategies (entropy, gradient, loss)? My suggestion is to validate the influence using an ablation study.\n    2. a key distinguishing factor between the proposed formulation with prior work is the variational treatment and specifically, the entropy term in Eq. (7). Are the authors aware whether this term leads to empirical gains? Intuitively, I posit that it would encourage sampling around decision boundaries. But since it is estimated on an initially noisy randomly-initialized model, I am not sure how good a signal this results in.\n    3. In Fig. 2, why is random sampling a straight line? It appears straight-forward to variably change the size of the random queries and obtain a curve, right?\n    4. Table 1 is somewhat unclear: what do rows with a \"-\", but 100% queries mean?\n\n**4. Disconnect: Formalisms and Approach**\n- While I appreciate the formalism on \"distributional equivalence\", the first part of the paper (Sec. 1-4) led me to different expectations than what's presented later. Specifically, there are multiple claims in the paper on e.g., \"generalizing these three approaches using a novel definition ...\". However, it appears that the approach and evaluation is tailored on a specific approach on maximizing task accuracy, and somewhat overlooks fidelity, functional equivalence, etc. This was a bit misleading in my opinion.\n\n**4. Misc.**\n- Please improve the quality of figures in the paper. Particularly by increasing the font size (highly illegible on paper).\n- The authors can complement their related work section by additionally including:\n    - data-free model stealing attacks e.g., Truong, Jean-Baptiste, et al. \"Data-free model extraction.\" CVPR '21 (see follow-up works on this as well)\n    - Dataset distillation e.g., Wang, Tongzhou, et al. \"Dataset distillation.\" arXiv '18 (who tackle a similar bi-level optimization problem)", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "MdBD160kz_z", "reviewer": "Reviewer_BNEH", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes Marich - a model extraction attack with an adaptive query selection. The three main contributions are formalism, algorithm, and experimental analysis. The authors formally formulate the problem as the Max-Information attack, where the adversary aims to maximize the mutual information between the extracted and target models’ distributions. The proposed algorithm Marich - optimizes the objective of the variational optimization problem. The experimental evaluation is done with both image and text datasets as well as on standard vision (ResNet18) and NLP (BERT) architecture types.", "summary_of_the_review": "The paper does not provide any new important insights into the problem of model extraction. The comparison with baselines is missing. ", "strength_and_weaknesses": "Comments:\n\n- The paper considers the model extraction attack from a theoretical perspective and tries to optimize the query selection.\n- The background on classifiers is totally unnecessary, at the beginning of Section 3, page 3.\n- Merge part of Section 2 on Related Work \"Taxonomy of Model Extraction\" with parts of Section 3: \"Model Extraction Attack\" - this is too repetitive.\n- The formalism of model extraction in Section 4 \"DISTRIBUTIONAL EQUIVALENCE & MAX-INFORMATION MODEL EXTRACTIONS: A VARIATIONAL OPTIMISATION FORMULATION\" is not informative - it only formalizes model stealing while it does not provide any insight on the model extraction itself. Most of this Section could be moved to the appendix.\n- The ideas behind Marich: (1) max entropy of the predictions for the extracted model, and (2) an adversary selects queries where the target and extracted models mismatch the most - are very intuitive and known without a formal derivation. The 1st one is the principle behind active learning while the 2nd naturally evolves from fidelity-driven extraction.\n- Even in the initialization phase of Marich, you could select the most diverse set of queries - these whose, e.g. L2 distance between each other is the largest.\n- Note, you still select queries for stealing that are close in terms of their distribution to the training samples of the victim models.\n\n\nQuestions:\n\n- How do the samplings: of entropy, gradient, and loss reflect the optimization stated in Equation 7? \n- Why does NOT the random selection of queries (the red line) in Figure 2 start from 0 and goes up with more queries sent against the public API?\n- On page 8, what is the meaning behind the percentage: \"with 6,950 (6.15%) queries from STL10 dataset\"?\n- I assume the authors do not train the BERT model from scratch on the BBCNews data but only fine-tuned a pre-trained BERT. Which exactly BERT model was used? \n- Why does the model extracted using random selection achieves only 24.44% - \"the model extracted using RS achieves 24.44% (26.12% of fTResNet) accuracy (Fig. 2c).\" The provided information: \"0.83 − 6.15% of the query dataset\" is not giving any valuable insight since the query dataset can be arbitrarily large. \n\nRelated work\n1. Type of Query dataset from page 3 - the data-free model extraction generates queries using a GAN (Truong et al. 2021).\n2. The related work is incomplete - check the latest approaches, for instance: https://openreview.net/forum?id=EAy7C1cgE1L\n3. This paper cites Jagielski et al. but does not compare with any other attacks, such as MixMatch - proposed by Jagielski et al. or does not compare with baselines that use active learning (e.g., via entropy sampling, k-centers, or margin sampling). The attacks like KnockOff (Orekondy et al. 2020) that also leverages active learning (Chandrasekaran et al. 2020) or defenses that use information leakage via privacy leakage, margin sampling, or entropy (Dziedzic et al. 2022), are not even discussed. \n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject