Marich: A Query-efficient Distributionally Equivalent Model Extraction Attack
Authors: Pratik Karmakar
Debabrota Basu
Venue: NeurIPS 2023 poster (2023)
Keywords: Differential Privacy
Model Extraction Attacks
Active Sampling
Max-Information Attack
TL;DR: We propose a maximum information extraction framework for black-box model extraction. Our framework leads to an active algorithm that constructs a predictive distribution closest to the one induced by the target model.

Abstract: We study design of black-box model extraction attacks that can *send minimal number of queries from* a *publicly available dataset* to a target ML model through a predictive API with an aim *to create an informative and distributionally equivalent replica* of the target.
First, we define *distributionally equivalent* and *Max-Information model extraction* attacks, and reduce them into a variational optimisation problem. The attacker sequentially solves this optimisation problem to select the most informative queries that simultaneously maximise the entropy and reduce the mismatch between the target and the stolen models. This leads to *an active sampling-based query selection algorithm*, Marich, which is *model-oblivious*. Then, we evaluate Marich on different text and image data sets, and different models, including CNNs and BERT. Marich extracts models that achieve $\sim 60-95\%$ of true model's accuracy and uses $\sim 1,000 - 8,500$ queries from the publicly available datasets, which are different from the private training datasets. Models extracted by Marich yield prediction distributions, which are $\sim2-4\times$ closer to the target's distribution in comparison to the existing active sampling-based attacks. The extracted models also lead to 84-96$\%$ accuracy under membership inference attacks. Experimental results validate that Marich is *query-efficient*, and capable of performing task-accurate, high-fidelity, and informative model extraction.

Marich: A Query-efficient Distributionally Equivalent Model Extraction Attack using Public Data
Pratik Karmakar'
School of Computing, National University of Singapore CNRS@ CREATE Ltd, Create Way. Singapore pratik. karmakar@u. nus. edu
Debabrota Basu
Équipe Scool, Univ. Lille, Inria, CNRS, Centrale Lille UMR 9189- CRISIAL, F-59000 Lille, France
debabrota. basu@inria.fr debabrota.basuuimria..f
Abstract
We study design of black-box model extraction attacks that can send minimal number of queries from publicly available datasel to a target ML model through a predictive API with an aim to create an informative and distributionally equiv- alent replicu of the target. First, we define distributionally equivalent and Max- Information model extruction attacks, and reduce them into variational optimisa- tion problem. The attacker sequentially solves this optimisation problem to select the most informative queries that simultaneously maximise the entropy and reduce the mismatch between the target and the stolen models. This leads sampling- -aaegd---ddddd query selection algorithm, MARICH, which is model-oblivions Then, evaluate MARICH different text and image data sets, and different mod- els, including CNNs and BERT. MARICH extracts models that achieve ~ 60 95% of true model's accuracy and
active
0.0000 500 queries from the publicly available datasets, which are different from the private training datasets. Models extracted by MARICH yield prediction distributions, which the target's 's distribution in comparison to the existing active sampling- based attacks. The extracted models also lead to 84-96% accuracy under membership inference attacks. Experimental results validate that MARICH is query-efficient, and capable of performing task-accurate, high-fidelity, and informative model extraction.
4x closer
1 Introduction
In recent years, Machine Learning as a Service (MLaaS) is widely deployed and used industries. In MLaaS [RGC15), an ML model is trained remotely on private dataset, deployed in Cloud, and offered for public access through a prediction API, such as Amazon AWS Google API Microsoft Azure, An API allows an user, including potential adversary, to send queries to the ML model and fetch corresponding predictions. Recent works have shown such models with public APIs can be stolen, or extracted, by designing black-box model extraction attacks [TJ] "h 16). In model extraction attacks, an adversary queries the target model with a query dataset, which might be same or different than the private dataset, collects the corresponding predictions from the target model, and builds
A significant portion of the work has been done part of P. Karmakar's masters Ramakrishna Mission Vivekananda Educational and Research Institute, Belur, India. ** Code available at: Ihttt::// thub. com det.bbbbbbbboaaacooooo waterrtttaaaaaaarrraaeeeaa basu/marich
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

replica model of the target model. The goal is to construct a model which is almost-equivalenn to the target model over imput space [JBB+00
Often, MI models are proprietary, guarded by IP rights, and expensive to build. These models might be trained datasets which are expensive obtain (DY+19) and consist of private data of individuals [LM05]. Also, extracted models can be used to perform other privacy attacks on the private dataset used for training, such as membership inference [NSH19]. Thus, understanding susceptibility of models accessible through MLaaS presents an important conundrum. This motivates us to investigate black- model extraction attacks while the adversary has no access to the private data or perturbed version of it [PMG* 17]. Instead, the adversary uses public dataset to query the target model [OSF19
20).
Query-efficient black-box model ex- traction poses tension between the number of queries sent to the tar- get model and the accuracy of ex- tracted model [PGS With more queries and predictions, an adversary can build a better replica. But query- ing an API too much can be expen- sive. as each query incurs a monetary cost in MLaaS, Also, researchers have developed algorithms that can detect adversarial queries, when they are not well-crafted or sent to the API in large numbers [JMMA1 PGKS2I). Thus, attack is
Private training dataset D
Predicted labels Y,
Black-bux model I
Individuals
(Q,X)
Queries 92
Train curacted
Marich
Public Query dataset
Extracted model f
D9
Send D2 to f
designing a query-effiecte paramount for practical deployment. Also, it exposes how more informa- tion can be leaked from a target model with less number of interactions.
Figure 1: Black-box model extraction with MARICH.
In this paper, we investigate effective definitions of efficiency of model extraction and corresponding algorithm design for query-efficient black-box model extraction attack with public data, which is oblivious to deployed model and applicable for any datatype.
Our contributions. Our investigation yields three contributions.
1. Formalism:
Distributional equivalence and Max-Information extraction. Often, the ML models, specifically classifiers, are stochastic algorithms. They also include different elements of randomness during training. Thus, rather than focusing on equivalence of extracted and target models in terms of a fixed dataset accuracy on that dataset [JCB + 20). propose distributional notion of equivalence. We propose that if the joint distribution induced by a query generating distribution and corresponding prediction distribution due to both the target and the extracted models same, they will be called distributionally equivalent (Sec. Another proposal is to reinforce the objective of the attack, i.e. to extract as much information as possible from the target model. This allows us formulate the attack, where the adversary aims to maximise the mutual information between the extracted and target models' distributions. Our hypothesis that if we can extract the predictive distribution of a model, it would be enough to replicate other properties of the model (e.g. accuracy) and also run other attacks (e.g. membership inference). rather than designing specific altacks to replicate the task accuracy or the model weights. We show that both the attacks can be performed by sequentially solving a single variational optimisation (SB12 problem (Eqn.
Algorithm: Adaptive query selection for extraction with MARICH. We propose an algorithm. MARICH (Sec. 4) that optimises the objective of the variational optimisation problem (Eqn. (
Given an extracted model, a target model, and previous queries, MARICH adaptively selects a batch of queries enforcing this objective. Then, it sends the queries the target model, collects the predictions (i.e. the class predicted by target model), and uses them to further train the extracted model (Algo. [ In order to select the most informative set of queries. it deploys three sampling strategies in cascade. These strategies select: a) the most informative set of queries, b) the most diverse set of queries in the first selection, and e) the final subset of queries where the target and extracted models mismatch the most. Together these strategies allow MARICH to select a small subset of queries that both maximise the information leakage, and align the extracted and target models (Fig. TT.
2

---

3. Experimental analysis. We perform extensive the most for given modelevaluation with both image and text datasets. and diverse model classes, such as Logistic Regression (LR), ResNet, CNN, and BERT (Sec. 5 Leveraging MARICH's 's model-obliviousness we even extract a ResNet trained on CIFARTO 10 with a CNN and out-of-class queries from ImageNet. Our experimental results validate that MARICH extracts more accurate replicas of the target model and high-fidelity replica of the target's prediction distributions in comparison to existing active sampling algorithms. While MARICH uses a small number of queries (~ 1k 8.5k) selected from publicly available query datasets, the extracted models yield accuracy comparable with the target model while encountering a membership inference attack. This shows that MARICH can extract alarmingly informative models query-efficientl. Additionally, as MARICH can extract the true model's predictive distribution with a different model architecture and mismatched querying dataset, it allows us to design model-oblivious and dataset-oblivious approach to attack.
1.1 Related works
Taxonomy of model extraction. Black-box model extraction (or model stealing or model inference) attacks aim to replicate of target ML model, commonly classifiers, deployed in a remote service and accessible through a public API [TZJ+ 16]. The replication is done in such way that the extracted model achieves one the three goals: a) accuracy close to that of the target model on the private training data used to train the target model, b) maximal agreement in predictions with the target model on the private training data, and c) maximal agreement in prediction with the target model the objective, they called task accuracy. fidelity.
over the whole input domain. Depending and functional equivalence model extractions. respectively JCB+20) Here, we generalise these three approaches using novel definition of distributional equivalence and also introduce novel coormattin-theoreer objective of model extraction which maximises the mutual information between the target and the extracted model over the whole data domain.
Frameworks of attack design. Following TZ]+ researchers have proposed multiple attacks to perform one of the three types of model extraction. The attacks are based on two main approaches: direct recovery (target model specific) [MSDH19 BBJP18 JCB+20) and learning (target model specific/oblivious). The learning- -ased approaches can also be categorised into supervised learning strategics, where the adversary has access to both the true labels of queries and the labels predicted by the target model (TZ) 16 JCB + 20]. and online active learning strategies, where the adversary has only access to the predicted labels of the target model, and actively select the future queries depending on the previous queries and predicted labels [PMG+ 17 PGS CCG+ (C)+20). As query-efficient is paramouns for an adversary while attacking an API to save the budget and to keep the attack hidden and also the assumption of access true label from the private data is restrictive, we focus on designing an online and active learning- -based attack strategy that is model oblivious.
CCG+ 20 focus on performing attacks against linear models, all others are specific to neural networks [SDH JCB+ PGS+ and even a specific architecture (CSBB* 18). In contrast, MARICH is capable of attacking both linear models and neural networks. Additionally, MARICH is model- -obliv-oblivious i.c. it can attack one model architecture (c.g. ResNet) using diffferent model architecture (e.g. CNN).
Types of target model. While [MDHI9
Types of query feedback. Learning-bart based attacks often assume access to either the probability vector of the target model over all the predicted labels 16 OSF19 PGS 20, of the last layer of the target neural network (MDDH19 MHS21], which are hardly available in a public API. In contrast, following [PMG 17), we assume access to only the predicted labels of the targel model for a set of queries, which is always available with a public API. Thus, experimentally, we cannot compare with existing active sampling attacks requiring access to the whole prediction vector [PGS 20 OSF19], and thus, compare with a wide-range of active sampling methods that can operate only with the predicted label, such K-center sampling, entropy sampling, least confidence sampling, margin sampling etc. 2X Details are in Appendix C
or the gradient
Choices of public datasets for queries. There are two approaches of querying target model: data-free and data-selection based. In data-fre free attacks, the attacker begins with noise. The informative queries are generated further using GAN-like model fed with responses obtained from an API [ZWL. 20 TMWP21 MHS21 22 SAB22]. Typically, it requires almost a million queries to the API to start generating sensible query data (e.g. sensible images that can leak from a model trained on CIFARIO). But since one of our main focus is query-efficiency, we focus on data-selection based attacks, where an adversary has access to query dataset to select the queries
3

---

from and to send it to the target model to obtain predicted labels. In literature, researchers assume three types of query datasets: synthetically generated samples [TZJ* 16), adversarially perturbed private (or lask domain) dataset 17 [SMA19] and publicly available (or out-of-takk domain) dataset [OSF19, PGS+20 As we do not want to restrict MARICH to have access to the knowledge of the private dataset or any perturbed version of it, we use publicly available datasets, which are different than the private dataset. To be specific, we only assume whether we should query the API with images, text, or tabular data and not even the identical set of labels. For example, we experimentally attack models trained on CIFARIO IO with ImageNet queries having different classes.
Further discussions on related active sampling algorithms and distinction of MARICH with the existing works are deferred Appendix [C]
2 Background: Classifiers, model extraction, membership inference attacks
Before proceeding to the details, we present the fundamentals of a classifier in MI., and two types of inference attacks: Model Extraction (ME) and Membership Inference (MI).
Classifiers. A classifier in ML [BBCB[6] is a function f X' + ) that maps set of input features X E X to an output Y E y The output space is a finite set of classes, i.e. {1..... R.). Specifically, classifier ʃ is a parametric function, denoted as fo. with parameters 0 IRd, and is trained on dataset D² i.e. a collection of n tuples {(x; vi) ) generated IID from an underlying distribution }), a loss function 1 3 y R2o. D. Training implies that given a model class J = [So]o E
and training dataset D", we aim to find the optimal parameter 6 - arg mingge Ei= 1(fo (x:). v.).
We use cross-entropy, i.e. I(fo(x;), v1) y (o:())(x))). as the loss function for classification.
Model extraction attack. A model extraction attack is an inference attack where an adversary aims to steal a target model f trained on a private dataset D and create another replica of it f [TZJ 16). In the black-box setting that are interested in, the adversary can only query the target model f by sending queries Q through publicly available API and to use the corresponding predictions y to construct f The goal of the adversary is to create a model which is either (a) as similar to the target model possible for all input features, i... (z() labels that has maximal agreement with that of the labels predicted by the target model for a given data- generating distribution, i.e. f arg min "1 (x), f (x))] [T]] 16 PGS+ 20 JCB+20) The first type of attacks are called the functionally equivalent attacks. The later family of attacks is referred as the fidelity extraction attacks. The third type of attacks aim to find an extracted model f that achieves maximal classification accuracy for the underlying private dataset used to train the f'r' These are called task accuracy extraction attacks [TI+ 16 MSDH19 OSF19]. In this paper, we generalise the first two type of attacks by proposing the distributionally equivalent attacks and experimentally show that yields both task accuracy and fidelity.
(z) Vi E X [SS20 CCG + 20) or (b) predicts
Membership inference attack. Another popular family of inference attacks on ML models is the Membership Inference (MI) attacks [SSSS17] YGEIT8] In MI attack, given a private (or member) dataset D to train f and another non-member dataset S with [D T n S zf 6, the goal of the adversary is to infer whether any X' is sampled from the member dataset D or the non-member dataset S. Effectiveness of an MI attacks can be measured by its accuracy of MI, i.e. the total fraction of times the MI adversary identifies the member and non-mebeer data points correctly. Accuracy of MI attack on the private data using f rather than f is considered as measure of effectiveness of the extraction attack [NSH19] We show that the model 1 extracted using MARICH allows us to obtain similar MI accuracy as that obtained by directly attacking the target model fr using even larger number of queries. This validates that the model f by MARICH in a black-box setting acts as an information equivalent replica of the target model fr.
3 Distributional equivalence and Max-Information model extractions
In this section, we introduce the distributionally equivalent and We further reduce both the attacks into a variational optimisation problem.
model extractions.
Definition 3.1 (Distributionally equivalent model extraction). For any query generating distri- bution Do over R. V, an extracted model f RR + Y is distributionally equivalent to target
We denote sets/vectors by bold letters, and the distributions by calligraphic letters. We express random variables in UPPERCASE, and assignment of random variable in lowercase.

---

model ff, Id + Y, the joint distributions of input features Q E IR ~ DD and predicted labels induced by both the models are same almost surely. This means that for any divergence D, two distributionally equivalent models 1e and f satisfy (Q), Q)Il Pr(s)((), Q)) = 4'4 DD.
To ensure query-efficiency in distributionally equivalent model extraction, an adversary aims to choose query generating distribution D' that minimises it further. If we assume that the extracted model is also a parametric function, i.e. far with parameters w E Q2 we can solve the query-efficient distributionally equivalent extraction by computing
(WDEqq Doo - arg min arg min
Q))
well
D2
Equation allows us to choose a different class of models with different parametrisation for extraction till the joint distribution induced by it matches with that of the target model. For example. the extracted model can be a logistic regression or a CNN if the target model is a logistic regression. This formulation also enjoys the freedom choose the data distribution D4 for which we want to test the closeness. Rather the distributional equivalence pushes us to find the best query distribution for which the mismatch between the posteriors reduces the most and to compute an extracted model fur that induces the joint distribution closest to that of the target model fø.
Connection with different types of model extraction. For D DRI. our formulation extends the fidelity extraction from label agreement to prediction distribution matching, which addresses the future work indicated by [JCB+20) If we choose D = D" and substitute D by prediction agreement, distributional equivalence retrieves the fidelity extraction attack. If choose min U2), distributional equivalent extraction coincides with functional equivalent extraction. Thus, distributional equivalence attack can lead to both fidelity and functional equivalence extractions depending on the choice of query generating distribution D& and the divergence D.
Theorem 3.2 (Upper bounding distributional closeness). we choose KL-divergence as the diver- gence function D, then for a given query generating distribution D
(KL (Q), Q)I Pr(fE (Q), Q)) S min BuinEq/(fft (Q) - (Q))
(2)
By variational principle, Theorem 3.2 implies that minimising the upper bound on the RHS leads to an extracted model which minimises the KL-divergence for chosen query distribution.
Max-Information model extraction. Objective of any inference attack is to leak as much information as possible from the target model f". Specifically. in model extraction attacks, we want to create an informative replica f of the target model f" such that it induces joint distribution Pr(f (Q), Q), which retains the most information regarding the target's s joint distribution. As adversary controls the query distribution, we aim choose a query distribution DR that maximises information leakage.
Definition 3.3 Inaormanoootioonn model extraction). A model f Rd - Y and a query dis- tribution DQ are called a Informattttno query distribution, respectively, if they maximise the mutual information between the joint distributions of input features Q E IR DD and predicted labels induced by f and that of the target model. Mathematically, (f DQ m is Informmtnnno extraction of fø- if
extraction of a target model f" : Rd + Y and a
(w Do max
A arg max arg max I(Pr Q) Pr(S Q))
(3)
w Do
Similar to Definition 3.1 Definition 3.3 also does not restrict us choose parametric model w different from that of the target 0. It also allows us to compute the data distribution D for which the information leakage is maximum rather than relying on the private dataset D' used for training f
Theorem 3.4 (Lower bounding information leakage). For any given distribution Da, the information leaked by any Max- Ino-Innoonntion attack ( (Equation) 3) is lower bounded as:
I(Pr(ST- (Q), Q)I Pr Wi
(Q),)) > max (E (Q)) + (Q)
(4)
By variational principle, Theorem 3.4 implies that maximising the lower bound in the RHS will lead to an extracted model which maximises the mutual information between target and extracted joint distributions for given query generating distribution.
5

---

Distributionally equivalent and Max-Information extractions: A variational optimisation formulation. From Theorem 3.2 and 3.4 observe that the lower and upper bounds of the objective functions of distribution equivalent and Max-Information attacks are negatives of each other. Specifically, ( (Q), Pr ADz. (Q); Q)) 2 max_ (w, DQ) and
T(Pr(J) (Q), Whaxdaa (Q):)) 2 max F(w, DQ). where
F(w, DQ, -Eol (Q)) H
(5)
Thus, following a variational approach, we aim solve optimisation problem F(w, DQ) in an online and frequentist manner. We do not assume a parametric family of D2, Instead, we choose set of queries Qt E I2 at each round E T. This leads an empirical counterpart of our problem:
T Σ 1=1
F(w, Qp,,1) A
Σ HC (6)
max we D
max w.R](o.]]
-T
As we need to evaluate fō- for each Qs we refer Qt's as queries, the dataset D C Rdd x y from where they are chosen as the query dataset, and the corresponding unobserved distribution DQ as the query generating distribution. Given the optimisation problem of Equation we propose an algorithm MARICH solve it effectively.
4 Marich: A query selection algorithm for model extraction
In this section, we propose algorithm, MARICH, solve Equation (6) in adaptive manner.
Algorithm design. We observe that once the queries Q 0, are selected, the outer maximisation problem of Eq. (6) is equivalent to regualrised loss minimisation. Thus, it can be solved using any standard empirical risk minimisation algorithm (e.g. Adam, SGD). Thus, to achieve query efficiency, we focus on designing query selection algorithm that selects batch of queries Qr at round / < T:
t-1
[-1
Qt - arg max
Σ
Uf (Q.
+
Σ HUE
(7)
QED t
1=1
Model mismatch
Entropy term
Algorithm MARICH
Input:
Target model: fr, Query dataset: D2, #Classes: k
Parameter: #initial samples: nog Training epochs: Emazx # #Bacches of queries: T, Query budget:
B, Subsampling ratios: ?,,- 2/2 E (0,
Output: Extracted model f
1: 1/s Initialisation of the extracted model
D Phase 1
2: Qiaain + no datapoints randomly chosen from DQ
3: Yfrain <- fd (Qurain)
Query the target model f with Qtrain
4: for epoch 4- to Fmaz do
5: fo <- Train f with (Qtrain Y!rain)
6: end for
7: l/a Adaptive query selection to build the extracted model*//
D Phase
8: for t 4- 1 to do
9: Qentropy +- EXTROPYSAMPLINGJJE,,DD
Qtrain B)
Qarad + EXTROPYGR.DDENSS..PPLNN Qentrops B)
10:
11:
Qloss +- LOSSSAMPLING((EE, Qrrad Qirain Y B)
12:
Ynew {T (Q)008)
Query the target model fr with Q
13:
Qtrain +- Qirain U Qloss
14: ytrain Ytrrin UYyee
15: for epoch + to Emaz do
16: f.P Train f. with (Qraain ytrain)
17: end for
18: end for
19: return Extracted model f + ff

---

Here, far is the model extracted by round t 1. Equation indicates two criteria to select the queries. With the entropy term, we want to select a query that maximises the entropy of predictions for the extracted model - This allows us select the queries which are most informative about the mapping between the input features and the prediction space. With the model-mismatel. term, Eq. 7 pushes the adversary to select queries where the target and extracted models mismatch the most. Thus, minimising the loss between target and extracted models for such a query forces them to match over the whole domain. Algorithm Iillustrates a pseudocode of MARICH (Appendix (A).
select a set of no queries, called Qraan uniformly randomly from the query dataset D We send these queries to the target model and collect corresponding predicted classes Ygrain (Line 3). We use these no samples of input- predicpreddtted label pairs to construct a primary extracted model fo"
Initialisation phase. To initialise the extraction,
Active sampling. As the adaptive sampling phase commences, we select 71 number of queries at round l. To maximise the entropy term and minimise the model-mismatch term of Eq. (7). we sequentially deploy ENTROPYS A APPLING and LossS AMPLING To achieve further query-efficiency, we refine the queries selected using ENTROPY AMMLIING by ENTROPYGRA GRADIENTDDSSS SANTTAMMPING which finds the most diverse subset from a given set of queries. Now, we describe the sampling strategies.
First, we aim to select the of queries which unveil most information about the mapping between the input features and the prediction space. Thus, we deploy ENTROPYSAME PLING. In ENTROPYSAMPLING we compute the output probability vectors from f for all the query points D인 \ Q and then select top points with highest entropy:
Qentropy
arg max XX
(Xim).
Thus, we select the queries Q about which f is most confused and training on these points makes the model more informative.
ENTROPPYRADDIE AMPLING. To be frugal about the number of queries, we refine Q to compute the most diverse subset of it. First, we compute the gradients of entropy of f (x). iee. (JH(EE for all x E Q Quntrovy The gradient at point x reflects the change at x in the prediction distribution induced by _ We use these gradients to embed the points E Q Now, we deploy K-means clustering to find (= #classes) clusters with centers Cin. Then, we sample 71 B points from these clusters:
Qarad
arg min
Σ
Σ IV
V V,,
XcQretrope [X|=m B x.E. 2,ECio
Selecting from & clusters ensures diversity of queries and reduces them by 7/-
LOSSSAMPLING.. We select points from Qorad for which the predictions of f and fE, are most dissimilar. To identify these points, we compute the loss ((f"(x), ffx(x)) for all x E Qirain Then, we select top-k points from Qt with the highest loss values (Line 11), and sample subset Q of size 7/2 B from Qarad which are closest to the l points selected from Qirain This ensures that fE would better align with f if trains on the points where the mismatch in predictions are higher.
At the end of Phase in each round of sampling, Q/ is sent to fo for fetching the labels Y,train predicted by the target model. We use (Qlese, Y!oos)) along with (Q ylrain) to train f further. Thus, MARICH performs no + 7=?2 BT number of queries through T + number of interactions with the target model fr to create the final extracted model f.F. We experimentally demonstrate effectiveness of the model extracted by MARICH to achieve high task accuracy and to act as an informative replica of the target for extracting private information regarding private training data D
Discussions. Eq. 7 dictates that the active sampling strategy should try to select queries that max- imise the entropy the prediction distribution of the extracted model, while decreases the mismatch in predictions of the target and the extracted models. We further use the PLING to choose a smaller but most diverse subset. As Eq. 77) does not specify any ordering between these objectives, one can argue about the sequence of using these three sampling strategies. We choose to use sampling strategies the decreasing order of runtime complexity as the first strategy selects the queries from the whole query dataset, while the following strategies work only on the
AAM-
7

---

Target
Marchh
KC
MS
25 29 15 195 125 1550 199
333 50 200 200
030 2033
300 1000 201 3XX 1oc 713 OXTT
(a) LR/EMNIST query
(b) LR/CIFARIO query (c) ResNet/ImgNet query (d) CNNTEMNIST query
Figure 2: Accuracy of the extracted models (mean + std. over 10 runs) w.r.t. the target model using MARICH, and competing active sampling methods (KC, LCC, MS, ES, RS). Each figure represents (a target model, a query dataset). Models extracted by MARICH are closer to the target models.
already selected queries. We show in Appendix [ that AMPLING incurs the highest runtime ADIENTS
followed by
while ENTROPY AMPPLING is significantly cheaper. Remark 4.1. Previously, researchers have deployed different active sampling methods efficiently select queries for attacks [PMG* 17 CCG 20 +20) But our derivation shows that the active query selection can be grounded on the objectives of distributionally equivalent and max- -nx-oofmrnttion extractions. Thus, though the end result of our formulation, i.e. MARICH, is an active query selection algorithm, our framework is different and novel with respect to existing active sampling based works.
5 Experimental analysis
Now, we perform an experimental evaluation of models extracted by MARICH. Here, we discuss the experimental setup, the objectives of experiments, and experimental results. We defer the source code, extended results, parametric similarity of the extracted models, effects of moiel-mismatch, details of different samplings, and hyperparameters to Appendix.
Experimental setup. We implement prototype of MARICH using Python 3.9 and PyTorch 1.12, and run on a NVIDIA GeForce RTX 3090 24 GB GPU. We perform attacks against four target models (J²)), ), namely Logistic Regression (LR), CNN [LBHI5), ResNet (HZRS16), BERT DCLT18). trained on three private datasets (D"); MNIST handwritten digits [Dn 12,, CIFARIO 0K
and BBC News) respectively. For model extraction, we use EMNIST letters dataset [CAvvS17]. CIFARIO, ImageNet [DDS+09) and AGNews [ZZL15), as publicly-available, mismatched query datasets D
To instantiate task accuracy, we compare accuracy of the extracted models f MMAIICH with the target model and models extracted by K-Center (KC) [SSI3). Ceatt-Confidence sampling (LC) (LS06). Margin sampling (MS) [BZ07 JG19, Entropy Sampling (ES) [G94], and Random Sampling (RS). To instantiate informativeness of the extracted models [NSH19] we compare the Membership Inference (MI), i.e. MI accuracy and MI agreements (% and AUC), performed on the target models, and the models extracted using MARICH and competitors with same query budget. For MI, use in-built membership attack from IBM ART [NST* 18). For brevity, we discuss Best of Competitors (BoC) against MARICH for each experiment (except Fig. 2 The objectives of the experiments are:
How do the accuracy of the model extracted using MARICH on the private dataset compare with that of the target model, and RS with same query budget?
How close are the prediction distributions of the model extracted using MARICH and the target model? Can MARICH produce better replicu of target's 's prediction distribution than other active sampling methods, leading to better distributional equivalence?
3. How do the models extracted by MARICH behave under Membership Inference (MI) in comparison to the target models, and the models extracted by RS with same budget? The MI accuracy achievable by attacking a model acts as a proxy of how informative is the model.
4. How does the performance of extracted models change if Differentially Private (DP) mecha- nisms (DMNS06) are applied on target model either during training or while answering the queries?
Accuracy of extracted models. MARICH extracts LR models with 1,863 and 959 queries selected from EMNIST and CIFARIO. 10, while attacking a target LR model, fiK trained on MNIST (test accuracy: 90. (2.8. The models extracted by MARICH using EMNIST and CIFAR10 10 achieve test accuracy 73.988 98% and 86.83% (81.46% and 95.60% of f(x) respectively (Figure 2a 2bp. The models extracted using BoC show test accuracy 52.60% and 79.09% (57.91% and 87.08% of f(R) i.c.
8

---

Moss
10
300 25
(a) LR with EMNIST (b) ResNet with ImageNet
(a) DP-SGD train target (b) Perturb oulput of query
Figure 3: Comparing fidelity of the prediction dis- tributions (in log scale) for different active learn- ing algorithms. MARICH achieves 2 - 4x lower Kivereeree than others.
Figure 4: Comparing test accuracy of the mod- els extracted by MARICH against different DP mechanisms (DP-SGD and Output Perturbation) applied on the target model.
Table 1: Statistics of accuracy & membership inference (MI) for different target models, datasets & attacks. 4_ means member dataset and target model used. #BoC means Best of Competitors.
Memberr dataset
model
Algoeitlun
Non Non-member
+Queries
MI
MI
agvemment AUC
MINIST 13
90883
MINIST
EMNIST
MARICH
EMSIST
84.27%
90.34% ROTTS
90.8%%
MINIST
13
EMNIST
EMNIST
780076
53,07%
MNNSST
1X
CIFARTO
MNIST
1R
CIFARIO
MARICH
CIFARIO
(1.92%)
96.32%
96.89
94.32%
81.6%%
MNIST
LR
CIFARI
BOC
CIFARIO
(1329)
93.70%
93.67%
91.35%
77.95%
MNIST
UNN
EMNIST
50.97%
MINIST
USN
EMNIST
MARCC
EMNIST
87.2%%
16.71%
36335
MINIST
UNN
EMSIST
BOC
EMSIST
50.33
07,331
86.975
CHARTI
EMSIST
CHARII
REaNe
ImageNe:
MARICH
EMNIST
1429
(163895
90.40%
03.849
76.31%
56.199
CHARTO
KERNeT
ImageNer
BOLY
EMKIST
14.5 (16399
50008
95.49
723975
BECNNDS
BERT
1.0001003
BECNDI
BERT
MARICH
AUN
91.0%
BECNSKS
BERT
NUNcws
BOC
AUNews
50,415
significantly less than that of MARICH. MARICH attacks a ResNet, fResMer trained on CIFARIO (test accuracy: 91.82%) with 8,429 queries from ImageNet dataset, and extracts CNN. The extracted CNN shows 56.1 % (61.10% of JeeeNee) test accuracy. But the model extracted using BoC achieves 42.05% (45.79% of Net) accuracy (Figure 2 We also attack a CNN with another CNN, which also reflects MARICH'S 's improved performance (Figure 2d.. To verify MARICH'S 's effectiveness for text data, we also attack a BERT, f ET trained on BBCNews (test accuracy: 98.65%) with queries from the AGNews dataset. By using only 474 queries, MARICH extracts model with 85.45 85.45% (86.64% of J'ÉERT) test accuracy. The model extracted by BoC shows test accuracy 79.25% (80.36% of [T' ( For all the models and datasets, MARICH extracts models that achieve test accuracy closer target models, and more accurate than models extracted by the other algorithms.
Distributional equivalence of extracted models. One of our aims is to extract a distributionally equivalent model of the target f using MARICH. Thus, in Figure 3 we illustrate the KL-divergence ist. over runs) between the prediction distributions of the target model and the model extracted by MARICH. Due to brevity, we show two cases in the main paper: when we attack i) an LR trained on MNIST with EMNIST with an LR, and ii) a ResNet trained on CIFARIO with ImageNet with CNN. In all cases, we observe that the models extracted by MARICH achieve 2 4x lower KL-divergence than the models extracted by all other active sampling methods. These results show that MARICH is extracts high-fidelit distributionally equivalent models than competing algorithmss
Membership inference with extracted models. In Table D we report accuracy, agreement in inference with target model, and agreement AUC of membership attacks performed on different target models and extracted models with different query datasets. The models extracted using MARICH demonstrate higher MI agreement with the target models than the models extracted using its competitors most of the cases. They also achieve MI accuracy close to the target model. These results indicate that the models extracted by MARICH act as informative replicas of the target models.
Performance against privacy defenses. We test the impact of DP-based defenses deployed in the target model on the performance of MARICH. First, we train four target models on MNIST using DP-SGD 16) with privacy budgets E {0.2, 0.5, oc} and 8 10 As illustrated in Figure 4a accuracy of the models extracted by querying DP target models are ~ 2.3 - 7.4% lower than the model extracted from non-private target models. Second, we apply an output perturbation method [DMNS06), where a calibrated Laplace noise is added to the responses of the target model against MARICH'S queries. This ensures eDD for the target model. Figure 4b shows that perfor- mance of the extracted models degrade slightly for 8 = 2, 8, but significantly for & 0.25. Thus, performance of MARICH decreases while operating against DP defenses but the degradation varies depending the defense mechanism.

---

Summary of results. From the experimental results, we deduce the following conclusions. 1. Accuracy. Test accuracy (on the subsets of private datasets) of the models f MMRRCIU are higher than the models extracted with the competing algorithms, and are ~ 60 - 95% of the target models (Fig. 2 This shows effectiveness of MARICH as a task accuracy extraction attack, while solving distributional equivalence and max-info extractions.
2. Distributional equivalence. We observe that the KL-divergence between the prediction distributions of the target model and f algorithms. This confirms that MARICH conducts more accurate distributionally equivalent extraction than existing active sampling attacks.
MARRCH are ~ 2-4x lower than the models extracted by other active sampling
3. Informative replicas: Effective membership inference. The agreement in MI achieved by attacking E MARRCH and the target model in most of the cases is higher than that of the BoC* (Table 1). Also, MI accuracy for f MMARICH's 's are 84.74% 96.32% (Table lp. This shows that the models extracted by MARICH act as informative replicas of the target model.
Query-effciiecy. Table ] shows that MARICH uses only 959 8, 429 queries from the public
datasets, i.e. a small fraction of data used to train the target models. Thus, MARICH is significantly query efficient, as existing active learning attacks use 10k queries to commence PPGS 20.
Table 2].
5. Performance against defenses. Performance of MARICH decreases with the increasing level of DP applied on the target model, which is expected. But when DP-SGD is applied to train the target, the degradation is little (~~ 7%) even for 5 = 0.2. In contrast, the degradation is higher when the output perturbation is applied with similar E (0.25).
Model-obliviouseess and out-of out-of-class data. By construction, MARICH is model-oblivious and can use out-of-class public data to extract target model. To test this flexibility of MARICH, we try and extract a ResNet trained on CIFAR 10 using a different model, i.e. CNN, and out-of-class data, i.e. ImageNet. We show CNNs extracted by MARICH are more accurate, distributionally close, and also lead to higher MI accuracy that the competitors, validating flexibility of MARICH.
7. Resilience to mismatch between D and D For MARICH, the datasets D and DC be significantly different. For example, we attack an MNIST-trained model with EMNIST and CIFARIO as query datasets. MNIST contains handwritten digits, CIFARIO contains images of aeroplanes, cats, dogs etc., and EMNIST contains handwritten letters. Thus, the data generating distributions and labels significantly different between the private and query datasets. We also attack CIFAR ResNet with ImageNet as D CIFAR10 10 and ImageNet are also known to have very different labels and images. Our experiments demonstrate that Marich can handle data mismatch as well as model mismatch, which is an addendum to the existing model extraction attacks.
Conclusion and future directions
We investigate the design of a model extraction attack against a target MI. model (classifier) trained on private dataset and accessible through a public API. The API returns only predicted label for a given query. We propose the notions of distributional equivalence extraction, which extends the existing notions of task accuracy and functionally equivalent model extractions. We also propose an information-theone theoretic notion, i.e. Max-Info model extraction. We further propose a variational relaxation of these two types of extraction attacks, and solve it using an online and adaptive query selection algorithm, MARICH. MARICH uses publicly available query dataset different from the private dataset. We experimentally show that the models extracted by MARICH achieve 56 86% accuracy on the private dataset while using 959 8,429 queries. For both text and image data, we demonstrate that the models extracted by MARICH act as informative replicas of the target models and also yield high-fidelity replicas of the targets' prediction distributions. Typically, the functional equivalence attacks require model-specific techniques, while MARICH is model-oblivious while performing distributional equivalence attack. This poses an open question: is distributional equivalence extraction 'easier' than functional equivalence extraction, which NP-hard [JCB 20
Acknowledgments
P. Karmakar acknowledges supports of the programme DesCartes and the National Research Founda- tion, Prime Minister's Office, Singapore under its Campus for Research Excellence and Technological Enterprise (CREATE) programme. D. Basu aeknowledges the Inria-Kyoto University Associate Team "RELIANT". and the ANR JCJC for the REPUBLIC project -AN--22-CF2300000-00) for support. We also thank Cyriaque Rousselot for the interesting initial discussions.
10

---

References
[ACG* 16] Martin Abadi, Andy Chu, lan Goodfellow, H Brendan McMahan, llya Mironov, Kunal Talwar, and Li Zhang. Deep learning with differential privacy. In Proceedings of the 2016 ACM SIGSAC conference on computer and communications security, pages 2016.
[BBJP18] Lejla Batina, Shivam Bhasin, Dirmanto Jap, and Stjepan Picek. CSI neural network: Using side-channels to recover your artificial neural network information. CoRR, abs/1810.09076. 2018.
[BBZ07] Maria-Florina Balcan, Andrei Broder, and Tong Zhang, Margin based active learning. In Learning Theory: 20th Annual Conference on Learning Theory, COLT 2007, San Diego CA, USA; June 13-15, 2007. Proceedings 20, pages 35-50. Springer, 2007.
[CATvS17] Gregory Cohen, Saeed Afshar, Jonathan Tapson, and André van Schaik. Emnist: Extending mnist to handwritten letters. In 2017 International Joint Conference on Neural Networks (IJCNN), pages 2921-2926, 2017.
[CCG * 20] Varun Chandrasekaran, Kamalika Chaudhuri, Irene Giacomelli, Somesh Jha, and Song- bai Yan. Exploring connections between active learning and model extraction. In 29th USENIX Security Symposium (USENIX Security 20), pages 1309-1326, 2020.
[CMS11] Kamalika Chaudhuri, Claire Monteleoni, and Anand D Sarwate. Differentially private empirical risk minimization. Journal of Machine Learning Research, 12(3), 2011.
[CSBB* 18] Jacson Rodrigues Correia-Silva, Rodrigo F Berriel, Claudine Badue, Alberto F de Souza, and Thiago Oliveira-Santos Copycat cnn: Stealing knowledge by persuading confession with random non-labeled data. In 2018 International Joint Conference on Neural Networks (IJCNN, pages 8. IEEE, 2018.
[DBB18] Ashish Dandekar, Debabrota Basu, and Stéphane Bressan. Differential privacy for regularised linear regression. In Database and Expert Systems Applications: 29th International Conference, DEXA 2018, Regensburg, Germany, September 3-6, 2018, Proceedings, Part II, pages 483-491. Springer, 2018.
[DBB21] Ashish Dandekar, Debabrota Basu, and Stéphane Bressan. Differential privacy at risk: Bridging randomness and privacy budget. Proceedings Privacy Enhancing Technologies, 1:64-84, 84, 2021.
[DCLT18] Jacob Devlin, Ming-Was Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre- training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810. 04805, 2018.
[DDS 09] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. 2009 IEEE conference on computer vision and pattern recognition, pages 248-255. leee, 2009.
[Den12] Li Deng. The mnist database of handwritten digit images for machine learning research. IEEE Signal Processing Magazine, 29(6):141-144) 2012.
[DMNS06] Cynthia Dwork, Frank McSherry. Kobbi Nissim, and Adam Smith. Calibrating noise to sensitivity in private data analysis. In Theory of cryptography conference, pages 265-286 Springer, 2006.
[GBCB16] Ian Goodfellow, Yoshua Bengio, Aaron Courville, and Yoshua Bengio. Deep learning, volume 1. MIT Press, 2016.
[Hua21] Kuan-Hao Huang. Deepal: Deep active learning in python. arXiv preprint arXiv:2111.15255 2021.
[HZRS16] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEFE conference on computer vision and pattern recognition, pages 770-778, 2016.

---

[JCB*20] Matthew Jagielski, Nicholas Carlini, David Berthelot, Alex Kurakin, and Nicolas Papernot. High accuracy and high fidelity extraction of neural networks. In 29th USENIX security symposium (USENIX Security 20), pages 1345-1362, 2020.
[JG19] Heinrich Jiang and Maya Gupta. Minimum-mann margin active learning. arXiv preprint arXiv: 3XXiv:9906.0025. 2019.
[JSMA19] Mika Juuti, Sebastian Szyller. Samuel Marchal, and N Asokan. Prada: protecting against dnn model stealing attacks. In 2019 IEEE European Symposium on Security and Privacy (EuroS&P), pages 512-527. IERE, 2019.
[KDR 21] Krishnateja Killamsetty, Sivasubramanian Durga, Ganesh Ramakrishnan, Abir De, and Rishabh lyer. Grad-match: Gradient matching based data subset selection for efficient deep model training. In International Conference on Machine Learning, pages 1464-5474 PMLR, 2021.
[KH+09] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
[KS22] Yeachan Kim and Bonggun Shin. In defense of core-set: A density-aware core-set selection for active learning. Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 604-812, 2022.
[KZC121] Krishnateja Killamsetty, Xujiang Zhao, Feng Chen, and Rishabh lyer. Retrieve: Core- set selection for efficient and robust semi- cuni-supee learning. Advances in Neural Information Processing Systems, 34:14488 14501, 2021.
[LBH15] Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. 521(7553):436 444, 2015.
Deep learning. nature,
[LG94] DD Lewis and WA Gale. A sequential algorithmfor training text classifiers. In SI- GIR'94: Proceedings of the Seventeenth Annual International ACM-SIGIR Conference on Research and Development in Information Retrieval, organised by Dublin City University, pages 3-12, 1994.
[LM05] Daniel Lowd and Christopher Meek. Good word attacks on statistical spam filters. In CEAS, volume 2005, 2005.
[LS06] Mingkun Li and Ishwar K Sethi. Confidence-based active learning. IEEE transactions on pattern analysis and machine intelligence, 28(8):1251-126) 2006.
[MBL20] Baharan Mirzasoleiman, Jeff Bilmes, and Jure Leskovec. Coresets for data-efficient training of machine learning models. In International Conference on Machine Learning. pages
PMLR, 2020.
[MHS21] Takayuki Miura, Satoshi Hasegawa, and Toshiki Shibahara. Megex: Data-free model ex- traction attack against gradient- based explainable ai. arXiv preprint arXiv:2107.08909, 2021.
[MSDH19] Smitha Milli, Ludwig Schmidt, Anca D Dragan, and Moritz Hardt. Model reconstruction from model explanations. In Proceedings of the Conference on Fairness, Accountability, and Transparency, pages 1-9, 2019.
[NSH19] Milad Nasr, Reza Shokri, and Amir Houmansadr. Comprehensive privacy analysis of deep learning: Passive and active white-box inference attacks against centralized and federated learning. In 2019 IEEE symposium on security and privacy (SP), pages 739-753. IEEE, 2019.
INST+ 18] Maria- Nicolae, Mathieu Sinn, Minh Ngoc Tran, Beat Buesser, Ambrish Rawat, Martin Wistuba, Valentina Zantedeschi, Nathalie Baracaldo, Bryant Chen, Heiko Lud- wig, Ian Molloy, and Ben Edwards. Adversarial robustness toolbox v1.2.0. CoRR, 1807.01069, 2018
12

---

[OSF19] Tribhuvanesh Orekondy, Bernt Schiele, and Mario Fritz. Knockoff nets: Stealing functionality of black-box models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 4964, 2019.
[PGKS21] Soham Pal, Yash Gupta, Aditya Kanade, and Shirish Shevade. Stateful detection of model extraction attacks. arXiv preprini arXiv:2107.05 2021.
[PGS*20] 20] Soham Pal, Yash Gupta, Aditya Shukla, Aditya Kanade, Shirish Shevade, and Vinod Ganapathy. Activethief: Model extraction using active learning and unannotated public data. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 665-872.70 2020.
[PMG+17] 17] Nicolas Papernot, Patrick McDaniel, Ian Goodfellow, Somesh Jha, Z. Berkay Celik, and Ananthram Swami. Practical black-box attacks against machine learning. In Proceedings of the 2017 ACM on Asia conference on computer and communications security, pages 506-519, 2017.
[RGCI5] Mauro Ribeiro, Katarina Grolinger, and Miriam AM Capretz. Mlaas: Machine learning as a service. In 2015 IEEE 14th International Conference on Machine Learning and Applications (ICMLA), pages 992.902. IEEE, 2015.
[RXC+21] Pengzhen Ren, Yun Xiao, Xiaojun Chang, Yao Huang, Zhihui Li, Brij B Gupta, Xiaojiang Chen, and Xin Wang. survey of deep active learning. ACM computing surveys (CSUR), 54(9):1-4) 2021.
[SAB22] Sunandini Sanyal, Sravanti Addepalli, and R Venkatesh Babu. Towards data-free model stcaling in a hard label setting. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5284-15593,, 2022.
[SB12] Joe Staines and David Barber. Variational optimization. arXiv preprint 12X2v:1212.4500 2012.
[Set09] Burr Settles. Active learning literature survey. 2009.
[SS18] Ozan Sener and Silvio Savarese. Active learning for convolutional neural networks: A core-set approach. In International Conference on Learning Representations, 2018.
[SS20] Congzheng Song and Vitaly Shmatikov. Overlearnigg reveals sensitive attributes. In 8th International Conference on Learning Representations, ICLR 2020, 2020.
[SSSS17] Reza Shokri, Marco Stronati, Congzheng Song, and Vitaly Shmatikov. Membership inference attacks against machine learning models. 2017 IEEE symposium on security and privacy (SP), pages 3-18. IEEE, 2017.
[TWP211] Jean-Baptiste Truong, Pratyush Maini, Robert J Walls, and Nicolas Papernot. Data-free model extraction. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 4771-4780, 2021.
16] Florian Tramèr, Fan Zhang, Ari Juels, Michael K Reiter, and Thomas Ristenpart. Stealing machine learning models via prediction (APIs). In 25th USENIX security symposium (USENIX Security 16), pages 601-6188 2016.
19]++19 Zhilin Yang. Zihang Dai, Yiming Yang, Jaime Carbonell, Russ Salakhutdinov, and Quoc V Le. Xlnet: Generalized autoregressive pretraining for language understanding. Advances in neural information processing systems, 32, 2019.
[YGFJ18] Samuel Yeom, Irene Giacomelli, Matt Fredrikson, and Somesh Jha. Privacy risk in machine learning: Analyzing the connection to overfitting, In 2018 IEEE 31st computer security foundations symposium (CSF), pages 268-282. IEEE, 2018
[YSS+21] Ashkan Yousefpour, Igor Shilov, Alexandre Sablayrolles, Davide Testuggine, Karthik Prasad, Mani Malek, John Nguyen, Sayan Ghosh, Akash Bharadwaj. Jessica Zhao Graham Cormode, and llya Mironov. Opacus: User-friendly differential privacy library in Py Torch. arXiv preprint
2021.
13

---

[ZLX*22] Jie Zhang, Bo Li, Jianghe Xu, Shuang Wu, Shouhong Ding, Lei Zhang, and Chao Wu. Towards efficient data free black-box adversarial attack. In Proceedings of the IEEE/CVE Conference Computer Vision and Pattern Recognition, pages 15115 151 15125, 2022.
[WWL=00] Mingyi Zhou, Jing Wu, Yipeng Liu, Shuaicheng Liu, and Ce Zhu. Dast: Data-free substitute training for adversarial attacks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 234-243, 2020.
ZZZ11] Xiang Zhang, Junbo Jake Zhao, and Yann LeCun. Character-level convolutional networks for text classification. In NIPS, 2015.
14

---

Appendix
Table of Contents
A Complete pseudocode of MARICH
16
Theoretical analysis: Prools of Section
17
review active sampling strategies
D
Extended experimental analysis
21
D.1
Test accuracy of extracted models
21
D.2
Fidelity of the prediction distributions of The extracted models
23
D.3
Fidelity of parameters of the extracted models Membership inference with the extracted models
25
D.4
26
Significance and comparison of three sampling strategies
F
Performance against diffcrentially private target models
31
G Effects of model mismatch
33
Choices of hyperparameters
34
Broader impact
In this paper, we design model extraction attack algorithm, MARICH, that aims construct model that has similar predictive distribution as that of a target model. In this direction, we show that popular deep Neural Network (NN) models can be replicated with few number of queries and only outputs from their predictive API. We also show that this can be further used to conduct membership inference about the private training data that the adversary has no access to. Thus, MARICH points our attention to the vulnerabilities of the popular deep NN models to preserve the privacy of the users, whose data is used to train the deep NN models. Though every attack algorithm can be used adversarially, our goal is not to promote any such adversarial use.
Rather, in the similar spirit that of the attacks developed in cryptography to help to design better defenses and to understand vulnerabilities of the computing systems better, we conduct this research understand the extent of information leakage done by an ML model under modest assumptions. We recommend to be further used and studied for developing better privacy defenses and adversarial attack detection algorithms.
15

---

A Complete pseudocode of MARICH
Algorithm 2 MARICH
Input:
Target model: f", Query dataset: D #Classes: k
Parameter: #initial samples: no. Training epochs: Emaxs #Batches of queries: T, Query budget: B. Subsampling ratios: 7/1 71. E (0, 1]
Output: Extracted model f
Ifa Initialisation of the extracted model*//
Phase
2: Qraai + no datapoints randomly chosen from D Yo Yfrain + (Qh
Query the target model f with Qrrin
4: for epoch f 1 to Emaz do
5: S0 Train f with (Qu Yfrein)
6: end for
7: Adaptive query selection to build the extracted model*//
b
Phase
8: for fort< to T do
9: Qentropy + ANTROPYSAAPLINNG/C DA Q Qi-1" ,B)
10: Qrrad GRADIENTSSAMPLIGGG.... Qentropy B)
11: Qloss t- LOSSSAMPLING Qtrad Q4-1n Ytrain
x 717BB)
12:
Ynew + -T
D Query the target model f with Qloss
13:
Qirain t Qi U
14:
Yirrin t Y UYme.
15: for epoch +- to Emaz do
16: f. <- Train f: with (Qtrain ytrrai)
17: end for
18: end for
19: return Extracted model f + ff
20:
EntropySampling (extracted model: f input data points: Xin, budget: B)
22: Qentropy + arg max max H (Xin))
D Select B points with maximum entropy
23: return Qentropy
24:
25: GradientSampling (extracted model: fE, input data points: X budget: "/a B)
26: E < (((x)(xx))
27: G [V,E X
28: Cin + centres of G' computed using K-means
Qyred arg min xxXxxxxX..xxx> Σz,xx Ex,ec.. EV,EEVV,,E
> Select in B points from Xin whose aE are closest to that of Oin
30: return Qgrad
31:
32: LossSampling (extracted model: f input data points: Xmn previous queries: Qirain- previous
predictions: Ytrain, budget: 7172 B)
33: L. +- I(Ytraine (
> Compute the mismatch vector Select top-k mismatching points
34: Quis ARGMAXSORT (L,
35: Qloss +o arg min xi.. N1 Eoeex Enje
||,; 2z l2
> Select 712 points closest to Qmis
return Qloss
16

---

B Theoretical analysis: Proofs of Section 3
In this section, we claborate the proofs for the Theorems 3.2 and 3.4
Theorem 3.2 (Upper Bounding Distributional Closeness). we choose KL-divergence as divergence function D, we can show that
DKL(Pr(fE (Q), Q)Il Pr (Q), - min (Q), H(J)(Q))
Proof. Let consider a query generating distribution DO on T&. A target model f Rd y induces joint distribution the query and the output (or label) space, denoted by Pr(f-- Q) Similarly, the extracted model f R + y also induces a joint distribution over the query and the output (or label) space, denoted by Pr Q).
(Pr (Q), Q)I Pr (Q), Q))
= - dPr(ff Q) log (Q).
Pr(f Q)
- -S (Q) g) Pr(Q log (Q) = 4) dq Pr(f 9)
S (Q( 9) Pr(Q 2) log Pr( (Q)lQ 9) dq
- (Q(1) = 4) Pr(Q = 9) log Pr (Q)QQ q) dg
JQER"
| Pr(J)(QQ)( 9) Pr(Q 2) log Pr(j (Q) v) dg + BD (1))
IQERS
S -H (Q) dq DD (1)
S -- (Q) dq + De
The last incquality holds true the extracted model f is trained using the outputs of the target model f Thus, by prooessoessiig inequality, its output distribution possesses less information than that of the target model. Specifically, we know that if Y f(X), H(Y) s H(X.
Now, by taking min, on both sides, we obtain
DKL(Pr (Pr (Q), Q)Il Pr(fE WDE.
()) min EE (Q) -((
Here, DD arg min DKL (Pr(ff (Q), Q)Il (Q). Q)). The equality exists if minima of LHS and RHS coincide.
Theorem 3.4 (Lower Bounding Information Leakage). The information leaked by any Max- Information attack (Equation 3 is lower bounded as follows:
I(Pr(f (Q), Q)Il Pr Wn (Q),2))
2 max (Q)
Proof. Let consider the same terminology the previous proof. Then,
= + -
2 H(I5(Q)))) (((E(Q))))-(((1000)0))))
(9)
2 H(EE(Q)) (1
(10)
2 H(fE(Q)) -EQNU)
(11)
'Throughout the proofs, we slightly abuse the notation to write ((Pr(X),Pr(Y)) as 1(X, Y) for avoiding cumbersome equations.
17

---

The inequality of Equation is due to the fact that entropy is always non-negative. Equation holdss true (X, Y)= 2 [ H (Y)) for two random variables X and Y. The last inequality due to the fact that conditional entropy of two random variables X and Y, i.e. HX)))), is smaller than or equal to their cross entropy, i.e. 1(X, Y) (Lemma B.I).
Now, by taking max, on both sides, we obtain
[(Pr(f0 (Q), Q)I Pr(fur
(Q), I max (Q), H(
Mavint
Here, Maxluf - arg max,, (( (Q): Q)I Pr(fe: W Maxtot of LHS and RHS coincide.
(Q), Q)). The equality exists if maxima
Lemma B.1 (Relating Cross Entropy and Conditional Entropy). Given two random variables and Y, conditional entropy
H(XIV) S ((X,Y).
(12)
Proof. Here,
Pr(e y) log Pr(x.u) du (X)drz((Y) and ((X,Y) Pr(y)
I(Pr(X), Pr(Y))
Pr(x)) In Pr(y)dus (X) (X))2(() denotes the cross-entropy, given ref-
erence measures Vn and 1/2
((X,Y) H(X) + (I()))())))))
H(X(Y) I(X;Y) + DRL (Px|IX))
H(|V)
The last inequality holds as both mutual information and KL-divergence DkL non-negative functions for any X and Y.
18

---

C A review of active sampling strategies
Least Confidence sampling (LC). Least confidence sampling method [Set09] LS06] iteratively selects the subset of k data points from a data pool. which are most uncertain at that particular instant. The uncertainty function (u(-Ifw) X [0, 1) is defined as u(x]fw) è - Pr(j(z), where i is the predicted class by a model fw for input a.
Margin Sampling (MS). In margin sampling [G19]. a subset of & points is selected from a data pool, such that the subset demonstrates the minimum margin, where margin(-)566 [ww) X + [0, 1] is defined as
margin(<\].)) A Pr(yi (z))a, fw) (r(ū2(x)) fw), where fu is the model, and ýn (x) and (2(x)) are respectively the highest and the second highest scoring classes returned by Su-
Entropy Sampling (ES). Entropy sampling, also known as uncertainty sampling [G94]] iteratively selects a subset of is datapoints with the highest uncertainty from data pool. The uncertainty is defined by the entropy function of the prediction vector, and is computed using all the probabilities returned by the model fw for datapoint x. For a given point x and a model fw, entropy is defined as
|
(ntropy(=]fw) =
Σ Pa log(Pa); a=l
where Pa Pr(ff((z) a] for any output class a E {L..... DBB [G94] mention that while using this strategy. "the initial classifier plays an important role, since without it there may be a long period of random sampling before examples of low frequency class are stumbled upon". This is similar to our experimental observation that ES often demonstrate high variance in its outcomes.
Core-set Algorithms. Here, discuss some other interesting works in active learning using core-sets, and the issues to directly implement them in our problem.
[ZC121] aims to identify subset of training dataset for training specific model. The algorithm needs white-box access to the model, and also needs to do forward pass over the whole training dataset. A white-box sampling algorithm and the assumption of being able to retrieve predictions over the whole training dataset are not feasible in our problem setup. Relaxing the white-box access, [KDR*22 proposes to us the average loss over the training dataset validation set. Then, the gradient of loss on this training or validation set is compared with that the selected batih of data points. In black-box attack, we do not have access average loss over a whole training or validation dataset. Thus, it is not feasible to deploy the proposed algorithm.
significantly diverse
Following another approach, [MB220] proposes an elegant pre- processing algorithm to select core-set of a training dataset. This selection further leads to an efficient incremental gradient based training methodology. But in our case, we sequentially query the black-box model to obtain label for a query point and use them further for training the extracted model. Thus, creating dataset beforehand and using them further to pre-process will not lead to adaptive attack and also will not reduce the query budget. Thus, it is out of scope of our work. [KS22] deploys an auxiliary classifier to first use subset of labelled datapoints to create low dimensional embeddings. Then according to the query budget, chooses the points from the sparse regions from each cluster found from the low dimensional embeddings. This is another variant of uncertainty sampling that hypothesises the points from the sparse region are more informative in terms of loss and prediction entropy. This design technique is at the same time model specific, and thus incompatible to interest. Additionally, this approach also increases the requirement of queries to the target model.
K-Center sampling (KC). Thus, we focus on a version of core-set algorithm, namely K-Center sampling, that is applicable in our context. K-Center sampling as an active learning algorithm that has been originally proposed to train CNNs sampling iteratively selects the k datapoints that minimise the core set loss (see Equation 3, (SSI8) the most for a given model fa. In this method, the embeddings (from the model under training) of the data points are used as the representative vectors, and K-Center algorithm is applied on these representative vectors.
[SS18]. For given data pool, K-C K-Center
19

---

Random Sampling (RS), In random sampling, subset of & datapoints are selected from a data pool uniformly random.
In our experiments, for query selection at time i, the extracted model at time - 1, i.e. J is used fur and data pool at time & is the corresponding query dataset, except the datapoints that has been selected before step t. Hereafter, we deploy a modified version of the framework developed by [Hua21] to run our experiments with the aforementioned active learning algorithms.
20

---

D Extended experimental analysis
In this section, we step-wise elaborate further experimental setups and results that we skipped for the brevity of space in the main draft. Specifically, we conduct our experiments in six experimental setups. Each experimental setup corresponds to triplet (target model architecture trained on private dataset, extracted model architecture, query dataset). Here, list these six experimental setups in detail
1. Logistic Regression (I.R) model trained on MNIST, a LR model for extraction, EMNIST dataset for querying
A Logistic Regression (LR) model trained on MNIST, a LR model for extraction, CIFARIO dataset for querying
CNN model trained on MNIST, a CNN model for extraction, EMNIST dataset for querying
4. A ResNet model trained on CIFARIO, a CNN model for extraction, ImageNet dataset for querying
5. A ResNet model trained CIFAR10, ResNer1 { model for extraction, ImageNet dataset for querying
6. A BERT mode modeFtrained trained on BBCNews, BERT model for extraction, AGNews dataset for querying
For each of the experimental setups, we evaluate five types of performance evaluations, which are elaborated in Section D.I] D.2.1 D.2.2 D.3 and D.4 While each the following sections contain illustrations of the different performance metrics evaluating efficacy of the attack and corresponding discussions, Table 2 4 contain summary of all queries used, accuracy, and membership inference statistics for all the experiments.
D.1
Test accuracy of extracted models
Test accuracy of the extracted model and its comparison with the test accuracy of the target model on a subset of the private training dataset, which was used by neither of these models, is the most common performance metric used to evaluate the goodness of the attack algorithm. The attacks designed solely to optimise this performance metric are called the task accuracy model extraction attacks (JCB +20).
With MARICH, we aim extract models that have prediction distributions closest that of the target model. Our hypothesis is constructing such prediction distribution lead to model that also has high accuracy on the private test dataset, since accuracy is a functional property of the prediction distribution induced by a classifier. In order to validate this hypothesis, we compute test accuracies of the target models, and models extracted by MARICH and other active sampling algorithms in six experimental setups. We illustrate the evolution curves of accuracies over increasing number of queries in Figure
To compare MARICH with other active learning algorithms, we attack the same target models using K-centre sampling, Least Confidence sampling, Margin Sampling, Entropy Sampling, and Random Sampling algorithms (ref. Appendix C) using the same number of queries as used for MARICH in each setup.
From Figure algorithms.
5 we observe that in most of the cases MARICH outperforms all other competing
In this process, MARICH uses ~ 500 8000 queries, which is a small fraction of the corresponding query datasets. This also indicates towards the query-efficiency of MARICH.
Extraction of a ResNet trained on CIFAR10 with a ResNet18. Along with the five experimental setups mentioned in the paper, we trained a ResNet with CIFARIO dataset (D" here), that shows a test accuracy of 91.82% on disjoint test set. We use ImageNet as D인 here, to extract a ResNet 8 model from the target model. We have restrained from discussing this setup in the main paper due to brevity of space.
'We begin with pre-trained ResNet18 model from http:://prreec....g///s///a/aaap cenerated/torchviiee models. resneete........ html
We use the pre-traine pre-trained BERT model from https : http:///gggggaaeeeo/b/bbbbbeeeee
21

---

Target
Marich
KC
LC
MS
ES
RS
90
80
Aecursoy %
70-
60
60
50
40
30
20
250
500
750
1000
1250
1500
1750
200
400
6.0
800
1000
Queries
Queries
(a) Logistic regression with EMNIST queries
(b) Logistic regression with CIFARIO R10 queries
80
80
Acouracy %
60
40
40
20
20
1000
2000
3000
4000
5000
6000
2000
1000
6000
8000
Queries
Ouertes
(c) CNN with EMNIST queries
(d) ResNet with CNN and ImageNet queries
100
80
80
Accurey %
Accuracy %
60
40
20
100
Queries
Quertes
(e) ResNet with ResNet18 and ImageNet queries
(I) BERT with AGNews queries
Figure 5: Comparison of test accuracies achieved by models extracted by different active sampling algorithms.
MARICH extracts a ResNet18 using 8429 queries, and the extracted ResNet18 shows test accuracy of 71.65 + 0.88%. On the other hand, models extracted using Best of Competitors (BoC) using ImageNet queries shows accuracy 70.67 + 0.12%.
22

---

D.2 Fidelity of the prediction distributions of The extracted models
Driven by the distributional equivalence extraction principle, the central goal of MARICH to construct extracted models whose prediction distributions are closest to the prediction distributions of corresponding target models. From this perspective. in this section, we study the fidelity of the prediction distributions of models extracted by MARICH and other active sampling algorithms, namely K-centre sampling, Least Confidence sampling, Margin Sampling, Entropy Sampling, and Random Sampling.
D.2.1 KL-divergence between prediction distributions
First, the metric of distributional equivalence, we evaluate the KI--divereecc between the pre- diction distributions of the models extracted by MARICH and other active sampling algorithms. In Figure we report the box-plot (mean, median + 25 percentiles) of KL-divergences (in log-scale) calculated from 5 runs for each of 10 models extracted by each of the algorithms.
6
10
10a
Log(KL divergence)
Logft divergence)
10
10-1
Median
Mean
10-2
10
10-3
Median
Mean
10-3
LC
Marich
LC
MS
(a) Logistic regression with EMNIST queries
(b) Logistic regression with CIFARIO queries
100
Median Mean
Median
LogtKL divergence)
Log'KL divatgence)
x 10
Mean
3 10-1
10
x 10-1
10-1
10-2
Marich
KC
LC
Marich
LiC
Ms
(c) CNN with EMNIST queries
(d) ResNet with CNN and ImageNet queries
Median Mean
Log(KL divergonce)
Log(Kt divergence)
10-1
10-2
10-1
Median
10-3
Mean
Marich
LC
Marich
LC
MS
(e) ResNet with ResNet18 and ImageNet queries
(f) BERT with AGNews queries
Figure 6: Comparison of KL-divergences (in log-scale) between the target prediction distributions
and prediction distributions of the models extracted by different active learning algorithms.
Results. Figure 6 shows that the KL--divergence achieved by the prediction distributions of models extracted using MARICH at least This validates our claim that MARICU yields distributionally closer extracted model f from the target model f than existing active sampling algorithms.
10 times less than that of the other competing algorithms.
23

---

D.2.2 Prediction agreement
100
Median
90
Mean
80
80
Agreement
60
50
50
Median
Mean
40
LC
MS
(a) Logistic regression with EMNIST queries
(b) Logistic regression with CIFAR1O queries
Median
80
Mean
Agreement
70
Agrvement
45
50
Median
40
Mean
35
Marich
KC
LC
MS
ES
RS
Marich
KC
LC
MS
ES
RS
(c) CNN with EMNIST queries
(d) ResNet with CNN and ImageNet queries
70
90
68
Agreement
Agreement
66-
70
64
60
Median
50
Median Mean
Mean
60
Marich
KC
LC
MS
ES
RS
Marich
KC
LC
MS
ES
RS
(c) ResNet with ResNet18 and ImageNet queries
(f) BERT with AGNews queries
Figure 7: Comparison of agreement in predictions (in %) between the target model and the models
extracted by different active learning algorithms.
In Figure 7
we illustrate the agreement in predictions of f with f on test datasets using different active learning algorithms. Prediction agreement functions as another metric of fidelity of prediction distributions constructed by extracted models in comparison with those of the target models.
Similar to Figure report the box-plot (mean, median + 25 percentiles) of prediction agreements (in %) calculated from 5 runs for each 10 models extracted by each the algorithms.
Results. We observe that the prediction distributions extracted by MARICH achieve almost same to 30% higher prediction agreement comparison with the competing algorithms. Thus, we infer that in this particular case MARICI achieves better fidelity than the other active sampling algorithms, some instances, while is similar to the BoC in some instances.
24

---

D.3
Fidelity of parameters of the extracted models
Log (Paramettr Distance (122)
4
LogrParmetrri Distance (L2))
x
2 10
10a
101
10"
Marich
KC
LC
MS
ES
RS
Marich
KC
LC
MS
ES
RS
(a) Logistic regression with EMNIST queries
(b) Logistic regression with CIFARIO queries
Log(Paremett Distance (12)
2.41
(Paramemtei Distance (12))
2.4 10
6.4 10
x 10a
2.30
6.2x
2.37 16
6x 10
2.36
5.8 10
10-1
x.34 10a
Martch
RS
Martch
RS
(c) with EMNIST queries
(d) BERT with AGNews queries
Figure 8: Comparison of parametric fidelity for MARICH and different active learning algorithms.
Generating extracted models with low parametric fidelity is not a main goal or basis of the design principle MARICH. Since parametric fidelity a popularly studied metric evaluate goodness of model extraction, in Figure 8 we depict the parametric fidelity of models extracted by different active learning algorithms.
Let wg be the parameters of the extracted model and wr be the parameters of the target model. We define parametric fidelity as Fw log || [/ -wrll2- Since the parametric fidelity only computable when the target and extracted models share the same architecture, we report the four instances here where MARICH is deployed with the same architecture as that of the target model. For logistic regression, we compare all the weights of the target and the extracted models. For BERT and CNN, we compare between the weights in the last layers of these models.
Results. For LR, observe that the LR models extracted by MARICH have 20 30 times lower parametric fidelity than the extracted LR models of the competing algorithms. For BERT, the BERT extracted by MARICH achieves 0.4 times lower parametric fidelity than the Best of Competitors (BoC). As an exception, for CNN, the model extracted by K-center sampling achieves 0.996 times less parametric fidelity than that of MARICH.
Thus, we conclude that MARICH as a by-product of its distributionally equivalent extraction principle also extracts model with high parametric fidelity, which is often better than the competing active sampling algorithms.
25

---

D.4 Membership inference with the extracted models
main goal of MARICH is conduct Max-Information attack on the target model, i.e. extract an informative replica of its predictive distribution that retains the most information about the private training dataset. Due lack of any direct measure of informativeness of an extracted model with respect to target model, we run Membership Inference (MI) attacks using the models extracted by MARICH, and other competing active sampling algorithms. High accuracy and agreement MI attacks conducted on extracted models of MARICH and the target models implicitly validate our claim that MARICH able to conduct a Max-Information attack.
Observation 1. From Figure 9 we see that in most cases the probability densities of the membership inference closer to the target model when the model is extracted using MARICII, than using all other active sampling algorithms (BoC, Best of Competitors).
Target member
Marich member
BoC member
Target nonmember
Marich nonmember
BoC nonmember
20.0
17.5
15.0
4
Denaity
20.0
7.5
5.0-
20.2
0.2
0.4
Probability
1.0
12
0.2
0.0
0.2
Probability
0.4
0.8
0.0
1.0
1.2
(a) Logistic regression with EMNIST queries
(b) Logistic regression CIFAR1O queries
3.0
Density
2.0-
1.5-
1.0-
0.2
0.0
0.2
0.4
0..5
0..8
1.0
1.2
0.0
0.2
0.4
Probability
Probubility
(c) CNN with EMNIST queries
(d) ResNet with CNN with ImageNet queries
3.0-
2.0
1.5-
Dosaty
1.0
0.8
1.0
Prohability
-7.2
Probubility
(e) ResNet with ResNet18 with ImageNet queries
(1) BERT with AGNews queries
Figure 9: Comparison among membership vs. non-membership probability densities for membership attacks against models extracted by MARICI, the best of competitors (BoC) and the target model. Each figure represents the model class and query dataset. Memberships and non-eemberships inferred from the model extracted by MARICH are significantly closer to the target model.
26

---

Observation 2. In Figure 10 we present the agreements from the member points, nonmember points and overall agreement curves for varying membership thresholds, along with the AUCs of the overall membership agreements. We see that in most cases, the agreement curves for the models extracted using MARICH are above those for the models extracted using random sampling. thus AUCs are higher for the models extracted using MARICH.
Marich overall
Marich nonmember
BoC member
Marich member
BoC overall
BoC nonmember
100
100
Mombership apremost
30-
Memberlbip apreemunt 36
20-
70
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.5
0.8
1.0
Probability thowhhlds
Probulality
(a) Logistic regression with EMNIST queries (b) Logistic regression with CIFARIO queries
100
100
Xenabeship sproemmet
80
Xenbessip
60
68-
40
46
0.0
0.2
0.4
8.6
0.8
1.0
0.0
0.2
0-4
0.5
0.8
1.0
Probability threcholde
Probuliiiity Chrecholis
(c) CNN with EMNIST queries
(d) ResNet with CNN and ImageNet queries
100.
Mounbership app aprommmnt 56
80
00-
Membership
40
60
0.0
0.2
0.4
0.81
1.60
0.0
0.2
0.6
0.8
1.0
Probabiiity threcholde
Probubility threcholds
(c) ResNet with ResNet18 & ImageNet queries
BERT with AGNews queries
Figure 10: Comparison of membership. nonmembership and overall agreements of membership attacks against models extracted by MARICH and the best of competitors with the target model trained with MNIST. Each figure represents the model class and query dataset. Membership agreement of models extracted by MARICH are higher.
Observation 3. In Table 2 and 4 summarise the MI accuracy the private training dataset, Nonmembership inference accuracy on the private training dataset, Agreement in MI w.r.t. the MI on the target model, and AUC of Agreement in MI with that of the target model for Logistic Regression (LR). CNN, ResNet, and BERT target models. We observe that, while compared with other active sampling algorithms, out of 6 combinations of (target model, extracted model, query dataset) under study, the models extracted by MARICH achieve the highest accuracy in MI and agreement in MI the target model, in most of the instances.
Results. These observations support our claim that model extraction using MARICH gives models accurate and informative replica of the target model.
27

---

Accurec
7108-55901
6142-62255
51.0-1.50 1.500
51 51.00-3.000 3.00%
4691
90325
70000- 183
6015-772
8132-18%
62 0..00%
02,723
43.07%
M2.30
90.0%
91.27
NEAES
W011
86.228
86.9%
Table 2: Model extraction membership inference statisties for extracting Logistic Regression model and CNN
W0.3
87.848
X0.2055
80.
Woating
90.376
02
W7.27%
81.3%
07-0095
84.77%
7533%
70.99g
77.378
700006
90025
9632%
03300
93.0%
95718
90.33%
Noumemberrt
$7.50%
15.00%
10.0%
18 184175
25 10s
11322
0214%
10.100
82377
ST.05
18.708
Membrrshie
03.45%
07.20
57 805
37 7.03
03.635
W8.374
97.22%
01.15
04.55
01.9%
19.305
91.7%
1063
1563
1963
1003
1363
300
300
300
1863
1803
1963
1963
EMINSST
EMINSST
EXINST
ISINSST
EXNNST
EMINST
EMINNT
CIFARTO
CIARRO
CIAXTO
CIAAER
CIFARO
CHARR
CIARROO
EMINIST
EMINSTT
EMNNST
EMWIST
ESINSST
Munec
MaReh
E.
KC
MS
R3
TC
13
BMNNSY
PONNSY
RMNNST
EMNNST
EMINST
EMINST
CHARCH
EMINST
BONNST
POINSY
DONNSS
L8
1.
CR
18
DK
CNN
CNN
CWN
CNN
CNN
CNN
MNSST
UNNST
MNIST
MSIST
MNI8T
MASST
ANTST
MNIST
MANST
MNSS7
MNNST
MNSST
KSNST
MNNST
MNIST
28

---

30.11.33
3. 122
3/
Acuuas
01.26
2045-15
Meunhe Bip sevemmnn KLC
11
1130
1D0%
11.9%
Ovraa mncea ceaalentpp Dieral memerraa Mereenat
13.0%
G1 11:0%
03.21%
Table 4: Model extraction and membership inference statistics for extracting BERT model
Table 3: Model extraction and membership inference statistics for extracting a ResNet
122 2e12
( 0015
7145
10.11
75,20%
62.004
Membersiig Neuncmlun are Owwall
Noemi Nomraleehhp stt
40.0%
00.42%
1/22'
71.85
11203
11A3'
00AY'S
71.8%%
152005
Mea bersl
07 205
703
(0
(3 301
00.00
8420
8420
8220
1220
3520
1220
Quzzie
Batoet
Now member
ImsgeNe
Imageher
Imagerer
Image
Imagoee
ImageNer
NO
Aluur stdd
Alpurillon need
K3
KC
Auaa Datlact
Dataeet
ImogeVve
InogeNe
Stund Mockl
CAN
CAN
CAN
CNN
CAN
CAN
Targt Moddl
LIFOATO
Sonber
Manker
29

---

E
Significance and comparison of three sampling strategies
Given the bi-level optimization problem, we came up with MARICH in which three sampling methods are used in the order: (i)
(ii)
and (iii)
LOSSSAMPLING.
These three sampling techniques contribute to different goals:
ENTROPYSAMPLING selects points about which the classifier particular time step is most confused
AMPLING uses gradients of entropy of outputs the extracted model w.r.t. the inputs embeddings and selects points behaving most diversely at every time step.
LOSSSAMPLINE selects points which produce highest loss when loss is calculated between target model's output and extracted model's output.
One can argue that the order is immaterial for the optimization problem. But looking the algorithm practically, we see that EETROOYYGRADINNTSE AMPLING and COSsSAMPLING incur much higher time complexity than ENTROPY more efficient than the others. This makes us put ENTROPY SAMPLING as the first query selection strategy.
Thus, using ANTROPYSAMPLIIN the entire query set
As per the optimization problem in Equation we are supposed to find points that show highest mismatch between the target and the extracted models after choosing the query subset maximising the entropy. This leads us to the idea of LOssS AMPLING. But as only focusing on loss between models may choose points from one particular region only, and thus, decreasing the diversity of the queries. We use ENTROPY GRADIENTSAMPLING before LOSSSAMPLING. This ensures selection of diverse points with high performance mismatch.
In Figure II] we experimentally see the time complexities of the three components used. These are calculated by applying the sampling algorithms on a logistic regression model, on mentioned slices of MNIST dataset.
Entropy sampling
40
Entropy Gradient sampling
Loss sampling
Time (8)
20
1000
1500 2000
2500
3000
3500 4000 4500
entire query
Figure 11: Runtime comparison of three sampling strategies to select queries from 4500 datapoints.
Table 5: Time complexity of different sampling Strategies
Sampling Algorithm
Query space size
#Selected queries
Time (s)
Entropy Sampling
4500
100
1.82 + 0.04
Entropy-Gradicnt Sampling
4500
100
10.56 + 0.07
Loss Sampling
4500
100
41.64 = 0.69
30

---

Performance against differentially private target models
In this section, we aim to verify performance of MARICI against privacy-preserving mechanisms. Specifically, apply (e, o)-Different Privacy (DP) inducing mechanism (DMNS06) DBB21 on the target model to protect the private training dataset. There are three types of methods to ensure DP: output perturbation (DMNS06), objective perturbation [CMSTT] [BBT8), and gradient perturbation (ACG* 16). Since output perturbation and gradient perturbation methods scale well for nonlinear deep networks, we focus on them the defense mechanism against MARICH'S queries.
Gradient perturbation-based defenses. DP-SGD (ACG+ 16 is used to train target model on the member dataset. This mechanism adds noise to the gradients and clip them while training the target model. We use the default implementation of Opacus 2 to conduct the training in PyTorch.
Following that, attack the (e, 6)-D target models using MARICH and compute the corresponding accuracy of the extracted models. In Figure 12 we show the effect of different privacy levels on the achieved accuracy of the extracted Logistic Regression model trained with MNIST dataset and queried with EMNIST dataset. Specifically, we assign ò = 10 and vary in {0.2, 0.5, 1, oo}. Here, E = oo corresponds the model extracted from the non con-private target model.
We observe that the accuracy of the models extracted from private target models are approximately 2.3 7.4% lower than the model extracted from the non-private target model. This shows that performance of MARICH decreases against DP defenses but not significantly.
40
Target model (LR)
0.2
20
3000
Queries
Figure 12: Performance of models extracted by MARICH against (c, d)fifferrentll private target models trained using DP-SGD. consider different privacy levels e and = 10-5, Accuracy the extracted models decrease with increase in privacy (decrease in E).
Output perturbation-baser defenses. Perturbing output of algorithm against certain queries with calibrated noise, in brief output perturbation, is one of the basic and oldest form of privacy- preserving mechanism [DMNS06). Here, specifically deploy the Laplace mechanism, where calibrated Laplace noise is added to the output of the target model generated against some queries. The noise is sampled from Laplace distribution Lap(O, & where A is sensitivity of the output and e is the privacy level. This mechanism ensures s-DP.
We compose Laplace mechanism to the target model while responding to MARICH'S query and evaluate the change accuracy of the extracted model as the impact the defense mechanism. We logistic regression model trained on MNIST the target model. We query it using EMNIST and CIFAR10 10 datasets respectively. We vary a (0.25, 2, 8, oo). For each e and query dataset, we report the mean and standard deviation of accuracy of the extracted models on a test dataset. Each experiment is run times.
We observe that decrease in i.e. increase in privacy, causes decrease in accuracy of the extracted model. For EMNIST queries (Figure 3abb. degradation in accuracy is around 10% for s = 2, 8 but 8 has practically
we observe significant drop for 0.25. For CIFAR 10 queries (Figure no impact on the performance of the extracted model. But for e and 0.25, the accuracy of extracted models drop down very fast.
31

---

Thus, we conclude that output perturbation defends privacy of the target model against MARICH for smaller values of E. But for larger values of E, the target model significantly against MARICH.
mechanism might not save the
80-
80
2
60
Targel model (CPF)
Apcurary
0.25
40
Target model (LR)
40
8
0.25
20
1000
2000
2000
3000
4000
5000
Samples
Sampless
(a) LR with EMNIST queries
(b) LR with CIFARIO queries
Figure 13: Performance of models extracted by MARICH against target models that perturbs the output of the queries to achieve E-DP. We consider different privacy levels Accuracy of the extracted models decrease with increase in privacy (decrease in c).
32

---

G Effects of model mismatch
From Equation we observe that functionality of MARICH is constrained by selecting the same model class for both the target model f and the extracted model f To elaborately study this aspect, in this section, we conduct experiments to show MARICH's capability handle model mismatch and impact of model mismatch on performance of the extracted models.
Specifically, we run experiments for three cases: (1) extracting an LR model with LR and CNN, (2) extracting CNN model with LR and CNN, and (3) extracting a ResNet model with CNN and ResNet18 We train the target LR and the target CNN model MNIST dataset. We further extract these two models using EMNIST as the query datasets. We train the target ResNet model on CIFAR10 dataset and extract using ImageNet queries. The results number of queries and achieved accuracies are summarised in Table and Figure [4
Observation 1. In all the three experiments, cases when the model classes match and mismatch. This shows universality and model-obliviousness of MARICH model extraction attack.
MARICH without any modification for both the
we observe that model mismatch influences performance of model extracted by M When we extract the LR target model with LR and CNN, we observe that both the extracted models achieve almost same accuracy and the extracted CNN model achieves bit more accuracy than the extracted LR model. In contrast, when extract the CNN target model with LR and CNN, we observe that the extracted LR model achieves lower accuracy than the extracted CNN model. Similar observations are found for extracting the ResNet with ResNet18 and CNN, respectively.
Observation 2. From Figure 14
Conclusions. From these observations, we conclude that MARICH can function model-obliviouse, also observe that if we use less complex model to extract a more complex model, the accuracy drops significantly. But if we extract low complexity model with higher complexity one, obtain higher accuracy instead model mismatch. This intuitive as the low-complexity extracted model might have lower representation capacity mimic the non-linen non-linear decision boundary of the high-com high-complexity model but the opposite is not true. future, would interesting to delve into the Beaming-theoreet origins of this phenomenon.
CNN by CNN
CNN by
ANCMENIC No.
1.00
Target
20
20-
Resivet1
10
2000
4000
6000
8000
Sample
Sangle
Sumples
(a) LR extracted by
CNN extracted by CNN
(c) ResNet extracted by CNN
vs. LR extracted by CNN
vs. CNN extracted by LR
vs. ResNet extracted by ResNet18
Figure 14: Effect of model mismatches on models extracted by MARICI.
Table 6: Effect of model mismatch accuracy of The extracted models.
f
f
#samples
Accuracy
LR
LR
5130
82,37
5.7%
LR
CNN
5130
85,41
0.57%
CNN
LR
5440
57.81 = 3.64%
CNN
CNN
5440
91.63
0.42%
ResNet
CNN
8429
56.11
1.35%
ResNet
ResNet18
8429
71.65 = 0.88%
33

---

H Choices of hyperparameters
In this section, we list the choices of the hyperparameters of Algorithm for different experiments and also explain how we select them.
Hyperparameters "Y1 and "/2 are kept constant, i.e., 0.8, for all the experiments. These two parameters act as the budget shrinking factors.
Instead of changing these two, we change the number of points no, which are randomly selected the beginning, and the budget for every step. We obtain the optimal hyperparameters for each experiment by performing line search in the interval [100,500).
We further change the budget over the rounds. time step t, the budget, B; a² Bt-1. The idea is to select more points as f goes on reaching the performance of f Here, a > 1 and needs to be tuned. We use 1.02, which is obtained through line search [1.01, 1.99).
For number of rounds T, we perform a line search in [10, 20).
Table 7: Hyperparameters for different datasets and target models.
Member Patiave
Target Medel
Allack Model
Atlack Datasel
Budget
Tnitaal poinis
Rounds
Epeche/ounnl
Learning Rale
MINST
EMINST
10
1.012
MINST
CNN
CNN
EMNIST
550
500
0.8
0.8
10
0:015
MNNST
CNN
CNN
CIFARIO
750
500
0.8
0.8
10
10
0,03
CIFARIO
ReeNeL
CNN
ImogeNet
750
500
0.8
08
10
0,2
ImggoNor
0.02
BBC News
BERT
BERT
AG News
100
0.8
0.8
5 10
34


Reviews:
[{"review_id": "v9c4YuKoNN", "reviewer": "Reviewer_SZQw", "summary": "The authors propose a model extraction attack which queries a model's publicly available API and chooses samples based on maximizing entropy of the target model's predictions, and maximizing agreement with between extracted model and target model predictions. These samples are then used in training a surrogate model.", "questions": "1. Why does the definition for \"Distributionally equivalent model extraction (Def. 3.1) measure the divergence between joint distributions? It seems like just considering the divergence between the distribution of the two model model outputs would be sufficient.", "limitations": "The authors do not include a limitations section in their main body.", "rating": 7, "confidence": 2, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1. The paper is very well presented.\n2. The authors are very clear and good about motivating their proposed method. \n3. The results of the attack are quite impressive when compared to baselines. \n4. Experimentation is thorough, including a range of target models and data distributions considered.", "weaknesses": "1. While the attack is well motivated, it seems like the major practical contribution of this work is the addition of the entropy term in eq. 7. If so, this seems like a relatively minor contribution. I don't have background in this area, but it seems like any previous work should at least consider the Model-mismatch term in your optimization problem. Is this not the case? Nonetheless, I do think the work offers a principled approach to the problem, and has valuable contributions, so this is a minor weakness in my eyes. \n\nUpdate:\nI have read the authors' response to my review and my concerns are largely satisfied. I will keep my score as a 7.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank the reviewer for acknowledging the strengths and soundness of the contribution as well as for their comments to improving the manuscript.\n\n**Novelty of contributions:** We refer to the general comments for an in-depth discussion.\n\n**Formulation of distributionally equivalent model extraction:** A classifier induces a predictive distribution over labels/output for a given input. But it is a local view of the classifier. What we want to replicate here is the global predictive distribution of the classifier, i.e. how it induces a predictive distribution over all possible input which are sampled from a dataset or equivalently, a data-generating distribution. Later, we observe the importance of this formulation as it leads to the ERM-like query selection objective in Eq. (5) and (6) (Line 224, page 5). If we would have considered the KL divergence between only the pointwise predictive distributions, we would not have obtained the $E_Q$ term in Eq. (5) and thus, no natural ERM formulation as in Eq. (6).\n\nWe hope that our response addresses the reviewer's questions. Let us know if you have any further query."}, {"review_id": "Vg4kIGgXGJ", "reviewer": "Reviewer_tspy", "summary": "The authors studied black-box model extraction, which is a practical scenario in MLaaS. In order to boost attack efficiency in this extraction, they focused on distributional equivalence and max-information model extraction. As for distributional equivalence, they proposed a distributional notion of equivalence. On the other hand, they maximised the mutual information between the extracted and target models’ distributions. Experiments showed the effectiveness of MARICH.", "questions": "refer to weakness.", "limitations": "refer to weakness.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "+ The topic is of practical importance.\n+ The proposed two methods are novel and elegant.", "weaknesses": "- The writing section of this paper can be further improved and reorganized. The related work can be presented as a separate section in the paper. The results in Table 1 should be presented more clearly, especially the effectiveness of MARICH.\n- The experiments may not be comprehensive enough. For the field of computer vision, only MNIST and CIFAR-10, which are relatively small datasets, were considered. Since this paper focuses on black-box model extraction in real-world scenarios, I suggest conducting experiments on ImageNet or its subsets.\n- In the results of Table 1, MARICH does not show overwhelming advantages and even performs worse than entropy sampling on STL/ResNet-18. Can the tradeoff between them be analyzed?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank the reviewer his/her valuable time spent reviewing.\n\n**Improved presentation of Table 1:** We would like to refer the reviewer to the full paper with appendix which is provided in the supplementary materials. We apologize for the fact that we had made the rectifications in Table 1 in the full version, which contains the actual representation of our experimental results.\n\n**Experiments with ImageNet:** We have actually done experiments with ImageNet/ResNet-18. We have described this in the text of the main paper (line 297, line 321-322) and later added the rectified Table 1 in the extended paper. Please check the extended paper in the supplementary material.  The results show applicability of our algorithm to larger datasets. \nWe would also like to attract the reviewer's attention to the fact that the row with STL/ResNet-18 in Table 1 in the first draft is rectified in the extended version.\n\n**Trade-off of different performance measures:** We observe that even with the limited query budget (less than most of the related works), we achieve better performance in most the cases, or close enough to that of the ``Best of the Competing\" attacks whenever we cannot (Table 1 in extended paper in the supplementary material). It would be interesting to check whether performance of different attacks on different datasets/models can be characterised. This can be an interesting future question and is also related to the instance hardness of privacy attacks as in (Carlini et al., 2022: https://arxiv.org/abs/2112.03570). Since the question under study is not to design a better active learning algorithm but to design an adaptive extraction attack from first information-theoretic principles, different performance measures do not only depend on the query selection algorithm and the dataset but also the information leaked by its output of a model regarding its input.\n\nWe hope that our response addresses the reviewer’s questions, and will convince him/her to raise the score."}, {"review_id": "3tYgTvr9M1", "reviewer": "Reviewer_wp9P", "summary": "Given a dataset DQ and blackbox access to a model trained on a dataset DP,  the authors identify a subset of the dataset DQ that can be used to train another model. They use a metric based on energy to select the samples.\n\nThe authors use with model stealing/extraction attacks as their primary related work, though the work has more resemblance to the literature on coresets. In coresets, DP and DQ are the same. But, overall, the algorithm MARICH seems very similar to what would be used in the coresets work. The mismatch between DP and DQ appears to be relatively low in the experiments. For example, MNIST versus EMNIST; CIFAR 10 versus CIFAR10 (same?); and BBC News versus AGNews. Notably, the DQ datasets are about the same or larger than original datasets. \n\nIt seems a better baseline for the work would be substantial work on coresets, which also aims to find an efficient subset of a dataset that can be used to train a comparable model to one that is trained on the full dataset.  The reason is that DP and DQ are not that different in authors' experiments. Indeed, if trained on 100% of the dataset, the accuracy appears to be very high. Thus, it seems an adversary has little incentive to select a subset, if the primary objective is to create a high-fidelity clone. Indeed, prior work on model stealing generally aims to exceed the performance of what would be achievable with just DQ. \n\nAuthors cite one of the coresets papers on that [lines 497-498, SS18].  But, in the last 5 years, there is a substantial body of work on coresets based on various metrics, including energy scores, forgetting, AUM, and Coverage-Centric core-set selection based on stratified sampling. I would have liked to see a comparison with that instead.\n\n\n\n", "questions": "Compare more extensively with core-set work in both related work and in the experiments.\n\nUse metrics and datasets that are similar to those in your baseline work, so that one can more directly compare your results with those in earlier work.\n\n", "limitations": "I did not see a discussion on limitations in the paper or potential negative societal impact of the work.\n\n", "rating": 4, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "Entropy-based sampling is a reasonable strategy to reduce the size of the selected state. They use 3 datasets, MNIST, CIFAR10, and BBC News (on BERT). Use of BERT in the evaluation is good, since most other papers on coresets use \n", "weaknesses": "The paper does not appear to be well-positioned with respect to prior work.\n\nExperiments could be better argued. Membership Inference attack is an interesting metric to use in the paper for comparing the extracted model with the original one. Is there a privacy goal here?  Why not use a straight Accuracy measure on a test dataset from the original dataset DP, as in most core-set work or model extraction work?\n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for his/her feedback. We answer below to his/her concerns and comments.\n\n**Explaining the experimental design:**\n\n**1. MI attack:** A membership inference attack shows informativeness of the extracted model, while never using the private training dataset. Our experimental results validate our hypothesis that since Marich is derived from the principle of max-information attack, it extracted models yielding high MI accuracy. This is a direct violation of privacy (in MI attack sense) of the users in the private training dataset. \n\n**2. Accuracy measure:** We agree with the reviewer about the test dataset and this is exactly what we have done. We use a subset of the private dataset as the test dataset for both the target and extracted models. We mention this also in our experimental goal ``How do the accuracy of the model extracted using MARICH \\textbf{on the private dataset} compare with that of the target model, and RS with same query budget?\" (Line 305 and 306)\n\n**Core-set literature:** Thanks for pointing us to the core-set literature. We are aware of this growing field, and we have compared our results with the models extracted using a compatible core-set approach, such as [SS17]. We have a detailed discussion on the related active sampling strategies in Appendix C of the extended paper in the supplementary material.\n\nHere, we discuss some other interesting works in this direction. We show why they are not directly implementable in our problem. \n\n[Kil+21b] aims to identify a subset of training dataset for training a specific model. The algorithm needs white-box access to the model, and also needs to do a forward pass over the whole training dataset. Thus, the objective of our work is different. Also a white-box sampling algorithm and the assumption of being able to retrieve predictions over the whole training dataset are not feasible in our problem setup.\n\n[Kil+21a] requires access to the average loss over the training dataset or a significantly diverse validation set. Then, the gradient of loss on this training or validation set is compared with that of the selected minibatch of datapoints. In a black-box attack, we do not have access to average loss over a whole training or validation dataset. Thus, it is not feasible to deploy the proposed algorithm. \n\n[MBL20] proposes an elegant preprocessing algorithm to select a coreset of a training dataset. This selection further leads to an efficient incremental gradient based training methodology. But in our case, we sequentially query the black-box model to obtain a label for a query point and use them further for training the extracted model. Thus, creating a dataset beforehand and using them further to preprocess will not lead to an adaptive attack and also will not reduce the query budget. Thus, it is out of scope of our work.\n\n[KS22] needs an auxiliary classifier to first use a subset of labelled datapoints to create low dimensional embeddings. Then  according to the query budget, it chooses the points from the sparse regions from each cluster found from the low dimensional embeddings. This is another variant of uncertainty sampling that hypothesises the points from the sparse region are more informative in terms of loss and prediction entropy. This design technique is at the same time model specific, thus incompatible to our interest, and also increases the requirement of queries to the target model.\n\nWe will add this discussion in the final version of our paper and add designing core-set based algorithms as an interesting future work.\n\n**Comparison with baselines and performance metrics.** We have shown our performance on all the relevant metrics of model extraction attacks (e.g. accuracy on test dataset (App. D.1.) [MSDH19, CSBB+18, OSF19], parametric fidelity (App. D.3) and prediction matching (App D.2.2) of the extracted models [LM05, BBJP18, JCB+20]). We also add new metrics such as KL-divergence between predictive distributions of the extracted and target models as a goodness of extraction attack (Figure 3, Appendix D.2.1), and accuracy of MI attacks with extracted models as informativeness of the extracted models (Table 1, Appendix D.4). \n\nWe have also compared with the previously used active learning algorithms for attacks. We would like to emphasise that this is not a work regarding generic active learning or training on a subset of dataset. The goal of this work is to design an adaptive and frugal model extraction algorithm from the first principles of distributionally equivalent/max-information extraction. Since the final algorithm is similar to an active query selection algorithm in spirit, we compare its performance with 5 other types of active selection approaches, namely  ``K-Center (KC) [SS18], Least-Confidence sampling (LC) [LS06], Margin sampling (MS) [BBZ07 , JG19], Entropy Sampling (ES) [LG94], and Random Sampling (RS)\". Designing better active query selection algorithm using any of these five approaches would be an interesting research work but out of the scope of this paper.\n\nHope we have answered your comments and concerns, and also look forward to respond if you have any further query. Hope our response will convince you to raise the score.\n\n**New references**\n\n[MBL20] Baharan Mirzasoleiman, Jeff Bilmes, and Jure Leskovec. “Coresets\nfor data-efficient training of machine learning models”. ICML 2020\n\n[Kil+21a] Krishnateja Killamsetty et al. “Grad-match: Gradient matching based\ndata subset selection for efficient deep model training”. ICML 2021\n\n[Kil+21b] Krishnateja Killamsetty et al. “Retrieve: Coreset selection for effi-\ncient and robust semi-supervised learning”. NeurIPS 2021\n\n[KS22] Yeachan Kim and Bonggun Shin. “In Defense of Core-set: A Density-\naware Core-set Selection for Active Learning”. ACM SIGKDD 2022"}, {"review_id": "5KKCbHJMnc", "reviewer": "Reviewer_jbqh", "summary": "This paper proposes a model-oblivious query-efficient black-box model extraction attack. This attack is achieved by solving a proposed distributionally equivalent and max-information model extraction problem. To solve the problem, this paper develops an active sampling-based query selection algorithm, MARICH, to select the most informative queries that simultaneously maximize the entropy and reduce the mismatch between the target and the stolen models. Experimental results validate the effectiveness and query efficiency of the proposed method.", "questions": "Based on W2, I am wondering if you can provide practical evidence that can demonstrate that your proposed method can be used to extract real-world APIs (e.g., classifiers deployed by Amazon AWS, Google API, Microsoft  Azure)?", "limitations": "Please refer to the weakness and questions.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "S1: This paper proposes a novel notion called distributional equivalence extraction that can extend the existing notions of task accuracy and functionally equivalent model extractions. This paper also theoretically demonstrates that solving the developed optimization problem which correlates the distributionally equivalent with another proposed max-information extraction can simultaneously maximize the entropy and reduce the mismatch between the target and the stolen models.\n\nS2: The experiments set up in the paper basically address general doubts that arose while reading the paper, which increases the credibility of the proposed method.\n\nS3: The layout of this paper is elegant, and the overall structure and logical progression are clear. These advantages make this paper enjoyable for reading and easy to follow.\n", "weaknesses": "W1: The employed datasets which are used for training the target model being extracted are considered to be simple. \n\nW2: Although this paper repeatedly foregrounds that their approach is designed to extract publically available APIs, there is no experiment that aims to extract models deployed in real-world APIs.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the time spent reviewing and encouraging comments about the novelty, presentation, and experimental results.\n\n**Querying industrial black-box APIs.** For genuine financial constraints, we cannot run experiments by querying the industrial APIs. Instead, we trained our own models and used them as the black-box APIs that predict for a given query datapoint. The code is at your disposal for reproducing the black-box models attacked in this work through API-like queries.\n\n**Experiments with larger datasets.** We would like to refer the reviewer to the full paper with appendix which is provided in the supplementary materials. This contains further experiments with larger datasets, such as ImageNet, being used as the public dataset for the attacks.\n\nWe hope that our response addresses the reviewer's concerns. Let us know if you have any other query."}]

Author rebuttal:
We would like to thank the reviewers for their valuable time and efforts towards improving the manuscript. In the following, we highlight the novelty of our contributions and width of our evaluation. We then address comments specific to each reviewer by responding to them directly.

**Novelty of contributions and width of evaluations:** The contributions of this work are three folds:

1. Rather than aiming for extracting replicating the task accuracy or the functional equivalence (i.e. replicating the model weights), we focus on extracting the predictive distribution of a model given a data generating distribution. We encode this objective in two ways: (a) distributionally equivalent extraction, (b) max-information extraction. Our hypothesis is that if we can extract the predictive distribution of a model that is good enough to replicate other properties of the model (e.g. accuracy) and also to run other attacks (e.g. membership inference).

2. We show that both of the problem formulations (i.e. distributionally equivalent extraction and max-information extraction) lead to a unique variational optimisation problem. This optimisation problem provides us a method to adaptively and sequentially choose queries from another dataset, without accessing any side information about or part of the private training dataset. Previously, researchers have deployed entropy sampling and other active sampling methods to efficiently select queries for attack. But our natural formulation shows that these methods can be further grounded on the objectives of distributionally equivalent extraction and max-information extraction. Thus, though the end result is an active query selection algorithm, the fundamental approach to design it is different and novel with respect to the existing works. 

3. We now test our proposed algorithms on multiple types of image (MNIST, CIFAR10, EMNIST, ImageNet etc.) and text (BBCNews, AGNews) data, and different types of models (logistic regression, CNN, ResNet-18, BERT). Experiments demonstrate three things. (a) Our methodological approach to design the adaptive extraction algorithm is query efficient. (b) Our hypothesis that replicating the predictive distribution of a model is enough to replicate its functionality (e.g. accuracy) and use it for MI attacks is valid. (c) Our approach allows us to design a model-oblivious and dataset-oblivious approach to attack as we can extract the true model's predictive distribution with a different model architecture and a mismatched querying dataset.

Hope the commentary clarifies the motivation behind the problem and its relevance. We are looking forward to respond if you have further questions.

Meta-review: The paper presents a novel model extraction attack utilising public APIs. It effectively motivates its approach, demonstrates impressive results against baselines, and conducts thorough experiments across various models and data distributions. However, certain weaknesses exist in terms of contribution significance, organisation, and real-world applicability.

Reviewers raised concerns about extra datasets (e.g. ImageNet) and, at times, over-claiming. Some of the concerns have been addressed in the extended version. I strongly encourage the authors to take these comments and revise the camera-ready version accordingly. In the end, the overall agreement of the reviewers was to accept the paper, and I concur.
Decision: Accept (poster)