SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Pratik Karmakar*
School of Computing, National University of Singapore
CNRS@CREATE Ltd, 1 Create Way, Singapore
pratik.karmakar@u.nus.edu
Debabrota Basu
Équipe Scool, Univ. Lille, Inria, CNRS, Centrale Lille
UMR 9189-CRISTAL,F-59000 Lille,France
debabrota.basu@inria.fr
We study design of black-box model extraction attacks that can send minimal number of queries from a publicly available dataset to a target ML model through a predictive API with an aim to create an informative and distributionally equivalent replica of the target. First, we define distributionally equivalent and MaxInformation model extraction attacks, and reduce them into a variational optimisation problem. The attacker sequentially solves this optimisation problem to select the most informative queries that simultaneously maximise the entropy and reduce the mismatch between the target and the stolen models. This leads to an active sampling-based query selection algorithm, MARICH, which is model-oblivious. Then, we evaluate MARICH on different text and image data sets, and different models, including CNNs and BERT. MARICH extracts models that achieve  $\sim 60 - 95\%$  of true model's accuracy and uses  $\sim 1,000 - 8,500$  queries from the publicly available datasets, which are different from the private training datasets. Models extracted by MARICH yield prediction distributions, which are  $\sim 2 - 4\times$  closer to the target's distribution in comparison to the existing active sampling-based attacks. The extracted models also lead to  $84 - 96\%$  accuracy under membership inference attacks. Experimental results validate that MARICH is query-efficient, and capable of performing task-accurate, high-fidelity, and informative model extraction.
In recent years, Machine Learning as a Service (MLaaS) is widely deployed and used in industries. In MLaaS [RGC15], an ML model is trained remotely on a private dataset, deployed in a Cloud, and offered for public access through a prediction API, such as Amazon AWS, Google API, Microsoft Azure. An API allows an user, including a potential adversary, to send queries to the ML model and fetch corresponding predictions. Recent works have shown such models with public APIs can be stolen, or extracted, by designing black-box model extraction attacks [TZJ+ 16]. In model extraction attacks, an adversary queries the target model with a query dataset, which might be same or different than the private dataset, collects the corresponding predictions from the target model, and builds a
$^{*}$ A significant portion of the work has been done as a part of P. Karmakar's masters in Ramakrishna Mission Vivekananda Educational and Research Institute, Belur, India.
** Code is available at: https://github.com/debabrota-basu/marich
replica model of the target model. The goal is to construct a model which is almost-equivalent to the target model over input space  $\left[\mathrm{JCB}^{+}20\right]$ .
Often, ML models are proprietary, guarded by IP rights, and expensive to build. These models might be trained on datasets which are expensive to obtain  $\mathrm{[YDY^{+}19]}$  and consist of private data of individuals [LM05]. Also, extracted models can be used to perform other privacy attacks on the private dataset used for training, such as membership inference [NSH19]. Thus, understanding susceptibility of models accessible through MLaaS presents an important conundrum. This motivates us to investigate black-box model extraction attacks while the adversary has no access to the private data or a perturbed version of it  $\mathrm{[PMG^{+}17]}$ . Instead, the adversary uses a public dataset to query the target model [OSF19,  $\mathrm{PGS^{+}20}$ ].
Figure 1: Black-box model extraction with MARICH.
Query-efficient black-box model extraction poses a tension between the number of queries sent to the target model and the accuracy of extracted model  $\mathrm{[PGS^{+}20]}$ . With more queries and predictions, an adversary can build a better replica. But querying an API too much can be expensive, as each query incurs a monetary cost in MLaaS. Also, researchers have developed algorithms that can detect adversarial queries, when they are not well-crafted or sent to the API in large numbers [JSMA19, PGKS21]. Thus, designing a query-efficient attack is paramount for practical deployment. Also, it exposes how more informa
tion can be leaked from a target model with less number of interactions.
In this paper, we investigate effective definitions of efficiency of model extraction and corresponding algorithm design for query-efficient black-box model extraction attack with public data, which is oblivious to deployed model and applicable for any datatype.
Our contributions. Our investigation yields three contributions.
1. Formalism: Distributional equivalence and Max-Information extraction. Often, the ML models, specifically classifiers, are stochastic algorithms. They also include different elements of randomness during training. Thus, rather than focusing on equivalence of extracted and target models in terms of a fixed dataset or accuracy on that dataset  $\left[\mathrm{JCB}^{+}20\right]$ , we propose a distributional notion of equivalence. We propose that if the joint distribution induced by a query generating distribution and corresponding prediction distribution due to both the target and the extracted models are same, they will be called distributionally equivalent (Sec. 3). Another proposal is to reinforce the objective of the attack, i.e. to extract as much information as possible from the target model. This allows us to formulate the Max-Information attack, where the adversary aims to maximise the mutual information between the extracted and target models' distributions. Our hypothesis is that if we can extract the predictive distribution of a model, it would be enough to replicate other properties of the model (e.g. accuracy) and also to run other attacks (e.g. membership inference), rather than designing specific attacks to replicate the task accuracy or the model weights. We show that both the attacks can be performed by sequentially solving a single variational optimisation [SB12] problem (Eqn. (6)).
2. Algorithm: Adaptive query selection for extraction with MARICH. We propose an algorithm, MARICH (Sec. 4), that optimises the objective of the variational optimisation problem (Eqn. (6)). Given an extracted model, a target model, and previous queries, MARICH adaptively selects a batch of queries enforcing this objective. Then, it sends the queries to the target model, collects the predictions (i.e. the class predicted by target model), and uses them to further train the extracted model (Algo. 1). In order to select the most informative set of queries, it deploys three sampling strategies in cascade. These strategies select: a) the most informative set of queries, b) the most diverse set of queries in the first selection, and c) the final subset of queries where the target and extracted models mismatch the most. Together these strategies allow MARICH to select a small subset of queries that both maximise the information leakage, and align the extracted and target models (Fig. 1).
3. Experimental analysis. We perform extensive the most for a given modeevaluation with both image and text datasets, and diverse model classes, such as Logistic Regression (LR), ResNet, CNN, and BERT (Sec. 5). Leveraging MARICH's model-obliviousness, we even extract a ResNet trained on CIFAR10 with a CNN and out-of-class queries from ImageNet. Our experimental results validate that MARICH extracts more accurate replicas of the target model and high-fidelity replica of the target's prediction distributions in comparison to existing active sampling algorithms. While MARICH uses a small number of queries ( $\sim 1k - 8.5k$ ) selected from publicly available query datasets, the extracted models yield accuracy comparable with the target model while encountering a membership inference attack. This shows that MARICH can extract alarmingly informative models query-efficiently. Additionally, as MARICH can extract the true model's predictive distribution with a different model architecture and a mismatched querying dataset, it allows us to design a model-oblivious and dataset-oblivious approach to attack.
Taxonomy of model extraction. Black-box model extraction (or model stealing or model inference) attacks aim to replicate a target ML model, commonly classifiers, deployed in a remote service and accessible through a public API  $\left[\mathrm{TZJ}^{+}16\right]$ . The replication is done in such a way that the extracted model achieves one of the three goals: a) accuracy close to that of the target model on the private training data used to train the target model, b) maximal agreement in predictions with the target model on the private training data, and c) maximal agreement in prediction with the target model over the whole input domain. Depending on the objective, they are called task accuracy, fidelity, and functional equivalence model extractions, respectively  $\left[\mathrm{JCB}^{+}20\right]$ . Here, we generalise these three approaches using a novel definition of distributional equivalence and also introduce a novel information-theoretic objective of model extraction which maximises the mutual information between the target and the extracted model over the whole data domain.
**Frameworks of attack design.** Following [TZJ+16], researchers have proposed multiple attacks to perform one of the three types of model extraction. The attacks are based on two main approaches: direct recovery (target model specific) [MSDH19, BBJP18, JCB+20] and learning (target model specific/oblivious). The learning-based approaches can also be categorised into supervised learning strategies, where the adversary has access to both the true labels of queries and the labels predicted by the target model [TZJ+16, JCB+20], and online active learning strategies, where the adversary has only access to the predicted labels of the target model, and actively select the future queries depending on the previous queries and predicted labels [PMG+17, PGS+20, CCG+20]. As query-efficiency is paramount for an adversary while attacking an API to save the budget and to keep the attack hidden and also the assumption of access true label from the private data is restrictive, we focus on designing an online and active learning-based attack strategy that is model oblivious.
Types of target model. While [MSDH19, CCG+20] focus on performing attacks against linear models, all others are specific to neural networks [MSDH19, JCB+20, PGS+20] and even a specific architecture [CSBB+18]. In contrast, MARICH is capable of attacking both linear models and neural networks. Additionally, MARICH is model-oblivious, i.e. it can attack one model architecture (e.g. ResNet) using a different model architecture (e.g. CNN).
Types of query feedback. Learning-based attacks often assume access to either the probability vector of the target model over all the predicted labels  $\mathrm{[TZJ^{+}16,OSF19,PGS^{+}20,JCB^{+}20]}$ , or the gradient of the last layer of the target neural network [MSDH19, MHS21], which are hardly available in a public API. In contrast, following  $\mathrm{[PMG^{+}17]}$ , we assume access to only the predicted labels of the target model for a set of queries, which is always available with a public API. Thus, experimentally, we cannot compare with existing active sampling attacks requiring access to the whole prediction vector  $\mathrm{[PGS^{+}20,OSF19]}$ , and thus, compare with a wide-range of active sampling methods that can operate only with the predicted label, such as  $K$ -center sampling, entropy sampling, least confidence sampling, margin sampling etc.  $\mathrm{[RXC^{+}21]}$ . Details are in Appendix C.
Choices of public datasets for queries. There are two approaches of querying a target model: data-free and data-selection based. In data-free attacks, the attacker begins with noise. The informative queries are generated further using a GAN-like model fed with responses obtained from an API [ZWL+20, TMWP21, MHS21, ZLX+22, SAB22]. Typically, it requires almost a million queries to the API to start generating sensible query data (e.g. sensible images that can leak from a model trained on CIFAR10). But since one of our main focus is query-efficiency, we focus on data-selection based attacks, where an adversary has access to a query dataset to select the queries
from and to send it to the target model to obtain predicted labels. In literature, researchers assume three types of query datasets: synthetically generated samples  $\left[\mathrm{TZJ}^{+}16\right]$ , adversarially perturbed private (or task domain) dataset  $\left[\mathrm{PMG}^{+}17, \mathrm{JSMA}19\right]$ , and publicly available (or out-of-task domain) dataset  $\left[\mathrm{OSF}19, \mathrm{PGS}^{+}20\right]$ . As we do not want to restrict MARICH to have access to the knowledge of the private dataset or any perturbed version of it, we use publicly available datasets, which are different than the private dataset. To be specific, we only assume whether we should query the API with images, text, or tabular data and not even the identical set of labels. For example, we experimentally attack models trained on CIFAR10 with ImageNet queries having different classes.
Further discussions on related active sampling algorithms and distinction of MARICH with the existing works are deferred to Appendix C.
2 Background: Classifiers, model extraction, membership inference attacks
Before proceeding to the details, we present the fundamentals of a classifier in ML, and two types of inference attacks: Model Extraction (ME) and Membership Inference (MI).
Classifiers. A classifier in ML [GBCB16] is a function  $f: \mathcal{X} \to \mathcal{Y}$  that maps a set of input features  $\mathbf{X} \in \mathcal{X}$  to an output  $Y \in \mathcal{Y}$ . The output space is a finite set of classes, i.e.,  $\{1, \dots, k\}$ . Specifically, a classifier  $f$  is a parametric function, denoted as  $f_{\theta}$ , with parameters  $\theta \in \mathbb{R}^d$ , and is trained on a dataset  $\mathbf{D}^T$ , i.e., a collection of  $n$  tuples  $\{(x_i, y_i)\}_{i=1}^n$  generated IID from an underlying distribution  $\mathcal{D}$ . Training implies that given a model class  $\mathcal{F} = \{f_{\theta} | \theta \in \Theta\}$ , a loss function  $l: \mathcal{Y} \times \mathcal{Y} \to \mathbb{R}_{\geq 0}$ , and training dataset  $\mathbf{D}^T$ , we aim to find the optimal parameter  $\theta^* = \arg \min_{\theta \in \Theta} \sum_{i=1}^{n} l(f_{\theta}(\mathbf{x}_i), y_i)$ . We use cross-entropy, i.e.,  $l(f_{\theta}(\mathbf{x}_i), y_i) = -y_i \log (f_{\theta}(\mathbf{x}_i))$ , as the loss function for classification.
Model extraction attack. A model extraction attack is an inference attack where an adversary aims to steal a target model  $f^T$  trained on a private dataset  $\mathbf{D}^T$  and create another replica of it  $f^E$  [TZJ+16]. In the black-box setting that we are interested in, the adversary can only query the target model  $f^T$  by sending queries  $Q$  through a publicly available API and to use the corresponding predictions  $\hat{Y}$  to construct  $f^E$ . The goal of the adversary is to create a model which is either (a) as similar to the target model as possible for all input features, i.e.  $f^T(x) = f^E(x) \forall x \in \mathcal{X}$  [SS20, CCG+20] or (b) predicts labels that has maximal agreement with that of the labels predicted by the target model for a given data-generating distribution, i.e.  $f^E = \arg \min_{\mathrm{Pr}_{x \sim \mathcal{D}}[l(f^E(x), f^T(x))]$  [TZJ+16, PGS+20, JCB+20]. The first type of attacks are called the functionally equivalent attacks. The later family of attacks is referred as the fidelity extraction attacks. The third type of attacks aim to find an extracted model  $f^E$  that achieves maximal classification accuracy for the underlying private dataset used to train the  $f^T$ . These are called task accuracy extraction attacks [TZJ+16, MSDH19, OSF19]. In this paper, we generalise the first two type of attacks by proposing the distributionally equivalent attacks and experimentally show that it yields both task accuracy and fidelity.
Membership inference attack. Another popular family of inference attacks on ML models is the Membership Inference (MI) attacks [SSSS17, YGFJ18]. In MI attack, given a private (or member) dataset  $\mathbf{D}^T$  to train  $f^T$  and another non-member dataset  $S$  with  $|\mathbf{D}^T \cap S| \neq \emptyset$ , the goal of the adversary is to infer whether any  $x \in \mathcal{X}$  is sampled from the member dataset  $\mathbf{D}^T$  or the non-member dataset  $S$ . Effectiveness of an MI attacks can be measured by its accuracy of MI, i.e. the total fraction of times the MI adversary identifies the member and non-member data points correctly. Accuracy of MI attack on the private data using  $f^E$  rather than  $f^T$  is considered as a measure of effectiveness of the extraction attack [NSH19]. We show that the model  $f^E$  extracted using MARICH allows us to obtain similar MI accuracy as that obtained by directly attacking the target model  $f^T$  using even larger number of queries. This validates that the model  $f^E$  by MARICH in a black-box setting acts as an information equivalent replica of the target model  $f^T$ .
3 Distributional equivalence and Max-Information model extractions
In this section, we introduce the distributionally equivalent and Max-Information model extractions. We further reduce both the attacks into a variational optimisation problem.
Definition 3.1 (Distributionally equivalent model extraction). For any query generating distribution  $\mathcal{D}^Q$  over  $\mathbb{R}^d\times \mathcal{V}$ , an extracted model  $f^{E}:\mathbb{R}^{d}\to Y$  is distributionally equivalent to a target
<sup>2</sup>We denote sets/vectors by bold letters, and the distributions by calligraphic letters. We express random variables in UPPERCASE, and an assignment of a random variable in lowercase.
model  $f^T: \mathbb{R}^d \to Y$ , if the joint distributions of input features  $Q \in \mathbb{R}^d \sim \mathcal{D}^Q$  and predicted labels induced by both the models are same almost surely. This means that for any divergence  $D$ , two distributionally equivalent models  $f^E$  and  $f^T$  satisfy  $D(\operatorname*{Pr}(f^T(Q), Q) \| \operatorname*{Pr}(f^E(Q), Q)) = 0 \forall \mathcal{D}^Q$ .
To ensure query-efficiency in distributionally equivalent model extraction, an adversary aims to choose a query generating distribution  $\mathcal{D}^Q$  that minimises it further. If we assume that the extracted model is also a parametric function, i.e.  $f_{\omega}^{E}$  with parameters  $\omega \in \Omega$ , we can solve the query-efficient distributionally equivalent extraction by computing
$$
\left(\omega_ {\mathrm {D E q}} ^ {*}, \mathcal {D} _ {\min } ^ {Q}\right) \triangleq \underset {\omega \in \Omega} {\arg \min } \underset {\mathcal {D} ^ {Q}} {\arg \min } D \left(\Pr \left(f _ {\theta^ {*}} ^ {T} (Q), Q\right) \| \Pr \left(f _ {\omega} ^ {E} (Q), Q\right)\right). \tag {1}
$$
Equation (1) allows us to choose a different class of models with different parametrisation for extraction till the joint distribution induced by it matches with that of the target model. For example, the extracted model can be a logistic regression or a CNN if the target model is a logistic regression. This formulation also enjoys the freedom to choose the data distribution  $\mathcal{D}^Q$  for which we want to test the closeness. Rather the distributional equivalence pushes us to find the best query distribution for which the mismatch between the posteriors reduces the most and to compute an extracted model  $f_{\omega^*}^E$  that induces the joint distribution closest to that of the target model  $f_{\theta^*}^T$ .
Connection with different types of model extraction. For  $D = D_{\mathrm{KL}}$ , our formulation extends the fidelity extraction from label agreement to prediction distribution matching, which addresses the future work indicated by  $[ \mathrm{JCB}^{+}20 ]$ . If we choose  $\mathcal{D}_{\min}^{Q} = \mathcal{D}^{T}$ , and substitute  $D$  by prediction agreement, distributional equivalence retrieves the fidelity extraction attack. If we choose  $\mathcal{D}_{\min}^{Q} = \operatorname{Unif}(\mathcal{X})$ , distributional equivalent extraction coincides with functional equivalent extraction. Thus, a distributional equivalence attack can lead to both fidelity and functional equivalence extractions depending on the choice of query generating distribution  $\mathcal{D}^{Q}$  and the divergence  $D$ .
Theorem 3.2 (Upper bounding distributional closeness). If we choose KL-divergence as the divergence function  $D$ , then for a given query generating distribution  $\mathcal{D}^Q$
$$
D _ {\mathrm {K L}} \left(\Pr \left(f _ {\theta^ {*}} ^ {T} (Q), Q\right) \| \Pr \left(f _ {\omega_ {\mathrm {D E q}} ^ {*}} ^ {E} (Q), Q\right)\right) \leq \min  _ {\omega} \mathbb {E} _ {Q} \left[ l \left(f _ {\theta^ {*}} ^ {T} (Q), f _ {\omega} ^ {E} (Q)\right) \right] - H \left(f _ {\omega} ^ {E} (Q)\right). \tag {2}
$$
By variational principle, Theorem 3.2 implies that minimising the upper bound on the RHS leads to an extracted model which minimises the KL-divergence for a chosen query distribution.
Max-Information model extraction. Objective of any inference attack is to leak as much information as possible from the target model  $f^T$ . Specifically, in model extraction attacks, we want to create an informative replica  $f^E$  of the target model  $f^T$  such that it induces a joint distribution  $\operatorname*{Pr}(f_{\omega}^{E}(Q), Q)$  which retains the most information regarding the target's joint distribution. As adversary controls the query distribution, we aim to choose a query distribution  $\mathcal{D}^Q$  that maximises information leakage.
Definition 3.3 (Max-Information model extraction). A model  $f^{E}:\mathbb{R}^{d}\to Y$  and a query distribution  $\mathcal{D}^Q$  are called a Max-Information extraction of a target model  $f^{T}:\mathbb{R}^{d}\to Y$  and a Max-Information query distribution, respectively, if they maximise the mutual information between the joint distributions of input features  $Q\in \mathbb{R}^{d}\sim \mathcal{D}^{Q}$  and predicted labels induced by  $f^{E}$  and that of the target model. Mathematically,  $(f_{\omega^{*}}^{E},\mathcal{D}_{\max}^{Q})$  is a Max-Information extraction of  $f_{\theta^{*}}^{T}$  if
$$
\left(\omega_ {\operatorname {M a x I n f}} ^ {*}, \mathcal {D} _ {\max } ^ {Q}\right) \triangleq \underset {\omega} {\arg \max } \underset {\mathcal {D} _ {Q}} {\arg \max } I \left(\Pr \left(f _ {\theta^ {*}} ^ {T} (Q), Q\right) \| \Pr \left(f _ {\omega} ^ {E} (Q), Q\right)\right) \tag {3}
$$
Similar to Definition 3.1, Definition 3.3 also does not restrict us to choose a parametric model  $\omega$  different from that of the target  $\theta$ . It also allows us to compute the data distribution  $\mathcal{D}^Q$  for which the information leakage is maximum rather than relying on the private dataset  $\mathbf{D}^T$  used for training  $f^T$ .
Theorem 3.4 (Lower bounding information leakage). For any given distribution  $\mathcal{D}^Q$ , the information leaked by any Max-Information attack (Equation 3) is lower bounded as:
$$
I \left(\Pr \left(f _ {\theta^ {*}} ^ {T} (Q), Q\right) \| \Pr \left(f _ {\omega_ {\operatorname* {M a x I n f}} ^ {*}} ^ {E} (Q), Q\right)\right) \geq \max  _ {\omega} - \mathbb {E} _ {Q} \left[ l \left(f _ {\theta^ {*}} ^ {T} (Q), f _ {\omega} ^ {E} (Q)\right) \right] + H \left(f _ {\omega} ^ {E} (Q)\right). \tag {4}
$$
By variational principle, Theorem 3.4 implies that maximising the lower bound in the RHS will lead to an extracted model which maximises the mutual information between target and extracted joint distributions for a given query generating distribution.