SlowGuess/NeurIPS_2023_OCR huggingface dataset:
[  $\mathrm{ZLX^{+}22}$  ] Jie Zhang, Bo Li, Jianghe Xu, Shuang Wu, Shouhong Ding, Lei Zhang, and Chao Wu. Towards efficient data free black-box adversarial attack. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15115- 15125, 2022.
$\left[\mathrm{ZWL}^{+}20\right]$  Mingyi Zhou, Jing Wu, Yipeng Liu, Shuaicheng Liu, and Ce Zhu. Dast: Data-free substitute training for adversarial attacks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 234-243, 2020.
[ZZL15] Xiang Zhang, Junbo Jake Zhao, and Yann LeCun. Character-level convolutional networks for text classification. In NIPS, 2015.
A Complete pseudocode of MARICH 16
B Theoretical analysis: Proofs of Section 3 17
C A review of active sampling strategies 19
D Extended experimental analysis 21
D.1 Test accuracy of extracted models 21
D.2 Fidelity of the prediction distributions of The extracted models 23
D.3 Fidelity of parameters of the extracted models 25
D.4 Membership inference with the extracted models 26
E Significance and comparison of three sampling strategies 30
F Performance against differentially private target models 31
G Effects of model mismatch 33
H Choices of hyperparameters 34
In this paper, we design a model extraction attack algorithm, MARICH, that aims to construct a model that has similar predictive distribution as that of a target model. In this direction, we show that popular deep Neural Network (NN) models can be replicated with a few number of queries and only outputs from their predictive API. We also show that this can be further used to conduct membership inference about the private training data that the adversary has no access to. Thus, MARICH points our attention to the vulnerabilities of the popular deep NN models to preserve the privacy of the users, whose data is used to train the deep NN models. Though every attack algorithm can be used adversarially, our goal is not to promote any such adversarial use.
Rather, in the similar spirit as that of the attacks developed in cryptography to help us to design better defenses and to understand vulnerabilities of the computing systems better, we conduct this research to understand the extent of information leakage done by an ML model under modest assumptions. We recommend it to be further used and studied for developing better privacy defenses and adversarial attack detection algorithms.
A Complete pseudocode of MARICH
Algorithm 2 MARICH
Input: Target model:  $f^T$ , Query dataset:  $\mathbf{D}^Q$ , #Classes:  $k$   
Parameter: #initial samples:  $n_0$ , Training epochs:  $E_{max}$ , #Batches of queries:  $T$ , Query budget:  $B$ , Subsampling ratios:  $\gamma_1, \gamma_2 \in (0, 1]$   
Output: Extracted model  $f^E$   
1: //* Initialisation of the extracted model*/  
2:  $Q_{0}^{train} \gets n_0$  datapoints randomly chosen from  $D^Q$   
3:  $Y_{0}^{train} \gets f^T(Q_{0}^{train})$   
4: for epoch  $\leftarrow 1$  to  $E_{max}$  do  
5:  $f_{0}^{E} \gets$  Train  $f^{E}$  with  $(Q_{0}^{train}, Y_{0}^{train})$   
6: end for  
7: //* Adaptive query selection to build the extracted model*/  
8: for  $t \gets 1$  to  $T$  do  
9:  $Q_{t}^{entropy} \gets$  ENTROPYSAMPLING  $(f_{t-1}^{E}, \mathbf{D}^{Q} \setminus Q_{t-1}^{train}, B)$   
10:  $Q_{t}^{grad} \gets$  GRADIENTSAMPLING  $(f_{t-1}^{E}, Q_{t}^{entropy}, \gamma_1B)$   
11:  $Q_{t}^{loss} \gets$  LOSSAMPLING  $(f_{t-1}^{E}, Q_{t}^{grad}, Q_{t-1}^{train}, Y_{t-1}^{train}, \gamma_1\gamma_2B)$   
12:  $Y_{t}^{new} \gets f^{T}(Q_{t}^{loss})$   
13:  $Q_{t}^{train} \gets Q_{t-1}^{train} \cup Q_{t}^{loss}$   
14:  $Y_{t}^{train} \gets Y_{t-1}^{train} \cup Y_{t}^{new}$   
15: for epoch  $\leftarrow 1$  to  $E_{max}$  do  
16:  $f_{t}^{E} \gets$  Train  $f_{t-1}^{E}$  with  $(Q_{t}^{train}, Y_{t}^{train})$   
17: end for  
18: end for  
19: return Extracted model  $f^{E} \gets f_{T}^{E}$   
20:  
21: EntropySampling (extracted model:  $f^{E}$ , input data points:  $X_{in}$ , budget:  $B$ )  
22:  $Q_{entropy} \gets$  arg max  $X \subset X_{in}, |X| = B$  H  $(f^{E}(X_{in}))$   
23: return  $Q_{entropy}$   
24:  
25: GradientSampling (extracted model:  $f^{E}$ , input data points:  $X_{in}$ , budget:  $\gamma_1B$ )  
26:  $E \gets H(f^{E}(X_{in}))$   
27:  $G \gets \{\nabla_x E \mid x \in X_{in}\}$   
28:  $C_{in} \gets k$  centres of  $G$  computed using K-means  
29:  $Q_{grad} \gets$  arg min  $X \subset X_{in}, |X| = \gamma_1B$ $\sum x_i \in X$ $\sum x_j \in C_{in}$ $\|\nabla_{x_i} E - \nabla_{x_j} E\|_2^2$   
30: return  $Q_{grad}$   
31:  
32: LossSampling (extracted model:  $f^{E}$ , input data points:  $X_{in}$ , previous queries:  $Q_{train}$ , previous predictions:  $Y_{train}$ , budget:  $\gamma_1\gamma_2B$ )  
33:  $L \gets l(Y_{train}, f^{E}(Q_{train}))$   
34:  $Q_{mis} \gets$  ARGMAXSORT(L, k)  
35:  $Q_{loss} \gets$  arg min  $X \subset X_{in}, |X| = \gamma_1\gamma_2B$ $\sum x_i \in X$ $\sum x_j \in Q_{mis}$ $\|x_i - x_j\|_2^2$   
36: return  $Q_{loss}$
B Theoretical analysis: Proofs of Section 3
In this section, we elaborate the proofs for the Theorems 3.2 and 3.4.<sup>3</sup>
Theorem 3.2 (Upper Bounding Distributional Closeness). If we choose KL-divergence as the divergence function  $D$ , we can show that
$$
D _ {\mathrm {K L}} (\operatorname * {P r} (f _ {\theta^ {*}} ^ {T} (Q), Q) \| \operatorname * {P r} (f _ {\omega_ {\mathrm {D E q}} ^ {*}} ^ {E} (Q), Q)) \leq \min _ {\omega} \mathbb {E} _ {Q} [ l (f _ {\theta^ {*}} ^ {T} (Q), f _ {\omega} ^ {E} (Q)) ] - H (f _ {\omega} ^ {E} (Q)).
$$
Proof. Let us consider a query generating distribution  $\mathcal{D}^Q$  on  $\mathbb{R}^d$ . A target model  $f_{\theta*}^T: \mathbb{R}^d \to \mathcal{V}$  induces a joint distribution over the query and the output (or label) space, denoted by  $\operatorname{Pr}(f_{\theta*}^T, Q)$ . Similarly, the extracted model  $f_{\theta*}^T: \mathbb{R}^d \to \mathcal{V}$  also induces a joint distribution over the query and the output (or label) space, denoted by  $\operatorname{Pr}(f_{\omega}^E, Q)$ .
$$
\begin{array}{l} D _ {\mathrm {K L}} \left(\Pr \left(f _ {\theta^ {*}} ^ {T} (Q), Q\right) \| \Pr \left(f _ {\omega} ^ {E} (Q), Q\right)\right) \\ = \int_ {Q \in \mathbb {R} ^ {d}} \mathrm {d} \Pr (f _ {\theta^ {*}} ^ {T} (Q), Q) \log \frac {\Pr (f _ {\theta^ {*}} ^ {T} (Q) , Q)}{\Pr (f _ {\omega} ^ {E} (Q) , Q)} \\ = \int_ {Q \in \mathbb {R} ^ {d}} \Pr (f _ {\theta^ {*}} ^ {T} (Q) | Q = q) \Pr (Q = q) \log \frac {\Pr (f _ {\theta^ {*}} ^ {T} (Q) | Q = q)}{\Pr (f _ {\omega} ^ {E} (Q) | Q = q)} d q \\ = \int_ {Q \in \mathbb {R} ^ {d}} \Pr (f _ {\theta^ {*}} ^ {T} (Q) | Q = q) \Pr (Q = q) \log \Pr (f _ {\theta^ {*}} ^ {T} (Q) | Q = q) d q \\ - \int_ {Q \in \mathbb {R} ^ {d}} \Pr (f _ {\theta^ {*}} ^ {T} (Q) | Q = q) \Pr (Q = q) \log \Pr (f _ {\omega} ^ {E} (Q) | Q = q) d q \\ = \int_ {Q \in \mathbb {R} ^ {d}} \operatorname * {P r} (f _ {\theta^ {*}} ^ {T} (Q) | Q = q) \operatorname * {P r} (Q = q) \log \operatorname * {P r} (f _ {\theta^ {*}} ^ {T} (Q) | Q = q) \mathrm {d} q + \mathbb {E} _ {q \sim \mathcal {D} ^ {\mathcal {Q}}} \left[ l (f _ {\theta^ {*}} ^ {T} (q)), f _ {\omega} ^ {E} (q)) \right] \\ \leq - H \left(f _ {\theta^ {*}} ^ {T} (Q) \mathrm {d} q + \mathbb {E} _ {q \sim \mathcal {D} ^ {\varrho}} \left[ l \left(f _ {\theta^ {*}} ^ {T} (q)\right), f _ {\omega} ^ {E} (q) \right]\right) \\ \leq - H \left(f _ {\omega} ^ {E} (Q) \mathrm {d} q + \mathbb {E} _ {q \sim \mathcal {D} ^ {\mathcal {Q}}} \left[ l \left(f _ {\theta^ {*}} ^ {T} (q)\right), f _ {\omega} ^ {E} (q)\right) \right] \tag {8} \\ \end{array}
$$
The last inequality holds true as the extracted model  $f_{\omega}^{E}$  is trained using the outputs of the target model  $f_{\theta^{*}}^{T}$ . Thus, by data-processing inequality, its output distribution possesses less information than that of the target model. Specifically, we know that if  $Y = f(X)$ ,  $H(Y) \leq H(X)$ .
Now, by taking  $\min_{\omega}$  on both sides, we obtain
Here,  $\omega_{\mathrm{DEq}}^{*} \triangleq \arg \min_{\omega} D_{\mathrm{KL}}(\operatorname{Pr}(f_{\theta^{*}}^{T}(Q), Q) \| \operatorname{Pr}(f_{\omega}^{E}(Q), Q))$ . The equality exists if minima of LHS and RHS coincide.
Theorem 3.4 (Lower Bounding Information Leakage). The information leaked by any Max-Information attack (Equation 3) is lower bounded as follows:
$$
I (\operatorname * {P r} (f _ {\theta^ {*}} ^ {T} (Q), Q) \| \operatorname * {P r} (f _ {\omega_ {\mathrm {M a x I n f}} ^ {E}} ^ {E} (Q), Q)) \geq \max _ {\omega} - \mathbb {E} _ {Q} [ l (f _ {\theta^ {*}} ^ {T} (Q), f _ {\omega} ^ {E} (Q)) ] + H (f _ {\omega} ^ {E} (Q)).
$$
Proof. Let us consider the same terminology as the previous proof. Then,
$$
\begin{array}{l} I \left(\Pr \left(f _ {\theta^ {*}} ^ {T} (Q), Q\right) \| \Pr \left(f _ {\omega} ^ {E} (Q), Q\right)\right) \\ = H \left(f _ {\theta^ {*}} ^ {T} (Q), Q\right) + H \left(f _ {\omega} ^ {E} (Q), Q\right) - H \left(f _ {\theta^ {*}} ^ {T} (Q), f _ {\omega} ^ {E} (Q), Q\right) \\ = H \left(f _ {\theta^ {*}} ^ {T} (Q), Q\right) + H \left(f _ {\omega} ^ {E} (Q), Q\right) - H \left(f _ {\omega} ^ {E} (Q), Q \mid f _ {\theta^ {*}} ^ {T} (Q)\right) + H \left(f _ {\theta^ {*}} ^ {T} (Q)\right) \\ \geq H \left(f _ {\omega} ^ {E} (Q), Q\right) - H \left(f _ {\omega} ^ {E} (Q), Q \mid f _ {\theta^ {*}} ^ {T} (Q)\right) (9) \\ \geq H \left(f _ {\omega} ^ {E} (Q)\right) - H \left(f _ {\omega} ^ {E} (Q), Q \mid f _ {\theta^ {*}} ^ {T} (Q)\right) (10) \\ \geq H \left(f _ {\omega} ^ {E} (Q)\right) - \mathbb {E} _ {Q} \left[ l \left(f _ {\omega} ^ {E} (Q), f _ {\theta^ {*}} ^ {T} (Q)\right) \right] (11) \\ \end{array}
$$
3 Throughout the proofs, we slightly abuse the notation to write  $l(\operatorname{Pr}(X), \operatorname{Pr}(Y))$  as  $l(X, Y)$  for avoiding cumbersome equations.
The inequality of Equation 9 is due to the fact that entropy is always non-negative. Equation 10 holds true as  $H(X,Y) \geq \max \{H(X), H(Y)\}$  for two random variables  $X$  and  $Y$ . The last inequality is due to the fact that conditional entropy of two random variables  $X$  and  $Y$ , i.e.  $H(X|Y)$ , is smaller than or equal to their cross entropy, i.e.  $l(X,Y)$  (Lemma B.1).
Now, by taking  $\max_{\omega}$  on both sides, we obtain
$$
I (\operatorname * {P r} (f _ {\theta^ {*}} ^ {T} (Q), Q) \| \operatorname * {P r} (f _ {\omega_ {\operatorname * {M a x I n f}} ^ {*} (Q)} ^ {E} (Q), Q)) \leq \max  _ {\omega} - \mathbb {E} _ {Q} [ l (f _ {\theta^ {*}} ^ {T} (Q), f _ {\omega} ^ {E} (Q)) ] + H (f _ {\omega} ^ {E} (Q)).
$$
Here,  $\omega_{\mathrm{MaxInf}}^* \triangleq \arg \max_{\omega} I(\operatorname{Pr}(f_{\theta^*}^T(Q), Q) \| \operatorname{Pr}(f_{\omega_{\mathrm{MaxInf}}^*}^E(Q), Q))$ . The equality exists if maxima of LHS and RHS coincide.
Lemma B.1 (Relating Cross Entropy and Conditional Entropy). Given two random variables  $X$  and  $Y$ , conditional entropy
$$
H (X | Y) \leq l (X, Y). \tag {12}
$$
Proof. Here,  $H(X|Y) \triangleq -\int \operatorname{Pr}(x,y) \log \frac{\operatorname{Pr}(x,y)}{\operatorname{Pr}(y)} \mathrm{d}\nu_1(X) \mathrm{d}\nu_2(Y)$  and  $l(X,Y) \triangleq l(\operatorname{Pr}(X), \operatorname{Pr}(Y)) = -\int \operatorname{Pr}(x) \ln \operatorname{Pr}(y) \mathrm{d}\nu_1(X) \mathrm{d}\nu_2(Y)$  denotes the cross-entropy, given reference measures  $\nu_1$  and  $\nu_2$ .
$$
\begin{array}{l} l (X, Y) = H (X) + D _ {\mathrm {K L}} (\Pr (X) \| \Pr (Y)) \\ = H (X | Y) + I (X; Y) + D _ {\mathrm {K L}} \left(P _ {X} \| P _ {Y}\right) \\ \geq H (X | Y) \\ \end{array}
$$
The last inequality holds as both mutual information  $I$  and KL-divergence  $D_{\mathrm{KL}}$  are non-negative functions for any  $X$  and  $Y$ .
C A review of active sampling strategies
Least Confidence sampling (LC). Least confidence sampling method [Set09, LS06] iteratively selects the subset of  $k$  data points from a data pool, which are most uncertain at that particular instant. The uncertainty function  $(u(\cdot |f_{\omega}):\mathcal{X}\to [0,1])$  is defined as
$$
u (x \mid f _ {\omega}) \triangleq 1 - \Pr (\hat {y} \mid x),
$$
where  $\hat{y}$  is the predicted class by a model  $f_{\omega}$  for input  $x$
Margin Sampling (MS). In margin sampling [JG19], a subset of  $k$  points is selected from a data pool, such that the subset demonstrates the minimum margin, where  $\mathrm{margin}(\cdot |f_{\omega}):\mathcal{X}\to [0,1]$  is defined as
$$
\operatorname {m a r g i n} (x \mid f _ {\omega}) \triangleq \Pr (\widehat {y} _ {1} (x) \mid x, f _ {\omega}) - \Pr (\widehat {y} _ {2} (x) \mid x, f _ {\omega}),
$$
where  $f_{\omega}$  is the model, and  $\hat{y}_1(x)$  and  $\hat{y}_2(x)$  are respectively the highest and the second highest scoring classes returned by  $f_{\omega}$ .
Entropy Sampling (ES). Entropy sampling, also known as uncertainty sampling [LG94], iteratively selects a subset of  $k$  datapoints with the highest uncertainty from a data pool. The uncertainty is defined by the entropy function of the prediction vector, and is computed using all the probabilities returned by the model  $f_{\omega}$  for a datapoint  $x$ . For a given point  $x$  and a model  $f_{\omega}$ , entropy is defined as
$$
\operatorname {e n t r o p y} (x \mid f _ {\omega}) \triangleq - \sum_ {a = 1} ^ {| y |} p _ {a} \log (p _ {a}),
$$
where  $p_a = \operatorname*{Pr}[f_\omega(x) = a]$  for any output class  $a \in \{1, \dots, |\mathcal{Y}|\}$ . [LG94] mention that while using this strategy, "the initial classifier plays an important role, since without it there may be a long period of random sampling before examples of a low frequency class are stumbled upon". This is similar to our experimental observation that ES often demonstrate high variance in its outcomes.
Core-set Algorithms. Here, we discuss some other interesting works in active learning using core-sets, and the issues to directly implement them in our problem.
[KZCI21] aims to identify a subset of training dataset for training a specific model. The algorithm needs white-box access to the model, and also needs to do a forward pass over the whole training dataset. A white-box sampling algorithm and the assumption of being able to retrieve predictions over the whole training dataset are not feasible in our problem setup. Relaxing the white-box access,  $\left[\mathrm{KDR}^{+}21\right]$  proposes to us the average loss over the training dataset or a significantly diverse validation set. Then, the gradient of loss on this training or validation set is compared with that of the selected mini-batch of data points. In a black-box attack, we do not have access to average loss over a whole training or validation dataset. Thus, it is not feasible to deploy the proposed algorithm.
Following another approach, [MBL20] proposes an elegant pre-processing algorithm to select a core-set of a training dataset. This selection further leads to an efficient incremental gradient based training methodology. But in our case, we sequentially query the black-box model to obtain a label for a query point and use them further for training the extracted model. Thus, creating a dataset beforehand and using them further to pre-process will not lead to an adaptive attack and also will not reduce the query budget. Thus, it is out of scope of our work. [KS22] deploys an auxiliary classifier to first use a subset of labelled datapoints to create low dimensional embeddings. Then according to the query budget, it chooses the points from the sparse regions from each cluster found from the low dimensional embeddings. This is another variant of uncertainty sampling that hypothesises the points from the sparse region are more informative in terms of loss and prediction entropy. This design technique is at the same time model specific, and thus incompatible to our interest. Additionally, this approach also increases the requirement of queries to the target model.
K-Center sampling (KC). Thus, we focus on a version of core-set algorithm, namely K-Center sampling, that is applicable in our context. K-Center sampling as an active learning algorithm that has been originally proposed to train CNNs sample-efficiently [SS18]. For a given data pool, K-Center sampling iteratively selects the  $k$  datapoints that minimise the core set loss (see Equation 3, [SS18]) the most for a given model  $f_{\omega}$ . In this method, the embeddings (from the model under training) of the data points are used as the representative vectors, and K-Center algorithm is applied on these representative vectors.
Random Sampling (RS). In random sampling, a subset of  $k$  datapoints are selected from a data pool uniformly at random.
In our experiments, for query selection at time  $t$ , the extracted model at time  $t - 1$ , i.e.  $f_{t - 1}^{E}$ , is used as  $f_{\omega}$ , and data pool at time  $t$  is the corresponding query dataset, except the datapoints that has been selected before step  $t$ . Hereafter, we deploy a modified version of the framework developed by [Hua21] to run our experiments with the aforementioned active learning algorithms.
D Extended experimental analysis
In this section, we step-wise elaborate further experimental setups and results that we skipped for the brevity of space in the main draft. Specifically, we conduct our experiments in six experimental setups. Each experimental setup corresponds to a triplet (target model architecture trained on a private dataset, extracted model architecture, query dataset). Here, we list these six experimental setups in detail
1. A Logistic Regression (LR) model trained on MNIST, a LR model for extraction, EMNIST dataset for querying
2. A Logistic Regression (LR) model trained on MNIST, a LR model for extraction, CIFAR10 dataset for querying
3. A CNN model trained on MNIST, a CNN model for extraction, EMNIST dataset for querying
4. A ResNet model trained on CIFAR10, a CNN model for extraction, ImageNet dataset for querying
5. A ResNet model trained on CIFAR10, a ResNet18 $^4$  model for extraction, ImageNet dataset for querying
6. A BERT model trained on BBCNews, a BERT model for extraction, AGNews dataset for querying
For each of the experimental setups, we evaluate five types of performance evaluations, which are elaborated in Section D.1, D.2.1, D.2.2, D.3, and D.4. While each of the following sections contain illustrations of the different performance metrics evaluating efficacy of the attack and corresponding discussions, Table 2-4 contain summary of all queries used, accuracy, and membership inference statistics for all the experiments.
D.1 Test accuracy of extracted models
Test accuracy of the extracted model and its comparison with the test accuracy of the target model on a subset of the private training dataset, which was used by neither of these models, is the most common performance metric used to evaluate the goodness of the attack algorithm. The attacks designed solely to optimise this performance metric are called the task accuracy model extraction attacks [JCB+20].
With MARICH, we aim to extract models that have prediction distributions closest to that of the target model. Our hypothesis is constructing such a prediction distribution lead to a model that also has high accuracy on the private test dataset, since accuracy is a functional property of the prediction distribution induced by a classifier. In order to validate this hypothesis, we compute test accuracies of the target models, and models extracted by MARICH and other active sampling algorithms in six experimental setups. We illustrate the evolution curves of accuracies over increasing number of queries in Figure 5.
To compare MARICH with other active learning algorithms, we attack the same target models using K-centre sampling, Least Confidence sampling, Margin Sampling, Entropy Sampling, and Random Sampling algorithms (ref. Appendix C) using the same number of queries as used for MARICH in each setup.
From Figure 5, we observe that in most of the cases MARICH outperforms all other competing algorithms.
In this process, MARICH uses  $\sim 500 - 8000$  queries, which is a small fraction of the corresponding query datasets. This also indicates towards the query-efficiency of MARICH.
Extraction of a ResNet trained on CIFAR10 with a ResNet18. Along with the five experimental setups mentioned in the paper, we trained a ResNet with CIFAR10 dataset  $(\mathbf{D}^T$  here), that shows a test accuracy of  $91.82\%$  on a disjoint test set. We use ImageNet as  $\mathbf{D}^Q$  here, to extract a ResNet18 model from the target model. We have restrained from discussing this setup in the main paper due to brevity of space.
<sup>4</sup>We begin with a pre-trained ResNet18 model from https://torch.org/vision/main/models/generated/torchvision.models.resnet18.html
<sup>5</sup>We use the pre-trained BERT model from https://huggingface.co/bert-base-cased
Target Marich KC LC MS ES RS
(a) Logistic regression with EMNIST queries
(b) Logistic regression with CIFAR10 queries
(c) CNN with EMNIST queries
(d) ResNet with CNN and ImageNet queries
(e) ResNet with ResNet18 and ImageNet queries
Figure 5: Comparison of test accuracies achieved by models extracted by different active sampling algorithms.
(f) BERT with AGNews queries
MARICH extracts a ResNet18 using 8429 queries, and the extracted ResNet18 shows test accuracy of  $71.65 \pm 0.88\%$ . On the other hand, models extracted using Best of Competitors (BoC) using ImageNet queries show accuracy of  $70.67 \pm 0.12\%$ .
D.2 Fidelity of the prediction distributions of The extracted models
Driven by the distributional equivalence extraction principle, the central goal of MARICH is to construct extracted models whose prediction distributions are closest to the prediction distributions of corresponding target models. From this perspective, in this section, we study the fidelity of the prediction distributions of models extracted by MARICH and other active sampling algorithms, namely K-centre sampling, Least Confidence sampling, Margin Sampling, Entropy Sampling, and Random Sampling.
D.2.1 KL-divergence between prediction distributions
First, as the metric of distributional equivalence, we evaluate the KL-divergence between the prediction distributions of the models extracted by MARICH and other active sampling algorithms. In Figure 6, we report the box-plot (mean, median  $\pm$  25 percentiles) of KL-divergences (in log-scale) calculated from 5 runs for each of 10 models extracted by each of the algorithms.
(a) Logistic regression with EMNIST queries
(b) Logistic regression with CIFAR10 queries
(c) CNN with EMNIST queries
Figure 6: Comparison of KL-divergences (in log-scale) between the target prediction distributions and prediction distributions of the models extracted by different active learning algorithms.
(e) ResNet with ResNet18 and ImageNet queries
(f) BERT with AGNews queries
Results. Figure 6 shows that the KL-divergence achieved by the prediction distributions of models extracted using MARICH are at least  $\sim 2 - 10$  times less than that of the other competing algorithms. This validates our claim that MARICH yields distributionally closer extracted model  $f^{E}$  from the target model  $f^{T}$  than existing active sampling algorithms.
D.2.2 Prediction agreement
(c) CNN with EMNIST queries
(e) ResNet with ResNet18 and ImageNet queries
Figure 7: Comparison of agreement in predictions (in %) between the target model and the models extracted by different active learning algorithms.
(d) ResNet with CNN and ImageNet queries
(f) BERT with AGNews queries
In Figure 7, we illustrate the agreement in predictions of  $f^{E}$  with  $f^{T}$  on test datasets using different active learning algorithms. Prediction agreement functions as another metric of fidelity of prediction distributions constructed by extracted models in comparison with those of the target models.
Similar to Figure 6, we report the box-plot (mean, median  $\pm 25$  percentiles) of prediction agreements (in  $\%$  ) calculated from 5 runs for each of 10 models extracted by each of the algorithms.
Results. We observe that the prediction distributions extracted by MARICH achieve almost same to  $\sim 30\%$  higher prediction agreement in comparison with the competing algorithms. Thus, we infer that in this particular case MARICH achieves better fidelity than the other active sampling algorithms, in some instances, while it is similar to the BoC in some instances.