SlowGuess/NeurIPS_2023_OCR huggingface dataset:
D.3 Fidelity of parameters of the extracted models
(a) Logistic regression with EMNIST queries
(c) CNN with EMNIST queries
Figure 8: Comparison of parametric fidelity for MARICH and different active learning algorithms.
(b) Logistic regression with CIFAR10 queries
(d) BERT with AGNews queries
Generating extracted models with low parametric fidelity is not a main goal or basis of the design principle of MARICH. Since parametric fidelity is a popularly studied metric to evaluate goodness of model extraction, in Figure 8, we depict the parametric fidelity of models extracted by different active learning algorithms.
Let  $\mathbf{w}_E$  be the parameters of the extracted model and  $\mathbf{w}_T$  be the parameters of the target model. We define parametric fidelity as  $F_{\mathbf{w}} \triangleq \log \| \mathbf{w}_E - \mathbf{w}_T \|_2$ . Since the parametric fidelity is only computable when the target and extracted models share the same architecture, we report the four instances here where MARICH is deployed with the same architecture as that of the target model. For logistic regression, we compare all the weights of the target and the extracted models. For BERT and CNN, we compare between the weights in the last layers of these models.
Results. For LR, we observe that the LR models extracted by MARICH have 20 - 30 times lower parametric fidelity than the extracted LR models of the competing algorithms. For BERT, the BERT extracted by MARICH achieves 0.4 times lower parametric fidelity than the Best of Competitors (BoC). As an exception, for CNN, the model extracted by K-center sampling achieves 0.996 times less parametric fidelity than that of MARICH.
Thus, we conclude that MARICH as a by-product of its distributionally equivalent extraction principle also extracts model with high parametric fidelity, which is often better than the competing active sampling algorithms.
D.4 Membership inference with the extracted models
A main goal of MARICH is to conduct a Max-Information attack on the target model, i.e. to extract an informative replica of its predictive distribution that retains the most information about the private training dataset. Due to lack of any direct measure of informativeness of an extracted model with respect to a target model, we run Membership Inference (MI) attacks using the models extracted by MARICH, and other competing active sampling algorithms. High accuracy and agreement in MI attacks conducted on extracted models of MARICH and the target models implicitly validate our claim that MARICH is able to conduct a Max-Information attack.
Observation 1. From Figure 9, we see that in most cases the probability densities of the membership inference are closer to the target model when the model is extracted using MARICH, than using all other active sampling algorithms (BoC, Best of Competitors).
(d) ResNet with CNN with ImageNet queries
(e) ResNet with ResNet18 with ImageNet queries
(f) BERT with AGNews queries
Figure 9: Comparison among membership vs. non-membership probability densities for membership attacks against models extracted by MARICH, the best of competitors (BoC) and the target model. Each figure represents the model class and query dataset. Memberships and non-memberships inferred from the model extracted by MARICH are significantly closer to the target model.
Observation 2. In Figure 10, we present the agreements from the member points, nonmember points and overall agreement curves for varying membership thresholds, along with the AUCs of the overall membership agreements. We see that in most cases, the agreement curves for the models extracted using MARICH are above those for the models extracted using random sampling, thus AUCs are higher for the models extracted using MARICH.
(e) ResNet with ResNet18 & ImageNet queries
(f) BERT with AGNews queries
Figure 10: Comparison of membership, nonmembership and overall agreements of membership attacks against models extracted by MARICH and the best of competitors with the target model trained with MNIST. Each figure represents the model class and query dataset. Membership agreement of the models extracted by MARICH are higher.
Observation 3. In Table 2, 3, and 4, we summarise the MI accuracy on the private training dataset, Nonmembership inference accuracy on the private training dataset, Agreement in MI w.r.t. the MI on the target model, and AUC of Agreement in MI with that of the target model for Logistic Regression (LR), CNN, ResNet, and BERT target models. We observe that, while compared with other active sampling algorithms, out of 6 combinations of (target model, extracted model, query dataset) under study, the models extracted by MARICH achieve the highest accuracy in MI and agreement in MI w.r.t. the target model, in most of the instances.
Results. These observations support our claim that model extraction using MARICH gives models are accurate and informative replica of the target model.
Table 2: Model extraction and membership inference statistics for extracting a Logistic Regression model and a CNN
Member Dataset	Target Model	Attack Dataset	Algorithm used	Non-member Dataset	#Queries	Membership acc	Nonmembership acc	Overall membership acc	Overall membership agreement	Membership agreement AUC	Accuracy
MNIST	LR	-	-	EMNIST	-	95.37%	65.84%	87.99%	-	-	90.82%
MNIST	LR	EMNIST	Marich	EMNIST	1863	93.43%	57.58%	84.47%	90.34%	90.89%	73.98+5.96%
MNIST	LR	EMNIST	ES	EMNIST	1863	97.22%	15.88%	75.88%	84.39%	82.32%	49.37+4.24%
MNIST	LR	EMNIST	KC	EMNIST	1863	97.80%	14.56%	76.99%	87.84%	82.30%	47.62+6.25%
MNIST	LR	EMNIST	MS	EMNIST	1863	97.63%	18.42%	77.83%	86.49%	82.74%	52.60+3.55%
MNIST	LR	EMNIST	LC	EMNIST	1863	95.64%	25.10%	78.00%	80.11%	83.07%	51.968+5.05%
MNIST	LR	EMNIST	RS	EMNIST	1863	98.23%	11.92%	76.65%	88.08%	82.26%	46.91+5.69%
MNIST	LR	-	-	CIFAR10	-	99.11%	94.76%	98.02%	-	-	90.82%
MNIST	LR	CIFAR10	Marich	CIFAR10	959	97.72%	92.14%	96.32%	96.89%	94.32%	81.06+0.56%
MNIST	LR	CIFAR10	ES	CIFAR10	959	96.47%	86.16%	93.89%	96.53%	91.29%	79.09+1.82%
MNIST	LR	CIFAR10	KC	CIFAR10	959	95.87%	81.64%	92.31%	92.32%	89.64%	56.26+7.97%
MNIST	LR	CIFAR10	MS	CIFAR10	959	96.79%	82.82%	93.30%	93.16%	90.92%	77.93+2.94%
MNIST	LR	CIFAR10	LC	CIFAR10	959	96.19%	86.26%	93.71%	93.67%	91.53%	73.78+2.95%
MNIST	LR	CIFAR10	RS	CIFAR10	959	95.84%	81.20%	92.18%	92.17%	90.11%	69.18+3.72%
MNIST	CNN	-	-	EMNIST	-	93.71%	78.76%	89.97%	-	-	94.83%
MNIST	CNN	EMNIST	Marich	EMNIST	1863	94.55%	73.66%	90.62%	87.27%	86.72%	86.83+1.62%
MNIST	CNN	EMNIST	ES	EMNIST	1863	95.44%	73.66%	89.99%	87.11%	86.38%	83.90+3.10%
MNIST	CNN	EMNIST	KC	EMNIST	1863	95.35%	74.48%	90.13%	87.49%	85.94%	84.94+2.44%
MNIST	CNN	EMNIST	MS	EMNIST	1863	94.59%	79.04%	90.70%	86.74%	86.08%	82.72+4.47%
MNIST	CNN	EMNIST	LC	EMNIST	1863	94.39%	76.30%	89.86%	86.75%	86.10%	85.53+2.45%
MNIST	CNN	EMNIST	RS	EMNIST	1863	94.17%	80.42%	90.73%	87.53%	86.97%	82.52+4.87%
Table 3: Model extraction and membership inference statistics for extracting a ResNet
Member Dataset	Target Model	Attack Model	Attack Dataset	Algorithm used	Non-member Dataset	#Queries	Membership acc	Nonmembership acc	Overall membership acc	Overall membership agreement	Membership agreement AUC	Accuracy
CIFAR10	ResNet	-	-	-	ImageNet	-	97.70%	56.80%	93.61%	-	-	91.82%
CIFAR10	ResNet	CNN	ImageNet	Marich	ImageNet	8429	70.84%	68.44%	69.64%	64.14%	71.78%	56.11+1.34%
CIFAR10	ResNet	CNN	ImageNet	ES	ImageNet	8429	65.00%	71.62%	68.31%	62.94%	71.36%	40.11+1.94%
CIFAR10	ResNet	CNN	ImageNet	KC	ImageNet	8429	65.80%	71.98%	68.89%	63.68%	71.53%	37.21+3.41%
CIFAR10	ResNet	CNN	ImageNet	MS	ImageNet	8429	68.48%	71.96%	70.22%	64.48%	71.97%	41.27+0.85%
CIFAR10	ResNet	CNN	ImageNet	LC	ImageNet	8429	70.60%	71.68%	71.14%	65.47%	72.29%	42.05+1.38%
CIFAR10	ResNet	CNN	ImageNet	RS	ImageNet	8429	67.56%	68.68%	68.12%	63.41%	71.39%	40.66+2.58%
CIFAR10	ResNet	ResNet18	ImageNet	Marich	ImageNet	8429	99.27%	10.54%	90.40%	93.84%	76.51%	71.65+0.88%
CIFAR10	ResNet	ResNet18	ImageNet	ES	ImageNet	8429	100.00%	0.02%	90.00%	97.29%	72.33%	69.92+1.54%
CIFAR10	ResNet	ResNet18	ImageNet	KC	ImageNet	8429	99.79%	2.06%	90.02%	94.06%	72.40%	70.67+0.12%
CIFAR10	ResNet	ResNet18	ImageNet	MS	ImageNet	8429	99.99%	0.28%	90.02%	96.28%	72.81%	70.66+2.29%
CIFAR10	ResNet	ResNet18	ImageNet	LC	ImageNet	8429	99.99%	0.04%	90.00%	96.77%	71.94%	70.26+1.44%
CIFAR10	ResNet	ResNet18	ImageNet	RS	ImageNet	8429	99.94%	1.40%	90.08%	95.41%	72.94%	70.49+2.02%
Table 4: Model extraction and membership inference statistics for extracting a BERT model
Member Dataset	Target Model	Attack Model	Attack Dataset	Algorithm used	Non-member Dataset	Queries	Membership acc	Nonmembership acc	Overall membership acc	Overall membership agreement	Membership agreement AUC	Accuracy
BBC News	BERT	-	-	-	AG News	-	-	-	-	-	-	98.62%
BBC News	BERT	BERT	AG News	Marich	AG News	474	83.60%	66.80%	75.20%	74.80%	65.14%	85.45+5.96%
BBC News	BERT	BERT	AG News	ES	AG News	474	82.80%	48.40%	65.60%	66.00%	59.36%	79.25+12.67%
BBC News	BERT	BERT	AG News	KC	AG News	474	68.40%	55.60%	62.00%	63.39%	61.10%	70.45+15.94%
BBC News	BERT	BERT	AG News	MS	AG News	474	77.20%	72.80%	75.00%	62.40%	71.07%	72.82+11.59%
BBC News	BERT	BERT	AG News	LC	AG News	474	80.00%	43.20%	61.60%	74.80%	57.93%	78.65+7.22%
BBC News	BERT	BERT	AG News	RS	AG News	474	78.00%	47.60%	62.80%	63.80%	58.43%	76.31+16.78%
E Significance and comparison of three sampling strategies
Given the bi-level optimization problem, we came up with MARICH in which three sampling methods are used in the order: (i) ENTROPYSAMPLING, (ii) ENTROPYGRADIENTSAMPLING, and (iii) LOSSAMPLING.
These three sampling techniques contribute to different goals:
- ENTROPY SAMPLING selects points about which the classifier at a particular time step is most confused
- ENTROPYGRADIENTSAMPLING uses gradients of entropy of outputs of the extracted model w.r.t. the inputs as embeddings and selects points behaving most diversely at every time step.
- LOSSAMPLING selects points which produce highest loss when loss is calculated between target model's output and extracted model's output.
One can argue that the order is immaterial for the optimization problem. But looking at the algorithm practically, we see that ENTROPYGRADIENTSAMPLING and LOSSAMPLING incur much higher time complexity than ENTROPYSAMPLING. Thus, using ENTROPYSAMPLING on the entire query set is more efficient than the others. This makes us put ENTROPYSAMPLING as the first query selection strategy.
As per the optimization problem in Equation (7), we are supposed to find points that show highest mismatch between the target and the extracted models after choosing the query subset maximising the entropy. This leads us to the idea of LOSSAMPLING. But as only focusing on loss between models may choose points from one particular region only, and thus, decreasing the diversity of the queries. We use ENTROPYGRADIENTSAMPLING before LOSSAMPLING. This ensures selection of diverse points with high performance mismatch.
In Figure 11, we experimentally see the time complexities of the three components used. These are calculated by applying the sampling algorithms on a logistic regression model, on mentioned slices of MNIST dataset.
Figure 11: Runtime comparison of three sampling strategies to select queries from 4500 datapoints.
Table 5: Time complexity of different sampling Strategies
Sampling Algorithm	Query space size	#Selected queries	Time (s)
Entropy Sampling	4500	100	1.82 ± 0.04
Entropy-Gradient Sampling	4500	100	10.56 ± 0.07
Loss Sampling	4500	100	41.64 ± 0.69
F Performance against differentially private target models
In this section, we aim to verify performance of MARICH against privacy-preserving mechanisms. Specifically, we apply a  $(\varepsilon, \delta)$ -Differential Privacy (DP) inducing mechanism [DMNS06, DBB21] on the target model to protect the private training dataset. There are three types of methods to ensure DP: output perturbation [DMNS06], objective perturbation [CMS11, DBB18], and gradient perturbation  $\left[\mathrm{ACG}^{+}16\right]$ . Since output perturbation and gradient perturbation methods scale well for nonlinear deep networks, we focus on them as the defense mechanism against MARICH's queries.
Gradient perturbation-based defenses. DP-SGD  $\left[\mathrm{ACG}^{+}16\right]$  is used to train the target model on the member dataset. This mechanism adds noise to the gradients and clip them while training the target model. We use the default implementation of Opacus  $\left[\mathrm{YSS}^{+}21\right]$  to conduct the training in PyTorch.
Following that, we attack the  $(\varepsilon, \delta)$ -DP target models using MARICH and compute the corresponding accuracy of the extracted models. In Figure 12, we show the effect of different privacy levels  $\varepsilon$  on the achieved accuracy of the extracted Logistic Regression model trained with MNIST dataset and queried with EMNIST dataset. Specifically, we assign  $\delta = 10^{-5}$  and vary  $\varepsilon$  in  $\{0.2, 0.5, 1, 2, \infty\}$ . Here,  $\varepsilon = \infty$  corresponds to the model extracted from the non-private target model.
We observe that the accuracy of the models extracted from private target models are approximately  $2.3 - 7.4\%$  lower than the model extracted from the non-private target model. This shows that performance of MARICH decreases against DP defenses but not significantly.
Figure 12: Performance of models extracted by MARICH against  $(\varepsilon, \delta)$ -differentially private target models trained using DP-SGD. We consider different privacy levels  $\varepsilon$  and  $\delta = 10^{-5}$ . Accuracy of the extracted models decreases with increase in privacy (decrease in  $\epsilon$ ).
Output perturbation-based defenses. Perturbing output of an algorithm against certain queries with calibrated noise, in brief output perturbation, is one of the basic and oldest form of privacy-preserving mechanism [DMNS06]. Here, we specifically deploy the Laplace mechanism, where a calibrated Laplace noise is added to the output of the target model generated against some queries. The noise is sampled from a Laplace distribution  $\mathrm{Lap}(0,\frac{\Delta}{\varepsilon})$ , where  $\Delta$  is sensitivity of the output and  $\varepsilon$  is the privacy level. This mechanism ensures  $\varepsilon$ -DP.
We compose a Laplace mechanism to the target model while responding to MARICH's query and evaluate the change in accuracy of the extracted model as the impact of the defense mechanism. We use a logistic regression model trained on MNIST as the target model. We query it using EMNIST and CIFAR10 datasets respectively. We vary  $\varepsilon$  in  $\{0.25, 2, 8, \infty\}$ . For each  $\varepsilon$  and query dataset, we report the mean and standard deviation of accuracy of the extracted models on a test dataset. Each experiment is run 10 times.
We observe that decrease in  $\varepsilon$ , i.e. increase in privacy, causes decrease in accuracy of the extracted model. For EMNIST queries (Figure 13a), the degradation in accuracy is around  $10\%$  for  $\varepsilon = 2, 8$  but we observe a significant drop for  $\varepsilon = 0.25$ . For CIFAR10 queries (Figure 13b),  $\varepsilon = 8$  has practically no impact on the performance of the extracted model. But for  $\varepsilon = 2$  and  $0.25$ , the accuracy of extracted models drop down very fast.
Thus, we conclude that output perturbation defends privacy of the target model against MARICH for smaller values of  $\varepsilon$ . But for larger values of  $\varepsilon$ , the privacy-preserving mechanism might not save the target model significantly against MARICH.
(a) LR with EMNIST queries
(b) LR with CIFAR10 queries
Figure 13: Performance of models extracted by MARICH against target models that perturbs the output of the queries to achieve  $\varepsilon$ -DP. We consider different privacy levels  $\varepsilon$ . Accuracy of the extracted models decreases with increase in privacy (decrease in  $\epsilon$ ).
G Effects of model mismatch
From Equation (7), we observe that functionality of MARICH is not constrained by selecting the same model class for both the target model  $f^T$  and the extracted model  $f^E$ . To elaborately study this aspect, in this section, we conduct experiments to show MARICH's capability to handle model mismatch and impact of model mismatch on performance of the extracted models.
Specifically, we run experiments for three cases: (1) extracting an LR model with LR and CNN, (2) extracting a CNN model with LR and CNN, and (3) extracting a ResNet model with CNN and ResNet18. We train the target LR and the target CNN model on MNIST dataset. We further extract these two models using EMNIST as the query datasets. We train the target ResNet model on CIFAR10 dataset and extract it using ImageNet queries. The results on number of queries and achieved accuracies are summarised in Table 6 and Figure 14.
Observation 1. In all the three experiments, we use MARICH without any modification for both the cases when the model classes match and mismatch. This shows universality and model-obliviousness of MARICH as a model extraction attack.
Observation 2. From Figure 14, we observe that model mismatch influences performance of the model extracted by MARICH. When we extract the LR target model with LR and CNN, we observe that both the extracted models achieve almost same accuracy and the extracted CNN model achieves even a bit more accuracy than the extracted LR model. In contrast, when we extract the CNN target model with LR and CNN, we observe that the extracted LR model achieves lower accuracy than the extracted CNN model. Similar observations are found for extracting the ResNet with ResNet18 and CNN, respectively.
Conclusions. From these observations, we conclude that MARICH can function model-obliviously. We also observe that if we use a less complex model to extract a more complex model, the accuracy drops significantly. But if we extract a low complexity model with a higher complexity one, we obtain higher accuracy instead of model mismatch. This is intuitive as the low-complexity extracted model might have lower representation capacity to mimic the non-linear decision boundary of the high-complexity model but the opposite is not true. In future, it would be interesting to delve into the learning-theoretic origins of this phenomenon.
(a) LR extracted by LR vs. LR extracted by CNN
(b) CNN extracted by CNN vs. CNN extracted by LR
(c) ResNet extracted by CNN vs. ResNet extracted by ResNet18
Figure 14: Effect of model mismatches on models extracted by MARICH.
Table 6: Effect of model mismatch on accuracy of The extracted models.
fE	fT	#samples	Accuracy
LR	LR	5130	82.37 ± 5.7%
LR	CNN	5130	85.41 ± 0.57%
CNN	LR	5440	57.81 ± 3.64%
CNN	CNN	5440	91.63 ± 0.42%
ResNet	CNN	8429	56.11 ± 1.35%
ResNet	ResNet18	8429	71.65 ± 0.88%
H Choices of hyperparameters
In this section, we list the choices of the hyperparameters of Algorithm 1 for different experiments and also explain how we select them.
Hyperparameters  $\gamma_{1}$  and  $\gamma_{2}$  are kept constant, i.e., 0.8, for all the experiments. These two parameters act as the budget shrinking factors.
Instead of changing these two, we change the number of points  $n_0$ , which are randomly selected in the beginning, and the budget  $B$  for every step. We obtain the optimal hyperparameters for each experiment by performing a line search in the interval [100, 500].
We further change the budget over the rounds. At time step  $t$ , the budget,  $B_{t} = \alpha^{t} \times B_{t-1}$ . The idea is to select more points as  $f^{E}$  goes on reaching the performance of  $f^{T}$ . Here,  $\alpha > 1$  and needs to be tuned. We use  $\alpha = 1.02$ , which is obtained through a line search in [1.01, 1.99].
For number of rounds  $T$ , we perform a line search in [10, 20].
Table 7: Hyperparameters for different datasets and target models.
Member Dataset	Target Model	Attack Model	Attack Dataset	Budget	Initial points	γ1	γ2	Rounds	Epochs/Round	Learning Rate
MNIST	LR	LR	EMNIST	250	300	0.8	0.8	10	10	0.02
MNIST	LR	LR	CIFAR10	50	100	0.8	0.8	10	10	0.02
MNIST	CNN	CNN	EMNIST	550	500	0.8	0.8	10	10	0.015
MNIST	CNN	CNN	CIFAR10	750	500	0.8	0.8	10	10	0.03
CIFAR10	ResNet	CNN	ImageNet	750	500	0.8	0.8	10	8	0.2
CIFAR10	ResNet	ResNet18	ImageNet	750	500	0.8	0.8	10	8	0.02
BBC News	BERT	BERT	AG News	60	100	0.8	0.8	6	3	5 × 10-6