SlowGuess/NeurIPS_2023_OCR huggingface dataset:
where the inner sup optimization in Eq. (4) is to compute and simulate the worst-case test data distribution (denoted as  $\pi$ ) around the training data distribution (denoted as  $\mu_0$ ).  $\pi$  is defined as the probability distribution that achieves  $\sup_{\mu \in \mathcal{P}} \mathbb{E}_{\boldsymbol{x} \sim \mu} \mathcal{L}(\boldsymbol{\theta}_T, \boldsymbol{x} + h_\omega(\boldsymbol{x}, \epsilon), y)$ .  $\mathcal{P}$  in Eq. (5) denotes the ambiguity set of probability measures for the test data distribution to characterize its uncertainty. One common choice to define  $\mathcal{P}$  is through KL divergence.  $\mathbb{KL}(\mu || \mu_0)$  denotes the KL divergence between probability measure  $\mu_0$  and  $\mu$ .  $\beta$  is a constant to characterize the closeness between  $\mu_0$  and  $\mu$  to ensure the worst-case test data distribution  $\pi$  does not deviate from the raw training data distribution  $\mu_0$  much. Eq. (6) ensures that model utility (output class probabilities, i.e.,  $T(\boldsymbol{x})$ ) does not change much after adding random perturbation, i.e.,  $T(\boldsymbol{x} + h_\omega(\boldsymbol{x}, \epsilon))$ . The probability perturbation magnitude is measured by  $l_1$  norm, i.e.,  $||\cdot||_1$ . It is crucial to guarantee target model output class probabilities after query perturbation close to that of without query perturbation since benign users need these informative probabilities to derive helpful knowledge [33].  $\tau$  is a constant.
To solve the above optimization problem, we convert Eq. (4-6) into the following unconstrained optimization problem by Lagrange duality [6] (detailed derivations are put in Appendix 11.1):
$$
\min  _ {\boldsymbol {\theta} _ {T}, \omega} \sup  _ {\mu} \left[ \mathbb {E} _ {\boldsymbol {x} \sim \mu} \mathcal {L} \left(\boldsymbol {\theta} _ {T}, \boldsymbol {x} + h _ {\omega} (\boldsymbol {x}, \epsilon), y\right) - \mathbb {K L} \left(\mu \mid \mid \mu_ {0}\right) \right] + \gamma \mathbb {E} _ {\boldsymbol {x} \sim \mu} | | T (\boldsymbol {x} + h _ {\omega} (\boldsymbol {x}, \epsilon)) - T (\boldsymbol {x}) | | _ {1} \tag {7}
$$
The constant  $\gamma$  controls the regularization magnitude for model utility. The KL-divergence  $\mathbb{KL}(\mu ||\mu_0)$  is handled by Wasserstein gradient flow (WGF) (presented in the following sections), therefore for simplicity, the regularization weight for it is set to 1.0 throughout the paper. We name Eq. (7) as Defense DRO. The optimization in Eq. (7) is still challenging to solve as the inner sup optimization is over probability measure space, which is an infinite-dimensional function space.
Solution to the Defense DRO. To make the solution of the Defense DRO (Eq. (7)) tractable, we reformulate it from a new continuous dynamics perspective. To achieve this goal, we first define the energy functional  $F(\mu)$  as follows:
$$
F (\mu) = V (\mu) + \mathbb {K L} (\mu | | \pi) \tag {8}
$$
where  $V(\mu) = -\mathbb{E}_{\boldsymbol{x}\sim \mu}\mathcal{L}(\pmb {\theta}_T,\pmb {x} + h_\omega (\pmb {x},\pmb {\epsilon}),y)$ . By defining such energy functional  $F(\mu)$ , the Eq. (7) can be equivalently formulated by the following gradient flow system Eq. (9, 10):
$$
\left\{ \begin{array}{l} \partial_ {t} \mu_ {t} = \nabla_ {W _ {2}} F (\mu_ {t}) := d i v \left(\mu_ {t} \nabla \frac {\delta F}{\delta \mu} (\mu_ {t})\right); \\ \frac {d \boldsymbol {\theta} _ {T}}{d t} = - \nabla_ {\boldsymbol {\theta} _ {T}} \left[ \mathbb {E} _ {\boldsymbol {x} \sim \mu_ {t}} \mathcal {L} \left(\boldsymbol {\theta} _ {T}, \boldsymbol {x} + h _ {\boldsymbol {\omega}} (\boldsymbol {x}, \boldsymbol {\epsilon}), y\right) + \gamma \mathbb {E} _ {\boldsymbol {x} \sim \mu_ {t}} | | T (\boldsymbol {x} + h _ {\boldsymbol {\omega}} (\boldsymbol {x}, \boldsymbol {\epsilon})) - T (\boldsymbol {x}) | | _ {1} \right], \end{array} \right. \tag {9}
$$
where Eq. (9) solves the inner sup problem in Eq. (7) with WGF in Wasserstein space (Details provided in Appendix 11.2) and Eq. (10) solves the outer minimization problem in Eq. (7) for parameter update with gradient flow in Euclidean space. Below, we propose a method for efficiently solving the Eq. (9). We view each training data as one particle and arrive at the following test data distribution simulation solution (solution to Eq. (9)) (Details are provided in Appendix 11.2):
$$
\boldsymbol {x} _ {t + 1} ^ {i} - \boldsymbol {x} _ {t} ^ {i} = - \frac {\alpha}{N} \sum_ {j = 1} ^ {j = N} \left[ k \left(\boldsymbol {x} _ {t} ^ {i}, \boldsymbol {x} _ {t} ^ {j}\right) \nabla_ {\boldsymbol {x} _ {t} ^ {j}} U \left(\boldsymbol {x} _ {t} ^ {j}, \boldsymbol {\theta} _ {T}\right) + \nabla_ {\boldsymbol {x} _ {t} ^ {j}} k \left(\boldsymbol {x} _ {t} ^ {i}, \boldsymbol {x} _ {t} ^ {j}\right) \right] \tag {11}
$$
where  $\pmb{x}_t^i$  denotes the  $i^{th}$  training data  $\pmb{x}^i$  perturbed at time  $t$ ;  $\alpha$  is the data transformation rate.  $U(\pmb{x},\pmb{\theta}_T) = -\mathcal{L}(\pmb{\theta}_T,\pmb{x},y)$ .  $k(\pmb{x}_i,\pmb{x}_j)$  is the Gaussian kernel, i.e.,  $k(\pmb{x}_i,\pmb{x}_j) = \exp \left(-\frac{(\pmb{x}_i - \pmb{x}_j)^2}{2\sigma^2}\right)$ . The first term in the RHS of Eq. (11) moves the training data towards the worst-case simulated test data distribution by raising the energy functional. The data distribution change is controlled by the sum of gradients from the mini-batch of the training data, which are weighted by the kernel function  $k(\pmb{x}_t^i,\pmb{x}_t^j)$ , smoothing the gradients of training data. The second term (kernel gradient) acts as a repelling force that keeps the transformed data from concentrating on a single data point, thus diversifying the simulated test data.
4.2 End-to-end Defensive Training for DFME
We summarize the defensive training algorithm in Algorithm 1. Line 3-4 simulates the worst-case test data distribution. Line 5-8 adds data-dependent random perturbation to the simulated test data and trains the target model and perturbation generator on the simulated test data.
MeCo Deployment. During testing, we perform similar perturbation for each query. Given any query input  $\pmb{x}$ ,  $\pmb{y}_p = T_{\theta_T}(\pmb{x} + h_\omega(\pmb{x}, \pmb{\epsilon}))$ ;  $\pmb{\epsilon} \sim N(0, I)$ . Then, return the results  $\pmb{y}_p$  to the user.
Algorithm 1 MeCo Training.
1: REQUIRE: Target model  $T$  with parameters  $\theta_T$ ; data-dependent perturbation generator  $h_\omega$  with parameters  $\omega$ ; target model learning rate  $\eta$ ;  $Q$  is the number of test data simulation steps;  $D_{tr}$  is the training dataset.
2: for  $k = 1$  to  $M$  do  $\{M$  is the number of training iterations\}
3: randomly sample a new mini-batch data  $(\pmb{x},y)$  from  $\mathcal{D}_{tr}$
4: perturb mini-batch data to be  $(\pmb{x}_Q, y) Q$  steps by Eq. (11).
5: add random perturbation to  $(\pmb{x}_Q, y)$  to be  $(\pmb{x}_Q + h_\omega(\pmb{x}_Q, \epsilon), y)$
6: calculate the loss  $g(\pmb{\theta}_T, \pmb{\omega}) = \mathcal{L}(\pmb{\theta}_T^k, \pmb{x}_Q + h_{\omega}(\pmb{x}_Q, \epsilon), y) + \gamma ||T(\pmb{x} + h_{\omega}(\pmb{x}, \epsilon)) - T(\pmb{x})||_1$ .
7: train the target model by  $\pmb{\theta}_T^{k + 1} = \pmb{\theta}_T^k -\eta \nabla_{\pmb{\theta}_T}[g(\pmb {\theta}_T,\pmb {\omega})]$
8: train the perturbation generator by  $\omega^{k + 1} = \omega^k -\eta \nabla_\omega [g(\pmb {\theta}_T,\omega)]$
9: end for
4.3 Why Can MeCo Defend against DFME
Below, we give explanations on why Algorithm 1 could intrinsically defend against both score-based and decision-based DFME from the perspective: (1) mismatch the KD loss for the attacker; (2) disturb the zeroth-order gradient estimation; (3) change the label prediction on attack query.
Mismatch the KD learning objective. The KL-divergence loss Eq. (1) is critical to match the probability output between the target and clone model. By adding data-dependent random perturbation, we can encourage the mismatch of the KD loss between the target model and clone model, i.e.,
$$
\mathbb {K L} \left(T (\boldsymbol {x} + h _ {\omega} (\boldsymbol {x}, \boldsymbol {\epsilon}); \boldsymbol {\theta} _ {T}), C (\boldsymbol {x}; \boldsymbol {\theta} _ {C})\right) \tag {12}
$$
The mismatch would mislead the attacker to learn in the wrong direction since they think that  $T(\pmb{x} + h_{\omega}(\pmb{x}, \epsilon); \pmb{\theta}_T)$  is the desired output for  $\pmb{x}$  but should be  $T(\pmb{x}, \pmb{\theta}_T)$ , as illustrated in Figure 1.
Disturb the zeroth-order gradient estimation. By adding data-dependent random perturbation, the zeroth-order gradient estimation of score-based DFME  $\left(\frac{\partial\mathcal{L}_G}{\partial x}\right.$  in Eq. (3)) becomes:
$$
\frac {\partial \mathcal {L} _ {G}}{\partial \boldsymbol {x}} = \frac {1}{m} \sum_ {i = 1} ^ {m} \frac {\mathcal {L} _ {G} \left(\boldsymbol {x} + \delta \boldsymbol {\mu} _ {i} + \boldsymbol {v} _ {i} ^ {1}\right) - \mathcal {L} _ {G} \left(\boldsymbol {x} + \boldsymbol {v} _ {i} ^ {2}\right)}{\delta} \boldsymbol {\mu} _ {i}, \tag {13}
$$
where  $\pmb{v}_i^1$  and  $\pmb{v}_i^2$  are two data-dependent perturbations correspond to the two model inputs  $\pmb{x} + \delta \pmb{\mu}_i$  and  $\pmb{x}$ , respectively. Compared with Eq. (3), Eq. (13) would make inaccurate gradient estimation by changing the gradient estimation direction due to the inconsistent random perturbation added to different inputs. Consequently, this would lead the gradient of the pseudo data generator in the wrong direction, i.e.,  $\nabla_{\theta_G} \mathcal{L}_G$  in Eq. (3) would be inaccurate. Then, the generator would not generate informative samples. The uninformative pseudo samples would cause the clone model Eq. (2) to learn incorrect information.
Therefore, by jointly (i) mismatching the KD loss for the attacker; (ii) disturbing the zeroth-order gradient estimation, MeCo can effectively defend against DFME. In summary, MeCo has many advantages compared to existing methods since it does not need to: (1) solve a complex optimization during testing; (2) store multiple models for ensemble; (3) detect the attack query from the benign query; (4) know the attack query data distribution. Due to the space limitations, we additionally show why MeCo can defend against decision-based DFME methods by changing the label prediction on attack query in Appendix 8.
Theoretical Analysis In this section, we delve into the theoretical analysis of our proposed method. We assess the effectiveness of our approach by analyzing it from the perspective of gradient-bias injection [2]. As depicted in Figure 4, our method progressively introduces perturbations into the gradients calculated by the attacker. This process increases the optimization error that accumulates during the model-stealing process, rendering it more challenging for the attacker to successfully
(b)
Figure 4: Theoretical explanations illustration: (a) We showcase the optimization trajectory employing the ground truth loss function that the attacker should ideally adopt (it remains inaccessible to the attacker owing to the unknown input perturbation generator). The cloned model converges toward the globally optimal stolen model, effectively emulating the target victim model. (b) Contrastingly, we depict the optimization trajectory utilizing a noisy and imprecise loss function, which is the actual choice of the attacker. The introduction of gradient bias due to the stochastic input perturbation generator causes the cloned model to deviate from the optimal stolen model, leading to ineffective model theft.
extract the target model. For a more comprehensive understanding, we have provided detailed theoretical analysis in the Appendix due to space constraints.
Datasets. We perform experiments on four standard datasets used in DFME literature, including MNIST, CIFAR10, CIFAR100 [25] and MiniImageNet [52] (100 classes).
Baselines. We compare SOTA DFME and defense baselines. Attack Baselines: (1) decision-based DFME methods: DFMS-HL [47]. We do not compare to ZSDB3KD [51] since it requires a much larger number of queries and is very slow. (2) score-based DFME methods: MAZE [21] and DFME [51]. Defense Baselines: we compare to: (1) undefended: the target model without using any defense strategy; (2) Random Perturb (RandP) [38]: randomly perturb the output probabilities. (3)  $P$ -poison [38]; (4) GRAD [33]: gradient redirection defense. We set a large  $l_{1}$  perturbation budget equal to 1.0 for those defense baselines in the experiments
Table 1: Clone model accuracy after applying defense methods on CIFAR-10 and CIFAR-100 with ResNet34-8x as the target model, which provides soft label
Attack	Defense	CIFAR10 Clone Model Architecture
ResNet18-8X	MobileNetV2	DenseNet121
DFME	undefended ↓	87.36 ± 0.78%	75.23 ± 1.53%	73.89 ± 1.29%
RandP ↓	84.28 ± 1.37%	70.56 ± 2.23%	70.03 ± 2.38%
P-poison ↓	78.06 ± 1.73%	66.32 ± 1.36%	68.75 ± 1.40%
GRAD ↓	79.33 ± 1.68%	65.82 ± 1.67%	69.06 ± 1.57%
MeCo ↓	51.68 ± 1.96%	46.53 ± 2.09%	61.38 ± 2.41%
MAZE	undefended ↓	45.17 ± 0.73%	23.28 ± 1.67%	20.03 ± 1.79%
RandP ↓	28.76 ± 2.38%	22.03 ± 1.50%	18.79 ± 1.38%
P-poison ↓	26.81 ± 2.19%	20.89 ± 1.58%	17.08 ± 2.28%
GRAD ↓	26.06 ± 1.81%	21.18 ± 1.58%	18.09 ± 1.72%
MeCo ↓	21.89 ± 2.07%	18.75 ± 2.11%	17.95 ± 1.46%
Attack	Defense	CIFAR100 Clone Model Architecture
ResNet18-8X	MobileNetV2	DenseNet121
DFME	undefended ↓	58.72 ± 2.82%	28.36 ± 1.97%	27.28 ± 2.08%
RandP ↓	41.69 ± 2.91%	22.75 ± 2.19%	23.61 ± 2.70%
P-poison ↓	38.72 ± 3.06%	20.87 ± 2.61%	21.89 ± 2.93%
GRAD ↓	39.07 ± 2.72%	20.71 ± 2.80%	22.08 ± 2.78%
MeCo ↓	29.57 ± 1.97%	12.18 ± 1.05%	10.79 ± 1.36%
MAZE	-	-	-
to generate strong defense. That is,  $||\pmb{y} - \hat{\pmb{y}}||_1 \leq 1.0$ ; where  $\pmb{y}$  and  $\hat{\pmb{y}}$  are the unmodified/modified output probabilities, respectively. We put more baseline details in Appendix 10.
Implementation Details. The random perturbation generator is a two-block ResNet structure with a filter size of 64, which is a small network compared to the backbone (only accounts for a tiny proportion of the backbone). For decision-based DFME methods, following [47], we use a query budget of  $10M$  for CIFAR100 and  $8M$  for CIFAR-10. For score-based DFME methods, following [51], we set the number of queries to be  $2M$  for MNIST,  $20M$  for CIFAR10, and  $200M$  for CIFAR100, respectively. We perform each experiment for 5 runs with a mean and standard deviation of results. We provide more implementation details in Appendix 7.
5.2 Results of Defense against DFME
Clone model accuracy. For score-based DFME setting (soft label), we show the results on CIFAR10 and CIFAR100 in Table 1. For decision-based DFME setting (hard label), we show the results in
Table 2: Clone model accuracy after applying different defense methods on CIFAR-10 and CIFAR-100 with ResNet34-8x as the target model, which only provides hard label
Attack	Defense	CIFAR10 Clone Model Architecture	CIFAR100 Clone Model Architecture
ResNet18-8X	MobileNetV2	DenseNet121	ResNet18-8X	MobileNetV2	DenseNet121
DFMS-HL	undefended ↓	84.67 ± 1.90%	79.28 ± 1.87%	68.87 ± 2.08%	72.57 ± 1.28%	62.71 ± 1.68%	63.58 ± 1.79%
RandP ↓	84.02 ± 2.31%	78.71 ± 1.93%	68.16 ± 2.23%	72.43 ± 1.43%	62.06 ± 1.82%	63.16 ± 1.73%
P-poison ↓	84.06 ± 1.87%	79.02 ± 1.96%	68.05 ± 2.17%	71.83 ± 1.32%	61.83 ± 1.79%	62.73 ± 1.91%
GRAD ↓	84.28 ± 1.95%	78.83 ± 1.91%	68.11 ± 1.93%	71.89 ± 1.37%	62.60 ± 1.71%	62.57 ± 1.80%
MeCo ↓	76.86 ± 2.09%	71.22 ± 1.87%	62.33 ± 2.01%	59.30 ± 1.70%	55.32 ± 1.65%	56.80 ± 1.86%
Table 3: Target model utility (test accuracy and  $l_{1}$  norm of the output probabilities perturbation magnitude)
Method	MNIST	CIFAR10	CIFAR100
ACC↑	l1norm↓	ACC↑	l1norm↓	ACC↑	l1norm↓
undefended	98.91 ± 0.16%	0.0	94.91 ± 0.37%	0.0	76.71 ± 1.25%	0.0
RandP	98.52 ± 0.19%	1.0	93.98 ± 0.28%	1.0	75.23 ± 1.39%	1.0
P-poison	98.87 ± 0.35%	1.0	94.58 ± 0.61%	1.0	75.42 ± 1.21%	1.0
GRAD	98.73 ± 0.31%	1.0	94.65 ± 0.67%	1.0	75.60 ± 1.45%	1.0
MeCo	98.63 ± 0.28%	0.0126	94.17 ± 0.56%	0.099	75.36 ± 0.68%	0.312
Table 2. Due to the space limitations, we present the results on MiniImageNet and MNIST in Table 8 and 7 in Appendix.  $\downarrow$  indicates the accuracy the lower, the better;  $\uparrow$  indicates the accuracy the higher, the better. For CIFAR10/CIFAR100 in Table 1, we use ResNet34 [13] as the target model. The clone model architecture includes, ResNet-18, MobileNetV2 [46], DenseNet121 [15]. We further change the target model architecture as GoogLeNet [48] for CIFAR10 in Table 9 in Appendix.
The results show MeCo significantly reduces the effectiveness of existing DFME methods by up to  $35\%$  and is substantially more effective than the compared methods since (1) RandP lacks data-dependent information; it maintains the utility for almost all the query data, which is unnecessary for attack queries since the attacker can still learn useful information. (2) P-poison needs a random initialized surrogate attacker model, which acts as an adversary model. (3) GRAD needs to know the attack query set to train the surrogate model. Those surrogates have large gaps compared to the DFME attacker model. Those methods thus perform poorly since the attack query data distribution and model are unknown to the defender. While MeCo does not need a surrogate model.
Target model utility. We evaluate the target model utility by (1) target model test accuracy after adopting the defense strategy; (2)  $l_{1}$  norm between the target model output probabilities with and without input perturbation averaged on the test dataset, i.e.,  $\mathbb{E}_{\boldsymbol{x} \sim \mathcal{D}_{test}}||T(\boldsymbol{x} + h_{\boldsymbol{\omega}}(\boldsymbol{x}, \boldsymbol{\epsilon})) - T(\boldsymbol{x})||_{1}$ . The results (see Table 3 (MNIST with LeNet, CIFAR10 and CIFAR100 with ResNet34-8x)) indicate that MeCo maintains the target model utility with a slight trade-off of benign accuracy but with much better preservation of output probabilities in terms of  $l_{1}$  norm. The baselines have large perturbations because the baseline defense methods perturb all the query data with the same magnitude ( $l_{1}$  perturbation budget of 1.0); this is unnecessary since the in-distribution benign query does not need such significant perturbation. In contrast, MeCo explicitly optimizes the  $l_{1}$  norm of perturbation with DRO on the simulated test data distribution so that the perturbation magnitude is much smaller.
5.3 Application on Data-Based ME (DBME) Attack
We apply MeCo to defend against traditional DBME methods with Knockoff Nets [37] and Jacobian-Based Dataset Augmentation (JBDA) [41]. We present the results in Appendix 9.1. MeCo can significantly outperform existing methods and further reduce the clone model accuracy. When the distribution of attack query data is closer to the training data distribution of the target model, the effectiveness of MeCo defense weakens. This is due to the fact that MeCo applies smaller perturbations to the data distribution similar to the training data of the target model as a result of defensive training.
5.4 Adaptive Attacks
We further analyze the robustness of MeCo against the attacker's countermeasures. Namely, we consider the situations where attackers know about our defense and have tailored the attacks to our defense method by adding data-dependent random perturbation to the query inputs to learn an additional random perturbation generator. In Table 4, MeCo is still effective since the random perturbation added by the defender and attacker are different due to randomness. There is still a mismatch in the KD loss and zeroth-order gradient estimation for the attacker. Interestingly, the performance of the clone model becomes even worse after the attacker performs an adaptive attack.
We believe this is because after the attacker adds the random perturbation, they will need to learn a random function, increasing the difficulty of model extraction. In addition, for score-based DFME, following [38, 33], we also compare to the adaptive attack method where attackers use only hard label, not probability outputs. We show the results in Table 5. MeCo is still very effective since the pseudo samples are nearer to the decision boundary, and perturbing the input would easily change the model outputs on those samples. Consequently, the attacker still cannot learn useful information.
Effect of DRO. We evaluate how much improvement DRO can bring to the model utility preservation in Table 12 in Appendix. We can observe that with DRO, our method significantly improves the model utility by  $5.3\% - 5.6\%$  compared to the one without DRO training.
Effect of different query budgets for attacker. To evaluate the effect of different query budgets on defense performance, we evaluate the clone model accuracy with varying defense methods on CIFAR10 in Figure 5 in Appendix. MeCo substantially outperforms various baselines with varying query budgets.
Table 4: Clone model accuracy after applying adaptive attack on CIFAR10 with ResNet34-8x as the target model
Attack	Defense	Clone Model Architecture
ResNet18-8X	MobileNetV2	DenseNet121
DFME	undetected ↓	87.36 ± 0.78%	75.23 ± 1.53%	73.89 ± 1.29%
MeCo ↓	51.68 ± 1.96%	46.53 ± 2.09%	61.38 ± 2.41%
MeCo, adaptive ↓	25.79 ± 0.81%	20.18 ± 1.17%	22.32 ± 1.82%
MAZE	undetected ↓	45.17 ± 0.73%	23.28 ± 1.67%	20.03 ± 1.79%
MeCo ↓	21.89 ± 2.07%	18.75 ± 2.11%	16.31 ± 1.76%
MeCo adaptive ↓	19.82 ± 2.03%	18.27 ± 2.23%	17.08 ± 2.28%
Table 5: Clone model accuracy that attacker only uses hard label instead of output probabilities on CIFAR10
Attack	Defense	Clone Model Architecture
ResNet18-8X	MobileNetV2	DenseNet121
DFME	undefended ↓	87.36 ± 0.78%	75.23 ± 1.53%	73.89 ± 1.29%
MeCo, soft label ↓	51.68 ± 1.96%	46.53 ± 2.09%	61.38 ± 2.41%
MeCo, hard label ↓	41.23 ± 0.58%	38.67 ± 0.97%	42.31 ± 1.78%
Hyperparameter Sensitivity. We evaluate the hyperparameter sensitivity for  $\gamma$  and  $Q$  in Table 10 and Table 11 in Appendix. We observe that model utility increases as  $\gamma$  increases with a trade-off of a decrease in defense performance. Results show that  $Q = 2$  performs the best. With the increase of  $Q$ , the simulated test data may be harder to learn; thus, the benign accuracy slightly decreases.
Test time speed and memory comparisons. We compare the running time of best-performing methods in Table 13 in Appendix. MeCo achieves  $17 \sim 172 \times$  speed up on CIFAR10 and CIFAR100. This is because P-poison and GRAD solve computationally expensive optimization during testing. In contrast, MeCo does not need this optimization. We provide the results in Table 14 in Appendix for memory consumption evaluation. MeCo is very memory efficient compared to baselines.
Training efficiency. We compare our training efficiency to baselines in Table 15 in Appendix. MeCo increases the training cost by  $1.3 \times$ . However, MeCo substantially improves the test time running efficiency. We believe this slightly additional computation cost is worth it.
In this paper, we explore the defense against DFME. We propose a memory and computation efficient (MeCo) defense method through distributionally robust defensive training by adding a data-dependent random perturbation generator to perturb the input data so that the attacker cannot steal useful information from the black-box model. At the same time, MeCo maintains the target model utility. Extensive experiments demonstrate MeCo's effectiveness, computation, and memory efficiency.
Limitations and Broader Impacts. Our proposed defensive training helps build safe and trustworthy AI systems. This would be beneficial for protecting current large-scale pre-trained models used in the public APIs since they spend lots of money and time to tune and deploy those valuable large-scale models. However, sharing models and insights is a key driver of progress in the AI research field, and if the community becomes more protective of their models, it could slow down advancements.
Acknowledgement This research was partly supported by NSF through grants IIS-1910492 and IIS-2239537. Tongliang Liu was partially supported by the following Australian Research Council projects: FT220100318, DP220102121, LP220100527, LP220200949, and IC190100031. We also gratefully acknowledge the support of NVIDIA Corporation with the donation of the A5000 used for this research.