SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Moreover, we note that  $\delta^{*}$  represents the ratio of injected poisoned data that achieves the optimal attack performance. In general,  $\delta^{*}$  can be any value in  $[0,\epsilon ]$ , but we show in Theorem 4.5, proven in Appendix A.2, that optimal poisoning can always be achieved with  $\epsilon$ -poisoning under mild conditions.
Theorem 4.5. The optimal poisoning attack performance defined in Definition 4.2 can always be achieved by choosing  $\epsilon$  as the poisoning ratio, if either of the following conditions is satisfied:
1. The support of the clean distribution  $\mathrm{supp}(\mu_c)\subseteq \mathcal{C}$
2.  $\mathcal{H}$  is a convex hypothesis class, and for any  $h_\theta \in \mathcal{H}$ , there always exists a distribution  $\mu$  such that  $\mathrm{supp}(\mu) \subseteq \mathcal{C}$  and  $\frac{\partial}{\partial_\theta} L(h_\theta; \mu) = 0$ .
Remark 4.6. Theorem 4.5 characterizes the conditions under which the optimal performance is guaranteed to be achieved with the maximum poisoning ratio  $\epsilon$ . Note that the first condition  $\mathrm{supp}(\mu_c) \subseteq \mathcal{C}$  is mild because it typically holds for poisoning attacks against undefended classifiers. When attacking classifiers that employ some defenses such as data sanitization, the condition  $\mathrm{supp}(\mu_c) \subseteq \mathcal{C}$  might not hold, due to the fact that the proposed defense may falsely reject some clean data points as outliers (i.e., related to false positive rates). The second condition complements the first one in that it does not require the victim model to be undefended, however, it requires  $\mathcal{H}$  to be convex. We prove in Appendix A.3 that for linear hypothesis with hinge loss, such a  $\mu$  can be easily constructed. The theorem enables us to conveniently characterize the optimal poisoning attacks in Section 5.1 by directly using  $\epsilon$ . When the required conditions are satisfied, this theorem also provides a simple sanity check on whether a poisoning attack is optimal. In particular, if a candidate attack is optimal, the risk of the induced model is monotonically non-decreasing with respect to the poisoning ratio. Note that the theorem above applies to any strongly convex loss function.
5 Characterizing Optimal Poisoning Attacks
This section characterizes the distributional optimal poisoning attacks with respect to linear hypothesis class. We first consider a theoretical 1-dimensional Gaussian mixture model and exactly characterize optimal poisoning attack and then discuss the implications on the underlying factors that potentially cause the inherent vulnerabilities to poisoning attacks for general high-dimensional distributions.
5.1 One-Dimensional Gaussian Mixtures
Consider binary classification tasks using hinge loss with one-dimensional inputs, where  $\mathcal{X} = \mathbb{R}$  and  $\mathcal{Y} = \{-1, + 1\}$ . Let  $\mu_{c}$  be the underlying clean data distribution, where each example  $(x,y)$  is assumed to be i.i.d. sampled according to the following Gaussian mixture model:
$$
\left\{ \begin{array}{l l} y = - 1, x \sim \mathcal {N} \left(\gamma_ {1}, \sigma_ {1} ^ {2}\right) & \text {w i t h p r o b a b i l i t y} p, \\ y = + 1, x \sim \mathcal {N} \left(\gamma_ {2}, \sigma_ {2} ^ {2}\right) & \text {w i t h p r o b a b i l i t y} 1 - p, \end{array} \right. \tag {2}
$$
where  $\sigma_1, \sigma_2 > 0$  and  $p \in (0,1)$ . Without loss of generality, we assume  $\gamma_1 \leq \gamma_2$ . Following our threat model, we let  $\epsilon \geq 0$  be the maximum poisoning ratio and  $\mathcal{C} := [-u, u] \times \mathcal{V}$  for some  $u > 0$  be the constraint set. Let  $\mathcal{H}_{\mathrm{L}} = \{h_{w,b} : w \in \{-1,1\}, b \in \mathbb{R}\}$  be the linear hypothesis class with normalized weights. Note that we consider a simplified setting where the weight parameter  $w \in \{-1,1\}$ . Since  $\|w\|_2$  is fixed, we also set  $\lambda = 0$  in the hinge loss function (1).
For clarify in presentation, we outline the connection between the definitions and the main theorem below, which will also show the high-level proof sketch of Theorem 5.3. We first prove that in order to understand the optimal poisoning attacks, it is sufficient to study the family of two-point distributions (Definition 5.1), instead of any possible distributions, as the poisoned data distribution. Based on this reduction and a specification of weight flipping condition (Definition 5.2), we then rigorously characterize the optimal attack performance with respect to different configurations of task-related parameters (Theorem 5.3). Similar conclusions may also be made with logistic loss.
Definition 5.1 (Two-point Distribution). For any  $\alpha \in [0,1]$ ,  $\nu_{\alpha}$  is defined as a two-point distribution, if for any  $(x,y)$  sampled from  $\nu_{\alpha}$ ,
$$
(x, y) = \left\{ \begin{array}{l l} (- u, + 1) & \text {w i t h p r o b a b i l i t y} \alpha , \\ (u, - 1) & \text {w i t h p r o b a b i l i t y} 1 - \alpha . \end{array} \right. \tag {3}
$$
1 Characterizing the optimal poisoning attack under the general setting of  $w\in \mathbb{R}$  is more challenging, since we need to consider the effect of any possible choice of  $w$  and its interplay with the dataset and constraint set factors. We leave the theoretical analyses of  $w\in \mathbb{R}$  to future work.
Definition 5.2 (Weight-Flipping Condition). Consider the assumed Gaussian mixture model (2) and the linear hypothesis class  $\mathcal{H}_L$ . Let  $g$  be an auxiliary function such that for any  $b \in \mathbb{R}$ ,
$$
g (b) = \frac {1}{2} \Phi \left(\frac {b + \gamma_ {1} + 1}{\sigma}\right) - \frac {1}{2} \Phi \left(\frac {- b - \gamma_ {2} + 1}{\sigma}\right),
$$
where  $\Phi$  is the cumulative distribution function (CDF) of standard Gaussian  $\mathcal{N}(0,1)$ . Let  $\epsilon > 0$  be the poisoning budget and  $g^{-1}$  be the inverse of  $g$ , then the weight-flipping condition is defined as:
$$
\max  \left\{\Delta (- \epsilon), \Delta (g (0)), \Delta (\epsilon) \right\} \geq 0, \tag {4}
$$
where  $\Delta (s) = L(h_{1,g^{-1}(s)};\mu_c) - \min_{b\in \mathbb{R}}L(h_{-1,b};\mu_c) + \epsilon \cdot (1 + u) - s\cdot g^{-1}(s)$ .
Now we are ready to present our main theoretical results. The following theorem rigorously characterizes the behavior of the distributional optimal poisoning adversary  $\mathcal{A}_{\mathrm{opt}}$  under the Gaussian mixture model (2) and the corresponding optimal attack performance:
Theorem 5.3. Suppose the clean distribution  $\mu_c$  follows the Gaussian mixture model (2) with  $p = 1/2$ ,  $\gamma_1 \leq \gamma_2$ , and  $\sigma_1 = \sigma_2 = \sigma$ . Assume  $u \geq 1$  and  $|\gamma_1 + \gamma_2| \leq 2(u - 1)$ . There always exists some  $\alpha \in [0,1]$  such that the optimal attack performance defined in Definition 4.2 is achieved with  $\delta = \epsilon$  and  $\mu_p = \nu_\alpha$ , where  $\nu_\alpha$  is defined by (3). More specifically, if  $h_p^* = \operatorname{argmin}_{h \in \mathcal{H}_L} \{L(h; \mu_c) + \epsilon \cdot L(h; \nu_\alpha)\}$  denotes the induced hypothesis with optimal poisoning, then
$$
\operatorname {R i s k} (h _ {p} ^ {*}; \mu_ {c}) = \left\{ \begin{array}{l l} \Phi \Big (\frac {\gamma_ {2} - \gamma_ {1}}{2 \sigma} \Big) & \text {i f c o n d i t i o n (4) i s s a t i s f i e d}, \\ \frac {1}{2} \Phi \Big (\frac {\gamma_ {1} - \gamma_ {2} + 2 s}{2 \sigma} \Big) + \frac {1}{2} \Phi \Big (\frac {\gamma_ {1} - \gamma_ {2} - 2 s}{2 \sigma} \Big) & \text {o t h e r w i s e}, \end{array} \right.
$$
where  $s = \max \{g^{-1}(\epsilon) - g^{-1}(0), g^{-1}(0) - g^{-1}(-\epsilon)\}$  and  $g(\cdot)$  is defined in Definition 5.2.
Remark 5.4. The proof of Theorem 5.3 is given in Appendix B.1. We considered the same variance of  $\sigma_{1} = \sigma_{2} = \sigma$  simply for the convenience in proof without involving minor but tedious calculations, and the same conclusion can be made using different variances. Theorem 5.3 characterizes the exact behavior of  $\mathcal{A}_{\mathrm{opt}}$  for typical combinations of hyperparameters under the considered model, including distribution related parameters such as  $\gamma_{1}, \gamma_{2}$ ,  $\sigma$  and poisoning related parameters such as  $\epsilon$ ,  $u$ . We use  $\epsilon$  directly because the poisoning setup in the 1-D Gaussian distribution satisfies the second condition in Theorem 4.5. Hence, the best performance can be achieved with the maximum poisoning ratio  $\epsilon$ . For  $u$ , a larger of it suggests the weight-flipping condition (4) is more likely to be satisfied, as an attacker can generate poisoned data with larger hinge loss to flip the weight parameter  $w$ . Class separability  $|\gamma_{1} - \gamma_{2}|$  and within-class variance  $\sigma$  also play an important role in affecting the optimal attack performance. If the ratio  $|\gamma_{1} - \gamma_{2}| / \sigma$  is large, then we know the initial risk  $\operatorname{Risk}(h_c; \mu_c) = \Phi\left(\frac{\gamma_1 - \gamma_2}{2\sigma}\right)$  will be small. Consider the case where condition (4) is satisfied. Note that  $\Phi\left(\frac{\gamma_2 - \gamma_1}{2\sigma}\right) = 1 - \Phi\left(\frac{\gamma_1 - \gamma_2}{2\sigma}\right)$  implies an improved performance of optimal poisoning attack thus an higher inherent vulnerabilities to data poisoning attacks. However, it is worth noting that there is an implicit assumption in condition (4) that the weight parameter can be flipped from  $w = 1$  to  $w = -1$ . A large value of  $|\gamma_{1} - \gamma_{2}| / \sigma$  also implies that flipping the weight parameter becomes more difficult, since the gap between the hinge loss with respect to  $\mu_c$  for a hypothesis with  $w = -1$  and that with  $w = 1$  becomes larger. Moreover, if condition (4) cannot be satisfied, then a larger ratio of  $|\gamma_{1} - \gamma_{2}| / \sigma$  suggests that it is more difficult to move the decision boundary to incur an increase in test error, because of the number of correctly classified boundary points will increase at a faster rate. In summary, Theorem 5.3 suggests that a smaller value of  $u$  and a larger ratio of  $|\gamma_{1} - \gamma_{2}| / \sigma$  increases the inherent robustness to indiscriminate poisoning for typical configurations under our model (2). Empirical verification of the above theoretical results is given in Appendix G.
5.2 General Distributions
Recall that we have identified several key factors (i.e.,  $u$ ,  $|\gamma_1 - \gamma_2|$  and  $\sigma$ ) for 1-D Gaussian distributions in Section 5.1 which are highly related to the performance of an optimal distributional poisoning adversary  $\mathcal{A}_{\mathrm{opt}}$ . In this section, we demonstrate how to generalize the definition of these factors to high-dimensional datasets and illustrate how they affect an inherent robustness upper bound to indiscriminate poisoning attacks for linear learners. In particular, we project the clean distribution  $\mu_c$  and the constraint set  $\mathcal{C}$  onto some vector  $\boldsymbol{w}$ , then compute those factors based on the projections.
Definition 5.5 (Projected Constraint Size). Let  $\mathcal{C} \subseteq \mathcal{X} \times \mathcal{Y}$  be the constraint set for poisoning. For any  $\boldsymbol{w} \in \mathbb{R}^n$ , the projected constraint size of  $\mathcal{C}$  with respect to  $\boldsymbol{w}$  is defined as:
$$
\operatorname {S i z e} _ {\boldsymbol {w}} (\mathcal {C}) = \max  _ {(\boldsymbol {x}, y) \in \mathcal {C}} \boldsymbol {w} ^ {\top} \boldsymbol {x} - \min  _ {(\boldsymbol {x}, y) \in \mathcal {C}} \boldsymbol {w} ^ {\top} \boldsymbol {x}
$$
According to Definition 5.5,  $\mathrm{Size}_{\pmb{w}}(\mathcal{C})$  characterizes the size of the constraint set  $\mathcal{C}$  when projected onto the (normalized) projection vector  $\pmb{w} / \| \pmb{w}\|_2$  then scaled by  $\| \pmb{w}\|_2$ , the  $\ell_2$ -norm of  $\pmb{w}$ . In theory, the constraint sets conditioned on  $y = -1$  and  $y = +1$  can be different, but for simplicity and practical considerations, we simply assume they are the same in the following discussions.
Definition 5.6 (Projected Separability and Standard Deviation). Let  $\mathcal{X} \subseteq \mathbb{R}^n$ ,  $\mathcal{Y} = \{-1, +1\}$ , and  $\mu_c$  be the underlying distribution. Let  $\mu_{-}$  and  $\mu_{+}$  be the input distributions with labels of  $-1$  and  $+1$  respectively. For any  $\boldsymbol{w} \in \mathbb{R}^n$ , the projected separability of  $\mu_c$  with respect to  $\boldsymbol{w}$  is defined as:
$$
\operatorname {S e p} _ {\boldsymbol {w}} \left(\mu_ {c}\right) = \left| \mathbb {E} _ {\boldsymbol {x} \sim \mu_ {-}} \left[ \boldsymbol {w} ^ {\top} \boldsymbol {x} \right] - \mathbb {E} _ {\boldsymbol {x} \sim \mu_ {+}} \left[ \boldsymbol {w} ^ {\top} \boldsymbol {x} \right] \right|.
$$
In addition, the projected standard deviation of  $\mu_c$  with respect to  $\pmb{w}$  is defined as:
$$
\mathrm {S D} _ {\boldsymbol {w}} \left(\mu_ {c}\right) = \sqrt {\operatorname {V a r} _ {\boldsymbol {w}} \left(\mu_ {c}\right)}, \operatorname {V a r} _ {\boldsymbol {w}} \left(\mu_ {c}\right) = p _ {-} \cdot \operatorname {V a r} _ {\boldsymbol {x} \sim \mu_ {-}} \left[ \boldsymbol {w} ^ {\top} \boldsymbol {x} \right] + p _ {+} \cdot \operatorname {V a r} _ {\boldsymbol {x} \sim \mu_ {+}} \left[ \boldsymbol {w} ^ {\top} \boldsymbol {x} \right],
$$
where  $p_{-} = \operatorname{Pr}_{(\boldsymbol{x},\boldsymbol{y})\sim \mu_c}[y = -1], p_{+} = \operatorname{Pr}_{(\boldsymbol{x},\boldsymbol{y})\sim \mu_c}[y = +1]$  denote the sampling probabilities.
For finite-sample settings, we simply replace the input distributions with their empirical counterparts to compute the sample statistics of  $\mathrm{Rep}_{\pmb{w}}(\mu_c)$  and  $\mathrm{SD}_{\pmb{w}}(\mu_c)$ . Note that the above definitions are specifically for linear models, but out of curiosity, we also provide initial thoughts on how to extend these metrics to neural networks (see Appendix F for preliminary results). Below, we provide justifications on how the three factors are related to the optimal poisoning attacks. Theorem 5.7 and the techniques used in its proof in Appendix B.2 are inspired by the design of Min-Max Attack [45].
Theorem 5.7. Consider input space  $\mathcal{X} \subseteq \mathbb{R}^n$ , label space  $\mathcal{Y}$ , clean distribution  $\mu_c$  and linear hypothesis class  $\mathcal{H}$ . For any  $h_{\boldsymbol{w},b} \in \mathcal{H}$ ,  $\boldsymbol{x} \in \mathcal{X}$  and  $y \in \mathcal{Y}$ , let  $\ell(h_{\boldsymbol{w},b}; \boldsymbol{x}, y) = \ell_{\mathrm{M}}(-y(\boldsymbol{w}^\top \boldsymbol{x} + b))$  be a margin-based loss adopted by the victim, where  $\ell_M$  is convex and non-decreasing. Let  $\mathcal{C} \subseteq \mathcal{X} \times \mathcal{Y}$  be the constraint set and  $\epsilon > 0$  be the poisoning budget. Suppose  $h_c = \operatorname*{argmin}_{h \in \mathcal{H}} L(h; \mu_c)$  has weight  $\boldsymbol{w}_c$  and  $h_p^*$  is the poisoned model induced by optimal adversary  $\mathcal{A}_{\mathrm{opt}}$ , then we have
$$
\operatorname {R i s k} \left(h _ {p} ^ {*}; \mu_ {c}\right) \leq \min  _ {h \in \mathcal {H}} \left[ L \left(h; \mu_ {c}\right) + \epsilon \cdot L \left(h; \mu_ {p} ^ {*}\right) \right] \leq L \left(h _ {c}; \mu_ {c}\right) + \epsilon \cdot \ell_ {\mathrm {M}} \left(\operatorname {S i z e} _ {\boldsymbol {w} _ {c}} (\mathcal {C})\right). \tag {5}
$$
Remark 5.8. Theorem 5.7 proves an upper bound on the inherent vulnerability to indiscriminate poisoning for linear learners, which can be regarded as a necessary condition for the optimal poisoning attack. A smaller upper bound likely suggests a higher inherent robustness to poisoning attacks. In particular, the right hand side of (5) consists of two terms: the clean population loss of  $h_c$  and a term related to the projected constraint size. Intuitively, the projected separability and standard deviation metrics highly affect the first term, since data distribution with a higher  $\mathrm{Sep}_{\boldsymbol{w}_c}(\mu_c)$  and a lower  $\mathrm{SD}_{\boldsymbol{w}_c}(\mu_c)$  implies a larger averaged margin with respect to  $h_c$ , which further suggests a smaller  $L(h_c; \mu_c)$ . The second term is determined by the poisoning budget  $\epsilon$  and the projected constraint size, or more precisely, a larger  $\epsilon$  and a larger  $\mathrm{Size}_{\boldsymbol{w}_c}(\mathcal{C})$  indicate a higher upper bound on  $\mathrm{Risk}(h_p^*; \mu_c)$ . In addition, we set  $h = h_c$  and the projection vector as  $\boldsymbol{w}_c$  for the last inequality of (5), because  $h_c$  achieves the smallest population surrogate loss with respect to the clean data distribution  $\mu_c$ . However, choosing  $h = h_c$  may not always produce a tighter upper bound on  $\mathrm{Risk}(h_p^*; \mu_c)$  since there is no guarantee that the projected constraint size  $\mathrm{Size}_{\boldsymbol{w}_c}(\mathcal{C})$  is small. An interesting future direction is to select a more appropriate projection vector that returns a tight, if not the tightest, upper bound on  $\mathrm{Risk}(h_p^*; \mu_c)$  for any clean distribution  $\mu_c$  (see Appendix D.2 for preliminary experiments). We also note that the results above apply to any (non-decreasing and strongly-convex) margin-based losses.
Recall from Theorem 4.3 and Remark 4.4 that the finite-sample optimal poisoning attack is a consistent estimator of the distributional one for linear learners. In this section, we demonstrate the theoretical insights gained from Section 5, despite proven only for the distributional optimal attacks, still appear to largely explain the empirical performance of best attacks across benchmark datasets.
Given a clean training data  $S_{c}$ , we empirically estimate the three distributional metrics defined in Section 5.2 on the clean test data with respect to the weight  $w_{c}$ . Since  $\| w_{c}\|_{2}$  may vary across different datasets while the predictions of linear models (i.e., the classification error) are invariant to the scaling of  $\| w_{c}\|_{2}$ , we use ratios to make their metrics comparable:  $\mathrm{Sep}_{\boldsymbol{w}_c}(\mu_c) / \mathrm{SD}_{\boldsymbol{w}_c}(\mu_c)$  (denoted as Sep/SD in Table 1) and  $\mathrm{Sep}_{\boldsymbol{w}_c}(\mu_c) / \mathrm{Size}_{\boldsymbol{w}_c}(\mathcal{C})$  (Sep/Size). According to our theoretical results, we expect datasets that are less vulnerable to poisoning have higher values for both metrics.
Metric	Robust	Moderately Vulnerable	Highly Vulnerable
MNIST 6-9	MNIST 1-7	Adult	Dogfish	MNIST 4-9	F. Enron	Enron
SVM	Error Increase	2.7	2.4	3.2	7.9	6.6	33.1	31.9
Base Error	0.3	1.2	21.5	0.8	4.3	0.2	2.9
Sep/SD	6.92	6.25	9.65	5.14	4.44	1.18	1.18
Sep/Size	0.24	0.23	0.33	0.05	0.14	0.01	0.01
LR	Error Increase	2.3	1.8	2.5	6.8	5.8	33.0	33.1
Base Error	0.6	2.2	20.1	1.7	5.1	0.3	2.5
Sep/SD	6.28	6.13	4.62	5.03	4.31	1.11	1.10
Sep/Size	0.27	0.27	0.27	0.09	0.16	0.01	0.01
Table 1: Explaining disparate poisoning vulnerability under linear models. The top row for each model gives the increase in error rate due to the poisoning, over the base error rate in the second row. The explanatory metrics are the scaled (projected) separability, standard deviation and constraint size.
Table 1 summarizes the results where Error Increase is produced by the best attacks from the state-of-the-art attacks mentioned in Section 3. The Sep/SD and Sep/Size metrics are highly correlated to the poisoning effectiveness (as measured by error increase). Datasets such as MNIST 1-7 and MNIST 6-9 are harder to poison than others, which correlates with their increased separability and reduced impact from the poisoning points. In contrast, datasets such as Enron and Filtered Enron are highly vulnerable and this is strongly correlated to their small separability and the large impact from the admissible poisoning points. The results of Filtered Enron (low base error, high increased error) and Adult (high base error, low increased error) demonstrate that poisoning vulnerability, measured by the error increase, cannot be trivially inferred from the initial base error.
When the base error is small, which is the case for all tested benchmark datasets except Adult, the empirical metrics are highly correlated to the error increase and also the final poisoned error. However, when the base error becomes high as it is for Adult, the empirical metrics are highly correlated to the final poisoned error, but not the error increase, if the metrics are computed on the entire (clean) test data. For the error increase, computing the metrics on the clean and correctly classified (by  $w_{c}$ ) test points is more informative as it (roughly) accounts for the (newly) induced misclassification from poisoning. Therefore, we report metrics based on correctly-classified test points in Table 1 and defer results of the whole test data to Appendix D.2. For datasets except Adult, both ways of computing the metrics produce similar results. The Adult dataset is very interesting in that it is robust to poisoning (i.e., small error increase) despite having a very high base error. As a preliminary experiment, we also extended our analysis to neural networks (Appendix F) and find the identified factors are correlated to the varied poisoning effectiveness when different datasets are compared under similar learners.
Our current results only show the correlation of the identified factors to the varied (empirical) poisoning effectiveness, not causality. However, we also speculate that these factors may cause different poisoning effectiveness because: 1) results in Section 5 characterize the relationship of these factors to the performance of optimal attacks exactly for 1-D Gaussian distributions and partially for the general distributions, and 2) preliminary results on MNIST 1-7 shows that gradual changes in one factor (with the other fixed) also causes gradual changes in the vulnerability to poisoning (Appendix D.2.6). We leave the detailed exploration on their causal relationship as future work.
7 Discussion on Future Implications
Our results imply future defenses by explaining why candidate defenses work and motivating defenses to improve separability and reduce projected constraint size. We present two ideas—using data filtering might reduce projected constraint size and using better features might improve separability.
Reduced projected constraint size. The popular data sanitization defense works by filtering out bad points. We speculate it works because the defense may be effectively limiting the projected constraint size of  $\mathcal{C}$ . To test this, we picked the combination of Sphere and Slab defenses considered in prior works [25, 45] to protect the vulnerable Enron dataset. We find that with the defense, the test error is increased from  $3.2\%$  to  $28.8\%$  while without the defense the error can be increased from  $2.9\%$  to  $34.8\%$ . Although limited in effectiveness, the reduced Error Increase with the defense is highly correlated to the significantly reduced projected constraint size  $\mathrm{Size}_{w_c}(\mathcal{C})$  (and hence a significantly higher Sep/Size)—the Sep/Size metric jumps from 0.01 without defense to 0.11 with defense while
Sep/SD remains almost the same at 1.18 with and without defense. Similar conclusions can also be drawn for MNIST 1-7 and Dogfish (detailed experimental results are in Appendix D.2.3).
Better feature representation. We consider a transfer learning scenario where the victim trains a linear model on a clean pretrained model. As a preliminary experiment, we train LeNet and ResNet18
models on the CIFAR10 dataset till convergence, but record the intermediate models of ResNet18 to produce models with different feature extractors (R-X denotes ResNet18 trained for  $X$  epochs). We then use the feature extraction layers of these models (including LeNet) as the pretrained models and obtain features of CIFAR10 images with labels "Truck" and "Ship", and train linear models on them.