SlowGuess/NeurIPS_2023_OCR huggingface dataset:
mode that results  $k(k - 1) / 2$  binary problems by enumerating over every pair of classes or in the "one vs rest" mode that results in  $k$  binary problems by picking one class as the positive class and the rest as the negative class. In practice, the "one vs rest" mode is preferred because it requires training smaller number of classifiers. In addition, the last classification layer of neural networks may also be roughly viewed as performing multi-class classification in "one vs rest" mode. Therefore, we only discuss and experiment with multi-class linear models trained in "one vs rest" mode in this section, consistent with the models in prior poisoning attacks [25, 32], but classifiers trained in "one vs one" mode can also be handled similarly.
Computation of the metrics. Although we consider linear models in "one vs rest" mode, when computing the metrics, we handle it in a way similar to the "one vs one" mode - when computing the metrics, given a positive class, we do not treat all the remaining k-1 classes (constitute the negative class) as a whole, instead, for each class in the remaining classes, we treat it as a "fake" negative class and compute the metrics as in the binary classification setting. Then from the  $k - 1$  metrics computed, we pick the positive and "fake" negative pair with smallest separability metric and use it as the metric for the current positive and negative class (includes all remaining  $k - 1$  classes) $^2$ . Once we compute the metrics for all the  $k$  binary pairs, we report the lowest metrics obtained. The reasoning behind computing the metrics in (similar to) "one vs one" mode is, for a given positive class, adversaries may target the most vulnerable pair from the total  $k - 1$  (positive, negative) pairs to cause more damage using the poisoning budget. Therefore, treating the remaining  $k - 1$  pairs as a whole when computing the metrics will obfuscate this observation and may not fully reflect the poisoning vulnerabilities of a dataset.
We provide a concrete example on how treating the remaining classes as a whole can lead to wrong estimates on the dataset separability: we first train simple CNN models on the full CIFAR-10 [26] and MNIST datasets and achieve models with test accuracies of  $\approx 70\%$  and  $>99\%$  respectively. When we feed the MNIST and CIFAR-10 test data through the model and inspect the feature representations, the t-SNE graph indicate that the CIFAR-10 dataset is far less separable than the MNIST, which is expected as CIFAR-10 has much lower test accuracy compared to MNIST. However, when we compute the separability metrics in our paper by considering all  $k - 1$  classes in the negative class, the separability of CIFAR-10 is similar to the separability of MNIST, which is inconsistent with drastic differences in the test accuracies of the respective CNN models. In contrast, if we treat each class in the remaining  $k - 1$  classes separately and pick the smallest value, we will again see the expected result that CIFAR-10 is far less separable than MNIST. Therefore, for the following experiments, we will compute the metrics by treating the remaining  $k - 1$  classes individually. We first provide the results of multi-class linear models for MNIST dataset below and then discuss our initial findings on the neural networks for CIFAR-10 and MNIST in Appendix F.2.
Results on multi-class linear learners. As explained above, when checking the  $k$ -binary pairs for a  $k$ -class problem, we report the lowest values for the Sep/SD and Sep/Size metrics. However, in some cases, the lowest values for the two metrics might be in two different pairs and in this case, we will report the results of both pairs. Table 12 summarizes the results, where the poisoned errors are directly quoted from the prior works—LR error is from Lu et al. [31] and the SVM error is from Koh et al. [25]. We can see that MNIST dataset is indeed more vulnerable than the selected MNIST 1-7 and MNIST 6-9 pairs because it is less separable and also impacted more by the poisoning points. We also note that the poisoned error of SVM is obtained in the presence of data sanitization defenses
2We still use the same projected constraint size for the  $k - 1$  positive and "fake" negative pairs because, the projected constraint size measures how much the decision boundary can be moved in presence of clean data points, which do not distinguish the points in the "fake" negative class and the remaining  $k - 1$  classes.
Datasets	Base Error (%)	Poisoned Error (%)	Increased Error (%)	Sep/SD	Sep/Size
MNIST	0.8%	1.9%	1.1%	4.58	0.10
CIFAR-10	31.0%	35.3%	4.3%	0.24	0.01
Table 13: Results on Simple CNN Models for MNIST and CIFAR-10 datasets using  $3\%$  poisoning ratio. The "Poisoned Error" of both datasets are directly quoted from Lu et al. [31]
Datasets	Base Error (%)	Increased Error (%)	Sep/SD	Sep/Size
MNIST	0.8%	9.6%	4.58	0.10
CIFAR-10	4.8%	13.7%	6.36	0.24
Table 14: Results on Simple CNN Model for MNIST and ResNet18 model for CIFAR-10 datasets using  $3\%$  poisoning ratio. The "Poisoned Error" of both datasets are directly quoted from Lu et al. [32].
and hence, the poisoned error may be further increased when there are no additional defenses. We also see that, for SVM, although the lowest values for Sep/SD and Sep/Size are in two different pairs, their results do not differ much, indicating that either of them can be used to represent the overall vulnerability of MNIST.
F.2 Extension to Multi-class Neural Networks
We first note that the insights regarding the separability and constraints set  $\mathcal{C}$  can be general, as the first metric measures the sensitivity of the dataset against misclassification when the decision boundary is perturbed slightly. The latter captures how much the decision boundary can be moved by the poisoning points once injected into the clean training data. The Sep/SD and Sep/Size metrics used in this paper are the concrete substantiations of the two metrics under linear models. Specific ways to compute the metrics in non-linear settings should still (approximately) reflect the high level idea above. Below, we use neural network (NN) as an example.
High level idea. We may partition the neural network into two parts of feature extractor and linear classification module and we may view the feature representations of the input data as a "new" data in the corresponding feature space, and so that we can convert the metric computations for non-linear neural network into metric computations (on feature space) for linear models. To be more concrete, we propose to use a fixed feature extractor, which can be extractors inherited from pretrained models (e.g., in transfer learning setting) or trained from scratch on the clean data, to map the input data to the feature space. Here, if the victim also uses the same pretrained feature extractor (as in the transfer learning setting), then our metrics can have higher correlation with the poisoned errors from existing attacks because the non-linear feature extractor is now independent from the poisoned points used in the victim's training. Below, we consider the from-scratch training case as it is more challenging.
Computation of the metrics. Although the feature extractor will also be impacted by the poisoning points now, in our preliminary experiment, we will still use the extractor trained on the clean data and leave the exploration of other better feature extractors as future work. Using the transformation from the clean feature extractor, the projected separability and standard deviation can be easily computed. But the computation of the projected constraint size can be tricky, because the set  $\mathcal{C}$  after transforming through the feature extractor can be non-convex and sometimes, for complicated  $\mathcal{C}$ , computing such transformation can be very challenging (can be a case even for linear models), but is a very interesting direction to explore. For the simple forms of  $\mathcal{C}$  such as the dimension-wise box constraints considered in this paper, we may leverage the convex outer polytope method [53] to bound the activation in each layer till the final feature layer so that we can obtain a final transformed convex set  $\mathcal{C}'$  using the feature extractor, which is a set that contains the original  $\mathcal{C}$ . However, due to time limitation, when computing the projected constraint size in the following experiments, we simply set  $\mathcal{C}$  as the dimension-wise box-constraints, whose minimum and maximum values are computed from the feature representations of the clean data points, similar to the transfer-learning experiment in Section 7.
Figure 6: Measuring the performance of optimal poisoning attacks with different values of separability ratio  $|\gamma_1 - \gamma_2| / \sigma$ . Best Heuristic denotes the best performing attack in the literature.
Results under similar learners. For the experiments, we use the simple CNN models presented in Lu et al. [31] for MNIST and CIFAR-10 datasets (similar architecture). We directly quote the attack results of TGDA attack by Lu et al. [31] for both the CIFAR-10 and MNIST datasets. Note that, very recently, a stronger GC attack is also proposed by Lu et al. [32] and outperforms the TGDA attack. However, we could not include the newer result because the code is not published and the evaluation in the original paper also did not include the simple CNN for CIFAR-10 dataset. The results are shown in Table 13. From the table, we can see that CIFAR-10 tends to be more vulnerable than MNIST as the Sep/SD and Sep/Size metrics (in neural networks) are all much lower than those of MNIST. These significantly lower values of CIFAR-10 may also suggest that the poisoned error for CIFAR-10 with simple CNN maybe increased further (e.g., using the stronger attack in Lu et al. [32]).
Results under different learners. Above, we only showed results when the MNIST and CIFAR-10 datasets are compared under similar learners. However, in practical applications, one might use deeper architecture for CIFAR-10 and so, we computed the metrics for CIFAR-10 using ResNet18 model. Then we compare the metrics of MNIST under simple CNN and the metrics of CIFAR-10 under ResNet18 in Table 14, where the poisoned errors are quoted from the more recent GC attack [32] because the attack results are all available in the original paper. However, from the table we can see that, although MNIST is less separable and impacted more by the poisoning points, the error increase is still slightly smaller than CIFAR-10, which is not consistent with our metrics. If the current attacks are already performing well and the empirical poisoned errors are indeed good approximations to the inherent vulnerabilities, then we might have to systematically investigate the comparison of vulnerabilities of different datasets under different learners.
G Evaluation on Synthetic Datasets
In this section, we empirically test our theory on synthetic datasets that are sampled from the considered theoretical distributions in Section 5.1.
G.1 Synthetic Datasets
According to Remark 5.4, there are two important factors to be considered: (1) the ratio between class separability and within-class variance  $|\gamma_1 - \gamma_2| / \sigma$ , denoted by  $\beta$  for simplicity; (2) the size of constraint set  $u$ . We conduct synthetic experiments in this section to study the effect of these factors on the performance of (optimal) data poisoning attacks.
Figure 7: Impact of the (projected) constraint size  $u$  on poisoning effectiveness.  $u = 0$  means the test error without poisoning.
- |γ1- γ2|/σ: 10.00	— |γ1- γ2|/σ: 2.50	— |γ1- γ2|/σ: 1.25
- |γ1- γ2|/σ: 5.00	— |γ1- γ2|/σ: 1.67	— |γ1- γ2|/σ: 1.00
More specifically, we generate 10,000 training and 10,000 testing data points according to the Gaussian mixture model (2) with negative center  $\gamma_{1} = -10$  and positive center  $\gamma_{2} = 0$ . Throughout our experiments,  $\gamma_{1}$  and  $\gamma_{2}$  are kept fixed, whereas we vary the variance parameter  $\sigma$  and the value of  $u$ . The default value of  $u$  is set as 20 if not specified. Evaluations of empirical poisoning attacks require training linear SVM models, where we choose  $\lambda = 0.01$ . The poisoning ratio is still set as  $3\%$ , consistent with evaluations on the benchmark datasets.
Impact of  $\beta$ . First, we show how the optimal attack performance changes as we increase the value of  $\beta$ . We report the risk achieved by the OPT attack based on Theorem 5.3. Note that we can only obtain approximations of the inverse function  $g^{-1}$  using numerical methods, which may induce a small approximation error for evaluating the optimal attack performance. For the finite-sample setting, we also report the empirical test error of the poisoned models induced by the empirical OPT attack and the best current poisoning attack discussed in Section 3, where the latter is termed as Best Heuristic for simplicity. Since the poisoned models induced by these empirical attacks do not restrict  $w \in \{-1, 1\}$ , we normalize  $w$  to make the empirical results comparable with our theoretical results.
Figure 6 summarizes the attack performance when we vary  $\beta$ . As the ratio between class separability and within-class variance increases, the risk of the OPT attack and empirical test errors of empirical OPT and best heuristic attacks gradually decrease. This is consistent with our theoretical results discussed in Section 5.1. Note that there exists a minor difference between these attacks when the value of  $\beta$  is small, where the test error attained by the best current heuristic poisoning attack is slightly higher than that achieved by the empirical OPT attack. This is due to the small numerical error induced by approximating the inverse function  $g^{-1}$ .
Impact of  $u$ . Our theoretical results assume the setting where  $w \in \{-1, 1\}$ . However, this restriction makes the impact of the constraint set size  $u$  less significant, as it is only helpful in judging whether flipping the sign of  $w$  is feasible and becomes irrelevant to the maximum risk after poisoning when flipping is infeasible. In contrast, if  $w$  is not restricted, the impact of  $u$  will be more significant as larger  $u$  tends to reduce the value of  $w$ , which in turn makes the original clean data even closer to each other and slight changes in the decision boundary can induce higher risks (further discussions on this are found in Appendix G.2).
To illustrate the impact of  $u$  in a continuous way, we allow  $w$  to take real numbers. Since this relaxation violates the assumption of our theory, the maximum risk after poisoning can no longer be characterized based on Theorem 5.3. Instead, we use the poisoning attack inspired by our theory to get an empirical lower bound on the maximum risk. Since  $\gamma_{1} + \gamma_{2} < 0$ , Theorem 5.3 suggests that optimal poisoning should place all poisoning points on  $u$  with label  $-1$  when  $w \in \{1, -1\}$ . We simply use this approach even when  $w$  can now take arbitrary values. We vary the value of  $u$  gradually and record the test error of the induced hypothesis. We repeat this procedure for different dataset configurations (i.e., fixing  $\gamma_{1}, \gamma_{2}$  and varying  $\sigma$ ).
Figure 8: Impact of the (projected) constraint size  $u$  on the value of  $w$  after poisoning in 1-D Gaussian distribution.
The results are summarized in Figure 7. There are two key observations: (1) once  $w$  is no longer constrained, if  $u$  is large enough, the vulnerability of all the considered distributions gradually increases as we increase the value of  $u$ , and (2) datasets with smaller  $\beta$  are more vulnerable with the increased value of  $u$  compared to ones with larger  $\beta$ , which has larger increased test error under the same class separability and box constraint (choosing other values of  $\beta$  also reveals a similar trend). Although not backed by our theory, it makes sense as smaller  $\beta$  also means more points might be closer to the boundary (small margin) and hence small changes in the decision boundary can have significantly increased test errors.
G.2 Relationship Between Box Constraint Size and Model Weight
Our theory assumes that the weight vector of  $w$  can only take normalized value from  $\{-1, 1\}$  for one-dimensional case, while in practical machine learning applications, convex models are trained by optimizing the hinge loss with respect to both parameters  $w$  and  $b$ , which can result in  $w$  as a real number. And when  $w$  takes real numbers, the impact of  $u$  becomes smoother: when poisoning with larger  $u$ , the poisoning points generated can be very extreme and forces the poisoned model to have reduced  $w$  (compared to clean model  $w_c$ ) in the norm so as to minimize the large loss introduced by the extreme points. Figure 8 plots the relationship between  $u$  and  $w$  of poisoned model, and supports the statement above. When the norm of  $w$  becomes smaller, the original clean data that are well-separated also becomes less separable so that slight movement in the decision boundary can cause significantly increased test errors. This makes the existence of datasets that have large risk gap before and after poisoning more likely, which is demonstrated in Figure 7.