SlowGuess/NeurIPS_2023_OCR huggingface dataset:
In this section, we first present the best attack results on different benchmark datasets on the LR model in Figure 3. Similar to the case of linear SVM as shown in Figure 1, the performance of best known attacks vary drastically across the benchmark datasets. Although we only report the performance of the best known attack from the candidate state-of-the-art poisoning attacks, the difference among these candidate attacks do not vary much in many settings. In Figure 4 we show the attack performance of different attacks on the selected benchmark datasets of MNIST 1-7, Dogfish, and Enron and the main observation is that different attacks perform mostly similarly for a given dataset (but their performance varies a lot across datasets). Also, from the attack results on Enron (Figure 4c), we can see that several of the attacks perform worse at higher poisoning ratios. Although there is a chance that the attack performance can be improved by careful hyperparameter tuning, it also suggests that these attacks are suboptimal. Optimal poisoning attacks should never get less effective as the poisoning ratio increases, according to Theorem 4.5.
D.2.6 Possible causal relationship of identified factors to poisoning vulnerability
In this section, we describe our preliminary experiment on showing the possible causal relationship from the identified two factors (i.e., Sep/SD and Sep/Size) to the empirically observed vulnerability against poisoning attacks. Given a dataset, we aim to modify it such that we can gradually change one factor of the dataset while (ideally) keeping the other one fixed, and then measure the changes in its empirical poisoning vulnerability. To cause changes to the factor of "Sep/SD", we need to modify the original data points in the dataset. This modification may make it hard to keep the factor of "Sep/Size" (or "Size") fixed, as the computation of "Size" depends on the clean boundary  $\boldsymbol{w}_c$ , which is again dependent on the (modified) clean distribution. Therefore, even if we use the same constraint set  $\mathcal{C}$  for different modified datasets, the "Size" or "Sep/Size" can be different. We conducted preliminary experiments on MNIST 1-7 and the details are given below.
Scale Factor c	Error Increase (%)	Sep/Size
1.0	2.4	0.23
2.0	3.6	0.12
3.0	4.9	0.08
5.0	7.8	0.05
7.0	9.8	0.03
10.0	15.6	0.02
Table 8: Impact of scale factor  $c$  on poisoning effectiveness for  $\mathcal{C}$  in the form of dimension-wise box-constraint as  $[0,c]$  ( $c > 1$ ) for the MNIST 1-7 dataset. Originally, the dimension-wise box constraint is in  $[0,1]$ . The Base Error is  $1.2\%$  and the Sep/SD factor is 6.25 and these two values will be the same for all settings of  $c$  because the support set of the clean distribution is the strict subset of  $\mathcal{C}$ . Higher value of  $c$  implies poisoning points can be more harmful.
Number of Repetitions x	Base Error (%)	Error Increase (%)	Sep/SD	Sep/Size
0	1.2	2.4	6.25	0.23
1	1.0	4.4	5.78	0.21
2	1.0	5.7	5.48	0.19
4	1.0	6.2	5.02	0.17
6	1.2	6.8	4.75	0.15
8	1.4	8.0	4.62	0.13
10	1.1	8.1	4.55	0.12
Table 9: Changing the distribution of the MNIST 1-7 dataset by replacing  $x \cdot 3\%$  of correctly classified points farthest from the clean boundary with  $3\%$  of correctly classified points (repeated  $x$  times) closest to the boundary. Higher value of  $x$  denotes that more near boundary points will be present in the modified training data.
Change Sep/Size., In order to measure the impact of Sep/Size without impacting Sep/SD, we can choose  $\mathcal{C}$  that covers even larger space so that the resulting constraint set will not filter out any clean points from the original training data, which then implies both the  $w_{c}$  and the corresponding metric of Sep/SD will not change. We performed a hypothetical experiment on MNIST 1-7 where we scale the original box-constraint from [0, 1] to  $[0, c], c >= 1$ , where  $c$  is the scale factor and  $c = 1$  denotes the original setting for MNIST 1-7 experiment. The experiments are hypothetical because, in practice, a box-constraint of  $[0, c]$  with  $c > 1$  will result in invalid images. We experimented with  $c = 2, 3, 5, 7, 10$  and the results are summarized in Table 8. From the table, we see that as  $c$  becomes larger, the Size increases and correspondingly the Sep/Size decreases. These changes are associated with a gradual increase in the "Error Increase" from poisoning.
Change Sep/SD. As mentioned above, in order to change Sep/SD, we need to modify the original data points, and this modification makes it hard for us to keep Sep/Size fixed for MNIST 1-7 even when using the same  $\mathcal{C}$ . We leave the exploration of better ways to change data distributions with the Sep/Size fixed as future work. To modify the original data points, we first train a linear SVM on the original MNIST 1-7 dataset. Then, we pick  $3\%$  of all (training + test data) correctly classified points that are closest to the clean boundary. These points, once repeated  $x$  times, replace  $x \times 3\%$  of correctly classified training and testing data points that are furthest from the boundary. This process can be viewed as oversampling in the near-boundary region and undersampling in the region far from the boundary, and aim to create a dataset with a low separability ratio. We tested with  $x = 1,2,4,6,8,10$ . We only deal with correctly classified points because we want to ensure the modified dataset still has high clean accuracy. The results are summarized in Table 9 and we can see that as we increase  $x$ , the Sep/SD gradually decreases, and the attack effectiveness (measured by the
Figure 5: Impact of feature representations on the poisoning effectiveness. The training data of the pretrained model excludes the downstream data of classes "Truck" and "Ship".
Experiments	Sep/SD vs Error Increase	Sep/Size vs Error Increase
New	-0.66 (-0.90)	-0.90 (-0.99)
Original	-0.98 (-0.97)	-0.98 (-0.97)
Table 10: Pearson correlation between Error Increase and each of Sep/SD and Sep/Size. The value in the parentheses denotes the Pearson correlation coefficient when we exclude the LeNet structure and only consider ResNet-18 models.
error increase) increases. Interestingly, the Sep/Size also decreases even though it is computed with the same  $\mathcal{C}$ . Although the preliminary results above are not sufficient to prove a causation claim, they still show the possibility with the empirical evidence on the MNIST 1-7 dataset.
D.2.7 Evaluating feature extractors with downstream data removed
The experiments in Figure 2 uses the entire CIFAR10 training data to train the pretrained feature extractor, which also includes the data of class "Truck" and "Ship" that are used in the downstream training and analysis. Although our original experiment aims to show how the victim can gain improved resistance to poisoning by using better feature extractors, for the particular experiment in Figure 2, a more realistic setting is to exclude the downstream training data used to train the (upstream) pretrained feature extractors. Therefore, we repeated the same experiment, but now removed the data points from the classes of "Ship" and "Truck" from the upstream training data and then train the (pretrained) models. The results are summarized in Figure 5 and the overall finding is similar to the one in Figure 2 that when we adopt a deep architecture and train it for more epochs, the quality of the obtained feature extractor increases and so, it appears to improve the resistance of downstream classifiers to poisoning attacks. The increased quality of the feature extractor is in general correlated to the increased Sep/SD and Sep/Size. To quantitatively measure the correlation of the error increase and each of the two factors, we show the Pearson correlation coefficient of the relevant metrics and the results are summarized in Table 10. From the table, we can see that when we directly measure the Pearson correlation with each individual factor, the correlation between the two factors and error increase is still very strong as the lowest (absolute) correlation coefficient of -0.66 still indicates a strong negative correlation. And if we further restrict ourselves to the ResNet-18 models only, then the (absolute) correlation coefficient is even higher, which again confirms the two factors are strongly (and negatively) correlated with the error increase.
In Table 10, we also compare the new experiment with the original experiment in Figure 2 and observe that the (absolute) Pearson correlation is reduced slightly, which is due to the fact that when the pretrained model does not involve any downstream data, the two factors may not consistently change in the same direction and undermines the Pearson correlation of each individual factor with
the error increase. However, if we compute the Pearson correlation between the number of epochs for training the ResNet-18 model (roughly incorporates the combined effects of both factors) and the error increase from poisoning, we observe an even stronger negative correlation (-0.99) compared to the original experiment (-0.95), which again supports the claim that training the deep architecture for more epochs seems to be one possible approach to improve resistance to poisoning. The slight increase in correlation in the new experiment results from the slightly higher covariance between the error increase and the number of epochs, but this difference is not significant since we only consider a limited number of samples when computing the correlation.
Developing general methods for learning meaningful feature extractors such that the two factors can be improved simultaneously in a controlled way would be an interesting research direction, which we plan to pursue in the future. However, we also note that this is a challenging task from the practical perspective since the goal is to extract useful feature representations for downstream applications, so the base error should also be considered in addition to the error increase from poisoning, especially when no downstream data are used in the upstream training. Our initial thought is to consider an intermediate few-shot learning setup [52], where a small number of (clean) downstream training data are allowed to be incorporated in training the feature extractor. This few-shot learning setup might correspond to an intermediate setting between the original experiment with full knowledge of the (clean) downstream training data and the new experiment with almost zero knowledge but still reflects certain real-world application scenarios. To conclude, all results in both the original and new experiments support our main claim that it is possible for the victim to choose better feature extractors to improve the resistance of downstream classifiers to indiscriminate poisoning attacks.
E Comparison to LDC and Aggregation Defenses
We first provide a more thorough discussion on the differences between our work and the Lethal Dose Conjecture (LDC) [50] from NeurIPS 2022, which had similar goals in understanding the inherent vulnerabilities of datasets but focused on targeted poisoning attacks (Appendix E.1). Then, we discuss how our results can also be related to aggregation based defenses whose asymptotic optimality on robustness against targeted poisoning attacks is implied by the LDC conjecture (Appendix E.2).
E.1 Relation to LDC
As discussed in Section 1, LDC is a more general result and covers broader poisoning attack goals (including indiscriminate poisoning) and is agnostic to the learning algorithm, dataset and also the poisoning generation setup. However, this general result may give overly pessimistic estimates on the power of optimal injection-only poisoning attacks in the indiscriminate setting we consider. We first briefly mention the main conjecture in LDC and then explain why the LDC conjecture overestimated the power of indiscriminate poisoning attacks, followed by a discussion on the relations of the identified vulnerability factors in this paper and the key quantity in LDC.
The main conjecture in LDC. LDC conjectures that, given a (potentially poisoned) dataset of size  $N$ , the tolerable sample size for targeted poisoning attacks (through insertion and/or deletion) by any defenses and learners, while still predicting a known test sample correctly, is an asymptotic guarantee of  $\Theta(N / n)$ , where  $n < N$  is the sample complexity of the most data-efficient learner (i.e., a learner that uses smallest number of clean training samples to make correct prediction). Although it is a conjecture on the asymptotic robustness guarantee, it is rigorously proven for cases of bijection uncovering and instance memorization, and the general implication of LDC is leveraged to improve existing aggregation based certified defenses against targeted poisoning attacks.
Overestimating the power of indiscriminate poisoning. LDC conjectures the robustness against targeted poisoning attacks, but the same conjecture can also be used in indiscriminate setting straightforwardly by taking the expectation over the tolerable samples for each of the test samples to get the expected tolerable poisoning size for the entire distribution (as mentioned in the original LDC paper) or by choosing the lowest value to give a worst case certified robustness for the entire distribution. The underlying assumption in the reasoning above is that individual samples are independently impacted by their corresponding poisoning points while in the indiscriminate setting, the entire distribution is impacted simultaneously by the same poisoning set. The assumption on the independence of the poisoning sets corresponding to different test samples might overestimate the power of indiscriminate
poisoning attacks as it might be impossible to simultaneously impact different test samples (e.g., test samples with disjoint poisoning sets) using the same poisoning set. In addition, the poisoning generation setup also greatly impacts the attack effectiveness—injection only attacks can be much weaker than attacks that modify existing points, but LDC provides guarantees against this worst case poisoning generation of modifying points. These general and worst-case assumptions mean that LDC might overestimate the power of injection-only indiscriminate poisoning attacks considered in this paper.
In practice, insights from LDC can be used to enhance existing aggregation based defenses. If we treat the fraction of (independently) certifiable test samples by the enhanced DPA [29] in Figure 2(d) (using  $k = 500$  partitions) in the LDC paper as the certified accuracy (CA) for the entire test set in the indiscriminate setting, the CA against indiscriminate poisoning attack is  $0\%$  at the poisoning ratio of  $\epsilon = 0.5\%$  (250/50000). In contrast, the best indiscriminate poisoning attacks [32] on CIFAR10 dataset reduces the model accuracy from  $95\%$  to  $81\%$  at the much higher  $\epsilon = 3\%$  poisoning ratio using standard training (i.e., using  $k = 1$  partition). Note that using  $k = 1$  partition is a far less optimal choice than  $k = 500$  as  $k = 1$  will always result in  $0\%$  CA for aggregation based defenses. Our work explicitly considers injection only indiscriminate poisoning attacks so as to better understand its effectiveness.
While it is possible that current indiscriminate attacks for neural networks are far from optimal and there may exist a very strong (but currently unknown) poisoning attack that can reduce the neural network accuracy on CIFAR10 to  $0\%$  at a  $0.5\%$  poisoning ratio, we speculate such likelihood might be low. This is because, neural networks are found to be harder to poison than linear models [31, 32] while our empirical findings in the most extensively studied linear models in Section 3 indicate some datasets might be inherently more robust to poisoning attacks.
Providing finer analysis on the vulnerability factors. As mentioned above, LDC might overestimate the power of indiscriminate poisoning attacks. In addition, the key quantity  $n$  is usually unknown and hard to estimate accurately in practice and the robustness guarantee is asymptotic while the constants in asymptotic guarantees can make a big difference in practice. However, the generic metric  $n$  still offers critical insights in understanding the robustness against indiscriminate poisoning attacks. In particular, our findings on projected separability and standard deviation can be interpreted as the first step towards understanding the dataset properties that can be related to the (more general) metric  $n$  (and maybe also the constant in  $\Theta(1/n)$ ) in LDC for linear learners. Indeed, it is an interesting future work to identify the learning task properties that impact  $n$  at the finer-granularity.
As for the projected constraint size (Definition 5.5), we believe there can be situations where it may be independent from  $n$ . The main idea is that in cases where changing  $\mathcal{C}$  arbitrarily will not impact the clean distribution (e.g., when the support set of the clean distribution is a strict subset of  $\mathcal{C}$ , arbitrarily enlarging  $\mathcal{C}$  will still not impact the clean distribution), the outcomes of learners trained on clean samples from the distribution will not change (including the most data-efficient learner) and hence  $n$  will remain the same for different permissible choices of  $\mathcal{C}$ , indicating that the vulnerability of the same dataset remains the same even when  $\mathcal{C}$  changes drastically without impacting the clean distribution. However, changes in  $\mathcal{C}$  (and subsequently changes in the projected constraint size) will directly impact the attack effectiveness, as a larger  $\mathcal{C}$  is likely to admit stronger poisoning attacks.
To illustrate how much the attack power can change as  $\mathcal{C}$  changes, we conduct experiments on MNIST 1-7 and show that scaling up the original dimension-wise box-constraint from  $[0,1]$  to  $[0,c]$  (where  $c > 1$  is the scale factor) can significantly boost attack effectiveness. Table 11 summarizes the results and we can observe that, as the scale factor  $c$  increases (enlarged  $\mathcal{C}$ , increased projected constraint size and reduced Sep/Size), the attack effectiveness also increases significantly. Note that this experiment is an existence proof and MNIST 1-7 is used as a hypothetical example. In practice, for normalized images, the box constraint cannot be scaled beyond  $[0,1]$  as it will result in invalid images.
E.2 Relation to Aggregation-based Defenses
Aggregation-based (provable) defenses, whose asymptotic optimality is implied by the LDC, work by partitioning the potentially poisoned data into  $k$  partitions, training a base learner on each partition and using majority voting to obtain the final predictions. These defenses provide certified robustness to poisoning attacks by giving the maximum number of poisoning points that can be tolerated to
Scale Factor c	Error Increase (%)	Sep/Size
1.0	2.2	0.27
2.0	3.1	0.15
3.0	4.4	0.10
Table 11: Impact of scale factor  $c$  on poisoning effectiveness for  $\mathcal{C}$  in the form of dimension-wise box-constraint as  $[0, c]$ . Base Error is  $1.2\%$ . Base Error and Sep/SD will be the same for all settings because support set of the clean distribution is the strict subset of  $\mathcal{C}$ .
correctly predict a known test point, which can also be straightforwardly applied to indiscriminate setting by treating different test samples independently, as mentioned in the discussion of LDC.
Because no data filtering is used for each partition of the defenses, at the partition level, our results (i.e., the computed metrics) obtained in each poisoned partition may still be similar to the results obtained on the whole data without partition (i.e., standard training, as in this paper), as the clean data and  $\mathcal{C}$  (reflected through the poisoning points assigned in each partition) may be more or less the same. At the aggregation level, similar to the discussion on LDC, these aggregation defenses may still result in overly pessimistic estimates on the effectiveness of injection only indiscriminate poisoning attacks as the certified accuracy at a particular poisoning ratio can be very loose, and the two possible reasons are: 1) straightforwardly applying results in targeted poisoning to indiscriminate poisoning might lead to overestimation and 2) considering the worst case adversary of modifying points might overestimate the power of injection only attacks in each poisoned partition. Therefore, our work can be related to aggregation defenses via reason 2), as it might be interpreted as the first step towards identifying factors that impact the attack effectiveness of injection only indiscriminate attacks in each poisoned partition, which may not always be highly detrimental depending on the learning task properties in each partition, while these aggregation defenses assume the existence of a single poisoning point in a partition can make the model in that partition successfully poisoned.
Loose certified accuracy in indiscriminate setting. Given a poisoning budget  $\epsilon$ , aggregation based defenses give a certified accuracy against indiscriminate poisoning attacks by first computing the tolerable fraction of poisoning points for each test sample and all the test samples with tolerable fraction smaller than or equal to  $\epsilon$  are certifiably robust. Then, the fraction of those test samples to the total test samples gives the certified accuracy for the test set. Similar to the result of CIFAR10 shown in LDC, here, we provide an additional result of certified accuracy for neural networks trained on the MNIST dataset: the state-of-the-art finite aggregation (FA) method [51] gives a certified accuracy of  $0\%$  at  $1\%$  poisoning ratio (600/60,000) using  $k = 1200$  partitions while at the much higher  $3\%$  poisoning ratio, the current state-of-the-art indiscriminate poisoning attack [32] can only reduce the accuracy of the neural network trained on MNIST without partitioning (i.e.,  $k = 1$ , a far less optimal choice from the perspective of aggregation defenses) from over  $99\%$  to only around  $90\%$ .
F Extension to Multi-class Settings and Non-linear Learners
In this section, we first provide the high-level idea of extending the metric computation from binary setting to multi-class setting and then provide empirical results on multi-class linear models and show that these metrics can still well-explain the observations in multi-class linear classifiers (Appendix F.1). Then, we provide the idea of extending the metrics from linear models to neural networks (NNs) and also the accompanying experimental results (Appendix F.2). In particular, we find that, for the same learner (e.g., same or similar NN architecture), our metrics may still be able to explain the different dataset vulnerabilities. However, the extended metrics cannot explain the vulnerabilities of datasets that are under different learners (e.g., NN with significantly different architectures), whose investigation is a very interesting future work, but is out of the scope of this paper.
F.1 Extension to Multi-class Linear Learners
Multi-class classifications are very common in practice and therefore, it is important to extend the computation of the metrics from binary classification to multi-class classification. For linear models,  $k$ -class classification ( $k > 2$ ) is handled by treating it as binary classifications in the "one vs one"
Datasets	Base Error (%)	Poisoned Error (%)	Increased Error (%)	Sep/SD	Sep/Size
SVM	7.4%	15.4%	8.0%	2.23/2.73	0.06/0.03
LR	7.7%	30.6%	22.9%	1.15	0.02
Table 12: Results of Linear Models on MNIST using  $3\%$  poisoning ratio. The "Poisoned Error" is directly quoted from Lu et al. [31] and SVM one is quoted from Koh et al. [25]. SVM contains two values for Sep/SD and Sep/Size because there are two binary pairs with the lowest value for each of the two metrics (lowest value is made bold).