SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Fnu Suya $^{1}$  Xiao Zhang $^{2}$  Yuan Tian $^{3}$  David Evans $^{1}$
<sup>1</sup>University of Virginia <sup>2</sup>CISPA Helmholtz Center for Information Security
<sup>3</sup>University California Los Angeles
suya@virginia.edu, xiao.zhang@cispa.de, yuant@ucla.edu, evans@virginia.edu
We study indiscriminate poisoning for linear learners where an adversary injects a few crafted examples into the training data with the goal of forcing the induced model to incur higher test error. Inspired by the observation that linear learners on some datasets are able to resist the best known attacks even without any defenses, we further investigate whether datasets can be inherently robust to indiscriminate poisoning attacks for linear learners. For theoretical Gaussian distributions, we rigorously characterize the behavior of an optimal poisoning attack, defined as the poisoning strategy that attains the maximum risk of the induced model at a given poisoning budget. Our results prove that linear learners can indeed be robust to indiscriminate poisoning if the class-wise data distributions are well-separated with low variance and the size of the constraint set containing all permissible poisoning points is also small. These findings largely explain the drastic variation in empirical attack performance of the state-of-the-art poisoning attacks on linear learners across benchmark datasets, making an important initial step towards understanding the underlying reasons some learning tasks are vulnerable to data poisoning attacks.
Machine learning models, especially current large-scale models, require large amounts of labeled training data, which are often collected from untrusted third parties [6]. Training models on these potentially malicious data poses security risks. A typical application is in spam filtering, where the spam detector is trained using data (i.e., emails) that are generated by users with labels provided often implicitly by user actions. In this setting, spammers can generate spam messages that inject benign words likely to occur in spam emails such that models trained on these spam messages will incur significant drops in filtering accuracy as benign and malicious messages become indistinguishable [37, 20]. These kinds of attacks are known as poisoning attacks. In a poisoning attack, the attacker injects a relatively small number of crafted examples into the original training set such that the resulting trained model (known as the poisoned model) performs in a way that satisfies certain attacker goals.
One commonly studied poisoning attacks in the literature are indiscriminate poisoning attacks [4, 54, 35, 45, 5, 46, 25, 31, 10], in which the attackers aim to let induced models incur larger test errors compared to the model trained on a clean dataset. Other poisoning goals, including targeted [42, 56, 24, 21, 18] and subpopulation [22, 46] attacks, are also worth studying and may correspond to more realistic attack goals. We focus on indiscriminate poisoning attacks as these attacks interfere with the fundamental statistical properties of the learning algorithm [45, 25], but include a summary of prior work on understanding limits of poisoning attacks in other settings in the related work section.
Indiscriminate poisoning attack methods have been developed that achieve empirically strong poisoning attacks in many settings [45, 46, 25, 31], but these works do not explain why the proposed attacks are sometimes ineffective. In addition, the evaluations of these attacks can be deficient in
some aspects [3, 4, 45, 5, 25, 31] (see Section 3) and hence, may not be able to provide an accurate picture on the current progress of indiscriminate poisoning attacks on linear models. The goal of our work is to understand the properties of the learning tasks that prevent effective attacks under linear models. An attack is considered effective if the increased error (or risk in the distributional sense) from poisoning exceeds the injected poisoning ratio [31, 25].
In this paper, we consider indiscriminate data poisoning attacks for linear models, the commonly studied victim models in the literature [3, 4, 24, 45, 10]. Attacks on linear models are also studied very recently [46, 25, 5, 9] and we limit our scope to linear models because attacks on the simplest linear models are still not well understood, despite extensive prior empirical work in this setting. Linear models continue to garner significant interest due to their simplicity and high interpretability in explaining predictions [30, 39]. Linear models also achieve competitive performance in many security-critical applications for which poisoning is relevant, including training with differential privacy [49], recommendation systems [15] and malware detection [7, 8, 41, 11, 44, 2]. From a practical perspective, linear models continue to be relevant—for example, Amazon SageMaker [1], a scalable framework to train ML models intended for developers and business analysts, provides linear models for tabular data, and trains linear models (on top of pretrained feature extractors) for images.
Contributions. We show that state-of-the-art poisoning strategies for linear models all have similar attack effectiveness of each dataset, whereas their performance varies significantly across different datasets (Section 3). All tested attacks are very effective on benchmark datasets such as Dogfish and Enron, while none of them are effective on other datasets, such as selected MNIST digit pairs (e.g., 6-9) and Adult, even when the victim does not employ any defenses (Figure 1). To understand whether this observation means such datasets are inherently robust to poisoning attacks or just that state-of-the-art attacks are suboptimal, we first introduce general definitions of optimal poisoning attacks for both finite-sample and distributional settings (Definitions 4.1 and 4.2). Then, we prove that under certain regularity conditions, the performance achieved by an optimal poisoning adversary with finite-samples converges asymptotically to the actual optimum with respect to the underlying distribution (Theorem 4.3), and the best poisoning performance is always achieved at the maximum allowable poisoning ratio under mild conditions (Theorem 4.5).
Building upon these definitions, we rigorously characterize the behavior of optimal poisoning attacks under a theoretical Gaussian mixture model (Theorem 5.3), and derive upper bounds on their effectiveness for general data distributions (Theorem 5.7). In particular, we discover that a larger projected constraint size (Definition 5.5) is associated with a higher inherent vulnerability, whereas projected data distributions with a larger separability and smaller standard deviation (Definition 5.6) are fundamentally less vulnerable to poisoning attacks (Section 5.2). Empirically, we find the discovered learning task properties and the gained theoretical insights largely explain the drastic difference in attack performance observed for state-of-the-art indiscriminate poisoning attacks on linear models across benchmark datasets (Section 6). Finally, we discuss the potential implications of our work by showing how one might improve robustness to poisoning via better feature transformations and defenses (e.g., data sanitization defenses) to limit the impact of poisoning points (Section 7).
Related Work. Several prior works developed indiscriminate poisoning attacks by injecting a small fraction of poisoning points. One line of research adopts iterative gradient-based methods to directly maximize the surrogate loss chosen by the victim [3, 35, 34, 24], leveraging the idea of influence functions [40]. Another approach bases attacks on convex optimization methods [45, 46, 25], which provide a more efficient way to generate poisoned data, often with an additional input of a target model. Most of these works focus on studying linear models, but recently there has been some progress on designing more effective attacks against neural networks with insights learned from attacks evaluated on linear models [31, 32]. All the aforementioned works focus on developing different indiscriminate poisoning algorithms and some also characterize the hardness of poisoning in the model-targeted setting [46, 32], but did not explain why certain datasets are seemingly harder to poison than others. Our work leverages these attacks to empirically estimate the inherent vulnerabilities of benchmark datasets to poisoning, but focuses on providing explanations for the disparate poisoning vulnerability across the datasets. Besides injection, some other works consider different poisoning settings from ours by modifying up to the whole training data, also known as unlearnable examples [19, 55, 16] and some of them, similar to us, first rigorously analyze poisoning on theoretical distributions and then generalize the insights to general distributions [48, 47].
Although much research focuses on indiscriminate poisoning, many realistic attack goals are better captured as targeted attacks [42, 56, 24, 21, 18], where the adversary's goal is to induce a model that misclassifies a particular known instance, or subpopulation attacks [22, 46], where the adversary's goal is to produce misclassifications for a defined subset of the distribution. A recent work that studies the inherent vulnerabilities of datasets to targeted data poisoning attacks proposed the Lethal Dose Conjecture (LDC) [50]: given a dataset of size  $N$ , the tolerable amount of poisoning points from any targeted poisoning attack generated through insertion, deletion or modifications is  $\Theta(N/n)$ , where  $n$  is the sample complexity of the most data-efficient learner trained on the clean data to correctly predict a known test sample. Compared to our work, LDC is more general and applies to any dataset, any learning algorithm, and even different poisoning settings (e.g., deletion, insertion). In contrast, our work focuses on insertion-only indiscriminate attacks for linear models. However, the general setting for LDC can result in overly pessimistic estimates of the power of insertion-only indiscriminate poisoning attacks. In addition, the key factor of the sample complexity  $n$  in LDC is usually unknown and difficult to determine. Our work complements LDC by making an initial step towards finding factors (which could be related to  $n$ ) under a particular attack scenario to better understand the power of indiscriminate data poisoning attacks. Appendix E provides more details.
We consider binary classification tasks. Let  $\mathcal{X} \subseteq \mathbb{R}^n$  be the input space and  $\mathcal{Y} = \{-1, +1\}$  be the label space. Let  $\mu_c$  be the joint distribution of clean inputs and labels. For standard classification tasks, the goal is to learn a hypothesis  $h: \mathcal{X} \to \mathcal{Y}$  that minimizes  $\operatorname{Risk}(h; \mu_c) = \mathbb{P}_{(\boldsymbol{x}, y) \sim \mu_c}[h(\boldsymbol{x}) \neq y]$ . Instead of directly minimizing risk, typical machine learning methods find an approximately good hypothesis  $h$  by restricting the search space to a specific hypothesis class  $\mathcal{H}$ , then optimizing  $h$  by minimizing some convex surrogate loss:  $\min_{h \in \mathcal{H}} L(h; \mu_c)$ . In practical applications with only a finite number of samples, model training replaces the population measure  $\mu_c$  with its empirical counterpart. The surrogate loss for  $h$  is defined as  $L(h; \mu) = \mathbb{E}_{(\boldsymbol{x}, y) \sim \mu}[\ell(h; \boldsymbol{x}, y)]$ , where  $\ell(h; \boldsymbol{x}, y)$  denotes the non-negative individual loss of  $h$  incurred at  $(\boldsymbol{x}, y)$ .
We focus on the linear hypothesis class and study the commonly considered hinge loss in prior poisoning literature [3, 4, 45, 25, 46] when deriving the optimal attacks for 1-D Gaussian distributions in Section 5.1. Our results can be extended to other linear methods such as logistic regression (LR). A linear hypothesis parameterized by a weight parameter  $\boldsymbol{w} \in \mathbb{R}^n$  and a bias parameter  $b \in \mathbb{R}$  is defined as:  $h_{\boldsymbol{w}, b}(\boldsymbol{x}) = \mathrm{sgn}(\boldsymbol{w}^\top \boldsymbol{x} + b)$  for any  $\boldsymbol{x} \in \mathbb{R}^n$ , where  $\mathrm{sgn}(\cdot)$  denotes the sign function. For any  $\boldsymbol{x} \in \mathcal{X}$  and  $y \in \mathcal{Y}$ , the hinge loss of a linear classifier  $h_{\boldsymbol{w}, b}$  is defined as:
$$
\ell \left(h _ {\boldsymbol {w}, b}; \boldsymbol {x}, y\right) = \max  \{0, 1 - y \left(\boldsymbol {w} ^ {\top} \boldsymbol {x} + b\right) \} + \frac {\lambda}{2} \| \boldsymbol {w} \| _ {2} ^ {2}, \tag {1}
$$
where  $\lambda \geq 0$  is the tuning parameter which penalizes the  $\ell_2$ -norm of the weight parameter  $\pmb{w}$ .
Threat Model. We formulate indiscriminate data poisoning attack as a game between an attacker and a victim in practice following the prior work [45]:
1. A clean training dataset  $S_{c}$  is produced, where each data point is i.i.d. sampled from  $\mu_c$ .
2. The attacker generates a poisoned dataset  $S_{p}$  using some poisoning strategy  $\mathcal{A}$ , which aims to reduce the performance of the victim model by injecting  $S_{p}$  into the training dataset.
3. The victim minimizes empirical surrogate loss  $L(\cdot)$  on  $S_{c} \cup S_{p}$  and produces a model  $\hat{h}_{p}$ .
The attacker's goal is to find a poisoning strategy  $\mathcal{A}$  such that the risk of the final induced classifier  $\operatorname{Risk}(\hat{h}_p; \mu_c)$  is as high as possible, which is empirically estimated on a set of fresh testing data that are i.i.d. sampled from  $\mu_c$ . We assume the attacker has full knowledge of the learning process, including the clean distribution  $\mu_c$  or the clean training dataset  $S_c$ , the hypothesis class  $\mathcal{H}$ , the surrogate loss function  $\ell$  and the learning algorithm adopted by the victim.
We impose two restrictions to the poisoning attack:  $|\mathcal{S}_p| \leq \epsilon \cdot |\mathcal{S}_c|$  and  $\mathcal{S}_p \subseteq \mathcal{C}$ , where  $\epsilon \in [0,1]$  is the poisoning budget and  $\mathcal{C} \subseteq \mathcal{X} \times \mathcal{Y}$  is a bounded subset that captures the feasibility constraints for poisoned data. We assume that  $\mathcal{C}$  is specified in advance with respect to different applications (e.g., normalized pixel values of images can only be in range [0, 1]) and possible defenses the victim may choose (e.g., points that have larger Euclidean distance from center will be removed) [45, 25]. Here, we focus on undefended victim models, i.e.,  $\mathcal{C}$  is specified based on application constraints, so
Figure 1: Performance of best current indiscriminate poisoning attacks with  $\epsilon = 3\%$  across different benchmark datasets on linear SVM. Datasets are sorted from lowest to highest base error rate.
as to better assess the inherent dataset vulnerabilities without active protections. However, defense strategies such as data sanitization [12, 45, 25] may shrink the size of  $\mathcal{C}$  so that the poisoned data are less extreme and harmful. We provide preliminary experimental results on this in Section 7.
3 Disparate Poisoning Vulnerability of Benchmark Datasets
Prior evaluations of poisoning attacks on convex models are inadequate in some aspects, either being tested on very small datasets (e.g., significantly subsampled MNIST 1-7 dataset) without competing baselines [3, 10, 34, 35], generating invalid poisoning points [45, 25] or lacking diversity in the evaluated convex models/datasets [31, 32]. This motivates us to carefully evaluate representative attacks for linear models on various benchmark datasets without considering additional defenses.
**Experimental Setup.** We evaluate the state-of-the-art data poisoning attacks for linear models: Influence Attack [24, 25], KKT Attack [25], Min-Max Attack [45, 25], and Model-Targeted Attack (MTA) [46]. We consider both linear SVM and LR models and evaluate the models on benchmark datasets including different MNIST [28] digit pairs (MNIST 1–7, as used in prior evaluations [45, 25, 3, 46], in addition to MNIST 4–9 and MNIST 6–9 which were picked to represent datasets that are relatively easier/harder to poison), and other benchmark datasets used in prior evaluations including Dogfish [24], Enron [36] and Adult [22, 46]. Filtered Enron is obtained by filtering out  $3\%$  of near boundary points (with respect to the clean decision boundary of linear SVM) from Enron. The motivation of this dataset (along with Adult) is to show that, the initial error cannot be used to trivially infer the vulnerability to poisoning attacks (Section 6). We choose  $3\%$  as the poisoning rate following previous works [45, 25, 31, 32]. Appendix D.1 provides details on the experimental setup.
Results. Figure 1 shows the highest error from the tested poisoning attacks (they perform similarly in most cases) on linear SVM. Similar observations are found on LR in Appendix D.2.5. At  $3\%$  poisoning ratio, the (absolute) increased test errors of datasets such as MNIST 6-9 and MNIST 1-7 are less than  $4\%$  for SVM while for other datasets such as Dogfish, Enron and Filtered Enron, the increased error is much higher than the poisoning ratio, indicating that these datasets are more vulnerable to poisoning. Dogfish is moderately vulnerable ( $\approx 9\%$  increased error) while Enron and Filtered Enron are highly vulnerable with over  $30\%$  of increased error. Consistent with prior work [45, 25, 32], throughout this paper, we measure the increased error to determine whether a dataset is vulnerable to poisoning attacks. However, in security-critical applications, the ratio between the increased error and the initial error might matter more but leave its exploration as future work. These results reveal a drastic difference in robustness of benchmark datasets to state-of-the-art indiscriminate poisoning attacks which was not explained in prior works. A natural question to ask from this observation is are datasets like MNIST digits inherently robust to poisoning attacks or just resilient to current attacks? Since estimating the performance of optimal poisoning attacks for benchmark datasets is very challenging, we first characterize optimal poisoning attacks for theoretical distributions and then study their partial characteristics for general distributions in Section 5.
4 Defining Optimal Poisoning Attacks
In this section, we lay out formal definitions of optimal poisoning attacks and study their general implications. First, we introduce a notion of finite-sample optimal poisoning to formally define the optimal poisoning attack in the practical finite-sample setting with respect to our threat model:
Definition 4.1 (Finite-Sample Optimal Poisoning). Consider input space  $\mathcal{X}$  and label space  $\mathcal{Y}$ . Let  $\mu_c$  be the underlying data distribution of clean inputs and labels. Let  $S_c$  be a set of examples sampled i.i.d. from  $\mu_c$ . Suppose  $\mathcal{H}$  is the hypothesis class and  $\ell$  is the surrogate loss function that are used for learning. For any  $\epsilon \geq 0$  and  $\mathcal{C} \subseteq \mathcal{X} \times \mathcal{Y}$ , a finite-sample optimal poisoning adversary  $\hat{\mathcal{A}}_{\mathrm{opt}}$  is defined to be able to generate some poisoned dataset  $S_p^*$  such that:
$$
\mathcal {S} _ {p} ^ {*} = \operatorname * {a r g m a x} _ {\mathcal {S} _ {p}} \operatorname {R i s k} (\hat {h} _ {p}; \mu_ {c}) \quad \text {s . t .}   \mathcal {S} _ {p} \subseteq \mathcal {C} \text {a n d} | \mathcal {S} _ {p} | \leq \epsilon \cdot | \mathcal {S} _ {c} |,
$$
where  $\hat{h}_p = \operatorname*{argmin}_{h\in \mathcal{H}}\sum_{(\boldsymbol {x},y)\in \mathcal{S}_c\cup \mathcal{S}_p}\ell (h;\boldsymbol {x},y)$  denotes the empirical loss minimizer.
Definition 4.1 states that no poisoning strategy can achieve a better attack performance than that achieved by  $\hat{\mathcal{A}}_{\mathrm{opt}}$ . If we denote by  $\hat{h}_p^*$  the hypothesis produced by minimizing the empirical loss on  $S_{c} \cup S_{p}^{*}$ , then  $\operatorname{Risk}(\hat{h}_p^*; \mu_c)$  can be regarded as the maximum achievable attack performance.
Next, we introduce a more theoretical notion of distributional optimal poisoning, which generalizes Definition 4.1 from finite-sample datasets to data distributions.
Definition 4.2 (Distributional Optimal Poisoning). Consider the same setting as in Definition 4.1. A distributional optimal poisoning adversary  $\mathcal{A}_{\mathrm{opt}}$  is defined to be able to generate some poisoned data distribution  $\mu_p^*$  such that:
$$
(\mu_ {p} ^ {*}, \delta^ {*}) = \operatorname * {a r g m a x} _ {(\mu_ {p}, \delta)} \operatorname {R i s k} (h _ {p}; \mu_ {c}) \quad \text {s . t .} \operatorname {s u p p} (\mu_ {p}) \subseteq \mathcal {C} \text {a n d} 0 \leq \delta \leq \epsilon ,
$$
where  $h_p = \operatorname*{argmin}_{h\in \mathcal{H}}\left\{L(h;\mu_c) + \delta \cdot L(h;\mu_p)\right\}$  denotes the population loss minimizer.
Similar to the finite-sample case, Definition 4.2 implies that there is no feasible poisoned distribution  $\mu_p$  such that the risk of its induced hypothesis is higher than that attained by  $\mu_p^*$ . Theorem 4.3, proven in Appendix A.1, connects Definition 4.1 and Definition 4.2. The formal definitions of uniform convergence, strong convexity and Lipschitz continuity are given in Appendix A.1.
Theorem 4.3. Consider the same settings as in Definitions 4.1 and 4.2. Suppose  $\mathcal{H}$  satisfies the uniform convergence property with function  $m_{\mathcal{H}}(\cdot, \cdot)$ . Assume  $\ell$  is  $b$ -strongly convex and  $\operatorname{Risk}(h; \mu_c)$  is  $\rho$ -Lipschitz continuous with respect to model parameters for some  $b, \rho > 0$ . Let  $\hat{h}_p^* = \operatorname{argmin}_{h \in \mathcal{H}} \sum_{(\boldsymbol{x}, y) \in S_c \cup S_p^*} \ell(h; \boldsymbol{x}, y)$  and  $h_p^* = \operatorname{argmin}_{h \in \mathcal{H}} \{L(h; \mu_c) + \delta^* \cdot L(h; \mu_p^*)\}$ . For any  $\epsilon', \delta' \in (0, 1)$ , if  $|\mathcal{S}_c| \geq m_{\mathcal{H}}(\epsilon', \delta')$ , then with probability at least  $1 - \delta'$ ,
$$
\left| \mathrm {R i s k} (\hat {h} _ {p} ^ {*}; \mu_ {c}) - \mathrm {R i s k} (h _ {p} ^ {*}; \mu_ {c}) \right| \leq 2 \rho \sqrt {\frac {\epsilon^ {\prime}}{b}}.
$$
Remark 4.4. Theorem 4.3 assumes three regularity conditions to ensure the finite-sample optimal poisoning attack is a consistent estimator of the distributional optimal one (i.e., insights on poisoning from distributional settings can transfer to finite-sample settings): the uniform convergence property of  $\mathcal{H}$  that guarantees empirical minimization of surrogate loss returns a good hypothesis, the strong convexity condition that ensures a unique loss minimizer, and the Lipschitz condition that translates the closeness of model parameters to the closeness of risk. These conditions hold for most (properly regularized) convex problems and input distributions with bounded densities. The asymptotic convergence rate is determined by the function  $m_{\mathcal{H}}$ , which depends on the complexity of the hypothesis class  $\mathcal{H}$  and the surrogate loss  $\ell$ . For instance, if we choose  $\lambda$  carefully, sample complexity of the linear hypothesis class for a bounded hinge loss is  $\Omega(1 / (\epsilon'))^2$ , where  $\epsilon'$  is the error bound parameter for specifying the uniform convergence property (see Definition A.3) and other problem-dependent parameters are hidden in the big- $\Omega$  notation (see Section 15 of [43] for details). We note the generalization of optimal poisoning attack for linear case is related to agnostic learning of halfspaces [23], which also imposes assumptions on the underlying distribution such as anti-concentration assumption [13, 17] similar to the Lipschitz continuity condition assumed in Theorem 4.3. The theorem applies to any loss function that is strongly convex (e.g., general convex losses with the  $\ell_2$ - or elastic net regularizers), not just the hinge loss.