Title: What Distributions are Robust to Indiscriminate Poisoning Attacks for Linear Learners?
Venue: NeurIPS 2023
Accepted: yes
Keywords: poisoning attacks; adversarial machine learning; machine learning security
Abstract: We study indiscriminate poisoning for linear learners where an adversary injects a few  crafted examples into the training data with the goal of forcing the induced model to incur higher test error. Inspired by the observation that linear learners on some datasets are able to resist the best known attacks even without any defenses, we further investigate whether datasets can be inherently robust to indiscriminate poisoning attacks for linear learners. For theoretical Gaussian distributions, we rigorously characterize the behavior of an optimal poisoning attack, defined as the poisoning strategy that attains the maximum risk of the induced model at a given poisoning budget. Our results prove that linear learners can indeed be robust to indiscriminate poisoning if the class-wise data distributions are well-separated with low variance and the size of the constraint set containing all permissible poisoning points is also small. These findings largely explain the drastic variation in empirical attack performance of the state-of-the-art poisoning attacks on linear learners across benchmark datasets, making an important initial step towards understanding the underlying reasons some learning tasks are vulnerable to data poisoning attacks.

What Distributions are Robust to Indiscriminate Poisoning Attacks for Linear Learners?

Abstract

We study indiscriminate poisoning for linear learners where an adversary injects a few crafted examples into the training data with the goal of forcing the induced model to incur higher test error. Inspired by the observation that linear learners on some datasets are able to resist the best known attacks even without any defenses, we further investigate whether datasets can be inherently robust to indiscriminate poisoning attacks for linear learners. For theoretical Gaussian distributions, we rigorously characterize the behavior of an optimal poisoning attack, defined as the poisoning strategy that attains the maximum risk of the induced model at a given poisoning budget. Our results prove that linear learners can indeed be robust to indiscriminate poisoning if the class-wise data distributions are well-separated with low variance and the size of the constraint set containing all permissible poisoning points is also small. These findings largely explain the drastic variation in empirical attack performance of the state-of-the-art poisoning attacks on linear learners across benchmark datasets, making an important initial step towards understanding the underlying reasons some learning tasks are vulnerable to data poisoning attacks.

1 Introduction

Machine learning models, especially current large-scale models, require large amounts of labeled training data, which are often collected from untrusted third parties [6]. Training models on these potentially malicious data poses security risks. A typical application is in spam filtering, where the spam detector is trained using data (i.e., emails) that are generated by users with labels provided often implicitly by user actions. In this setting, spammers can generate spam messages that inject benign words likely to occur in spam emails such that models trained on these spam messages will incur significant drops in filtering accuracy as benign and malicious messages become indistinguishable [37, 20]. These kinds of attacks are known as poisoning attacks . In a poisoning attack, the attacker injects a relatively small number of crafted examples into the original training set such that the resulting trained model (known as the poisoned model ) performs in a way that satisfies certain attacker goals.

One commonly studied poisoning attacks in the literature are indiscriminate poisoning attacks [4, 54, 35, 45, 5, 46, 25, 31, 10], in which the attackers aim to let induced models incur larger test errors compared to the model trained on a clean dataset. Other poisoning goals, including targeted [42, 56, 24, 21, 18] and subpopulation [22, 46] attacks, are also worth studying and may correspond to more realistic attack goals. We focus on indiscriminate poisoning attacks as these attacks interfere with the fundamental statistical properties of the learning algorithm [45, 25], but include a summary of prior work on understanding limits of poisoning attacks in other settings in the related work section.

Indiscriminate poisoning attack methods have been developed that achieve empirically strong poisoning attacks in many settings [45, 46, 25, 31], but these works do not explain why the proposed attacks are sometimes ineffective. In addition, the evaluations of these attacks can be deficient in

some aspects [3, 4, 45, 5, 25, 31] (see Section 3) and hence, may not be able to provide an accurate picture on the current progress of indiscriminate poisoning attacks on linear models. The goal of our work is to understand the properties of the learning tasks that prevent effective attacks under linear models. An attack is considered effective if the increased error (or risk in the distributional sense) from poisoning exceeds the injected poisoning ratio [31, 25].

In this paper, we consider indiscriminate data poisoning attacks for linear models, the commonly studied victim models in the literature [3, 4, 24, 45, 10]. Attacks on linear models are also studied very recently [46, 25, 5, 9] and we limit our scope to linear models because attacks on the simplest linear models are still not well understood, despite extensive prior empirical work in this setting. Linear models continue to garner significant interest due to their simplicity and high interpretability in explaining predictions [30, 39]. Linear models also achieve competitive performance in many security-critical applications for which poisoning is relevant, including training with differential privacy [49], recommendation systems [15] and malware detection [7, 8, 41, 11, 44, 2]. From a practical perspective, linear models continue to be relevant-for example, Amazon SageMaker [1], a scalable framework to train ML models intended for developers and business analysts, provides linear models for tabular data, and trains linear models (on top of pretrained feature extractors) for images.

Contributions. We show that state-of-the-art poisoning strategies for linear models all have similar attack effectiveness of each dataset, whereas their performance varies significantly across different datasets (Section 3). All tested attacks are very effective on benchmark datasets such as Dogfish and Enron, while none of them are effective on other datasets, such as selected MNIST digit pairs (e.g., 6-9) and Adult, even when the victim does not employ any defenses (Figure 1). To understand whether this observation means such datasets are inherently robust to poisoning attacks or just that state-of-the-art attacks are suboptimal, we first introduce general definitions of optimal poisoning attacks for both finite-sample and distributional settings (Definitions 4.1 and 4.2). Then, we prove that under certain regularity conditions, the performance achieved by an optimal poisoning adversary with finite-samples converges asymptotically to the actual optimum with respect to the underlying distribution (Theorem 4.3), and the best poisoning performance is always achieved at the maximum allowable poisoning ratio under mild conditions (Theorem 4.5).

Building upon these definitions, we rigorously characterize the behavior of optimal poisoning attacks under a theoretical Gaussian mixture model (Theorem 5.3), and derive upper bounds on their effectiveness for general data distributions (Theorem 5.7). In particular, we discover that a larger projected constraint size (Definition 5.5) is associated with a higher inherent vulnerability, whereas projected data distributions with a larger separability and smaller standard deviation (Definition 5.6) are fundamentally less vulnerable to poisoning attacks (Section 5.2). Empirically, we find the discovered learning task properties and the gained theoretical insights largely explain the drastic difference in attack performance observed for state-of-the-art indiscriminate poisoning attacks on linear models across benchmark datasets (Section 6). Finally, we discuss the potential implications of our work by showing how one might improve robustness to poisoning via better feature transformations and defenses (e.g., data sanitization defenses) to limit the impact of poisoning points (Section 7).

Related Work. Several prior works developed indiscriminate poisoning attacks by injecting a small fraction of poisoning points. One line of research adopts iterative gradient-based methods to directly maximize the surrogate loss chosen by the victim [3, 35, 34, 24], leveraging the idea of influence functions [40]. Another approach bases attacks on convex optimization methods [45, 46, 25], which provide a more efficient way to generate poisoned data, often with an additional input of a target model. Most of these works focus on studying linear models, but recently there has been some progress on designing more effective attacks against neural networks with insights learned from attacks evaluated on linear models [31, 32]. All the aforementioned works focus on developing different indiscriminate poisoning algorithms and some also characterize the hardness of poisoning in the model-targeted setting [46, 32], but did not explain why certain datasets are seemingly harder to poison than others. Our work leverages these attacks to empirically estimate the inherent vulnerabilities of benchmark datasets to poisoning, but focuses on providing explanations for the disparate poisoning vulnerability across the datasets. Besides injection, some other works consider different poisoning settings from ours by modifying up to the whole training data, also known as unlearnable examples [19, 55, 16] and some of them, similar to us, first rigorously analyze poisoning on theoretical distributions and then generalize the insights to general distributions [48, 47].

Although much research focuses on indiscriminate poisoning, many realistic attack goals are better captured as targeted attacks [42, 56, 24, 21, 18], where the adversary's goal is to induce a model that misclassifies a particular known instance, or subpopulation attacks [22, 46], where the adversary's goal is to produce misclassifications for a defined subset of the distribution. A recent work that studies the inherent vulnerabilities of datasets to targeted data poisoning attacks proposed the Lethal Dose Conjecture (LDC) [50]: given a dataset of size N , the tolerable amount of poisoning points from any targeted poisoning attack generated through insertion, deletion or modifications is Θ( N/n ) , where n is the sample complexity of the most data-efficient learner trained on the clean data to correctly predict a known test sample. Compared to our work, LDC is more general and applies to any dataset, any learning algorithm, and even different poisoning settings (e.g., deletion, insertion). In contrast, our work focuses on insertion-only indiscriminate attacks for linear models. However, the general setting for LDC can result in overly pessimistic estimates of the power of insertion-only indiscriminate poisoning attacks. In addition, the key factor of the sample complexity n in LDC is usually unknown and difficult to determine. Our work complements LDC by making an initial step towards finding factors (which could be related to n ) under a particular attack scenario to better understand the power of indiscriminate data poisoning attacks. Appendix E provides more details.

2 Preliminaries

We consider binary classification tasks. Let X ⊆ R n be the input space and Y = {-1 , +1 } be the label space. Let µ c be the joint distribution of clean inputs and labels. For standard classification tasks, the goal is to learn a hypothesis h : X → Y that minimizes Risk( h ; µ c ) = P ( x ,y ) ∼ µ c [ h ( x ) = y ] . Instead of directly minimizing risk, typical machine learning methods find an approximately good hypothesis h by restricting the search space to a specific hypothesis class H , then optimizing h by minimizing some convex surrogate loss: min h ∈H L ( h ; µ c ) . In practical applications with only a finite number of samples, model training replaces the population measure µ c with its empirical counterpart. The surrogate loss for h is defined as L ( h ; µ ) = E ( x ,y ) ∼ µ [ ℓ ( h ; x , y ) ] , where ℓ ( h ; x , y ) denotes the non-negative individual loss of h incurred at ( x , y ) .

̸

We focus on the linear hypothesis class and study the commonly considered hinge loss in prior poisoning literature [3, 4, 45, 25, 46] when deriving the optimal attacks for 1-D Gaussian distributions in Section 5.1. Our results can be extended to other linear methods such as logistic regression (LR). A linear hypothesis parameterized by a weight parameter w ∈ R n and a bias parameter b ∈ R is defined as: h w ,b ( x ) = sgn( w ⊤ x + b ) for any x ∈ R n , where sgn( · ) denotes the sign function. For any x ∈ X and y ∈ Y , the hinge loss of a linear classifier h w ,b is defined as:

where λ ≥ 0 is the tuning parameter which penalizes the ℓ 2 -norm of the weight parameter w .

Threat Model. We formulate indiscriminate data poisoning attack as a game between an attacker and a victim in practice following the prior work [45]:

A clean training dataset S c is produced, where each data point is i.i.d. sampled from µ c .

The attacker generates a poisoned dataset S p using some poisoning strategy A , which aims to reduce the performance of the victim model by injecting S p into the training dataset.

The victim minimizes empirical surrogate loss L ( · ) on S c ∪ S p and produces a model ˆ h p .

The attacker's goal is to find a poisoning strategy A such that the risk of the final induced classifier Risk( ˆ h p ; µ c ) is as high as possible, which is empirically estimated on a set of fresh testing data that are i.i.d. sampled from µ c . We assume the attacker has full knowledge of the learning process, including the clean distribution µ c or the clean training dataset S c , the hypothesis class H , the surrogate loss function ℓ and the learning algorithm adopted by the victim.

We impose two restrictions to the poisoning attack: |S p | ≤ ϵ · |S c | and S p ⊆ C , where ϵ ∈ [0 , 1] is the poisoning budget and C ⊆ X × Y is a bounded subset that captures the feasibility constraints for poisoned data. We assume that C is specified in advance with respect to different applications (e.g., normalized pixel values of images can only be in range [0 , 1] ) and possible defenses the victim may choose (e.g., points that have larger Euclidean distance from center will be removed) [45, 25]. Here, we focus on undefended victim models, i.e., C is specified based on application constraints, so

Filtered Enron

MNIST6-9

Dogfish

MNIST1-7

Enron

MNIST4-9

Adult

0

20

Test Error (%)

Trained on

c

Trained on

c

p

as to better assess the inherent dataset vulnerabilities without active protections. However, defense strategies such as data sanititation [12, 45, 25] may shrink the size of C so that the poisoned data are less extreme and harmful. We provide preliminary experimental results on this in Section 7.

3 Disparate Poisoning Vulnerability of Benchmark Datasets

Prior evaluations of poisoning attacks on convex models are inadequate in some aspects, either being tested on very small datasets (e.g., significantly subsampled MNIST 1-7 dataset) without competing baselines [3, 10, 34, 35], generating invalid poisoning points [45, 25] or lacking diversity in the evaluated convex models/datasets [31, 32]. This motivates us to carefully evaluate representative attacks for linear models on various benchmark datasets without considering additional defenses.

Experimental Setup. We evaluate the state-of-the-art data poisoning attacks for linear models: Influence Attack [24, 25], KKT Attack [25], Min-Max Attack [45, 25], and Model-Targeted Attack (MTA) [46]. We consider both linear SVM and LR models and evaluate the models on benchmark datasets including different MNIST [28] digit pairs (MNIST 1-7, as used in prior evaluations [45, 25, 3, 46], in addition to MNIST 4-9 and MNIST 6-9 which were picked to represent datasets that are relatively easier/harder to poison), and other benchmark datasets used in prior evaluations including Dogfish [24], Enron [36] and Adult [22, 46]. Filtered Enron is obtained by filtering out 3% of near boundary points (with respect to the clean decision boundary of linear SVM) from Enron. The motivation of this dataset (along with Adult) is to show that, the initial error cannot be used to trivially infer the vulnerability to poisoning attacks (Section 6). We choose 3% as the poisoning rate following previous works [45, 25, 31, 32]. Appendix D.1 provides details on the experimental setup.

Results. Figure 1 shows the highest error from the tested poisoning attacks (they perform similarly in most cases) on linear SVM. Similar observations are found on LR in Appendix D.2.5. At 3% poisoning ratio, the (absolute) increased test errors of datasets such as MNIST 6-9 and MNIST 1-7 are less than 4% for SVM while for other datasets such as Dogfish, Enron and Filtered Enron, the increased error is much higher than the poisoning ratio, indicating that these datasets are more vulnerable to poisoning. Dogfish is moderately vulnerable ( ≈ 9% increased error) while Enron and Filtered Enron are highly vulnerable with over 30% of increased error. Consistent with prior work [45, 25, 32], throughout this paper, we measure the increased error to determine whether a dataset is vulnerable to poisoning attacks. However, in security-critical applications, the ratio between the increased error and the initial error might matter more but leave its exploration as future work. These results reveal a drastic difference in robustness of benchmark datasets to state-of-the-art indiscriminate poisoning attacks which was not explained in prior works. A natural question to ask from this observation is are datasets like MNIST digits inherently robust to poisoning attacks or just resilient to current attacks? Since estimating the performance of optimal poisoning attacks for benchmark datasets is very challenging, we first characterize optimal poisoning attacks for theoretical distributions and then study their partial characteristics for general distributions in Section 5.

4 Defining Optimal Poisoning Attacks

In this section, we lay out formal definitions of optimal poisoning attacks and study their general implications. First, we introduce a notion of finite-sample optimal poisoning to formally define the optimal poisoning attack in the practical finite-sample setting with respect to our threat model:

Definition 4.1 (Finite-Sample Optimal Poisoning) . Consider input space X and label space Y . Let µ c be the underlying data distribution of clean inputs and labels. Let S c be a set of examples sampled i.i.d. from µ c . Suppose H is the hypothesis class and ℓ is the surrogate loss function that are used for learning. For any ϵ ≥ 0 and C ⊆ X × Y , a finite-sample optimal poisoning adversary ˆ A opt is defined to be able to generate some poisoned dataset S ∗ p such that:

where ˆ h p = argmin h ∈H ∑ ( x ,y ) ∈S c ∪S p ℓ ( h ; x , y ) denotes the empirical loss minimizer.

Definition 4.1 states that no poisoning strategy can achieve a better attack performance than that achieved by ˆ A opt . If we denote by ˆ h ∗ p the hypothesis produced by minimizing the empirical loss on S c ∪ S ∗ p , then Risk( ˆ h ∗ p ; µ c ) can be regarded as the maximum achievable attack performance.

Next, we introduce a more theoretical notion of distributional optimal poisoning , which generalizes Definition 4.1 from finte-sample datasets to data distributions.

Definition 4.2 (Distributional Optimal Poisoning) . Consider the same setting as in Definition 4.1. A distributional optimal poisoning adversary A opt is defined to be able to generate some poisoned data distribution µ ∗ p such that:

where h p = argmin h ∈H { L ( h ; µ c ) + δ · L ( h ; µ p ) } denotes the population loss minimizer.

Similar to the finite-sample case, Definition 4.2 implies that there is no feasible poisoned distribution µ p such that the risk of its induced hypothesis is higher than that attained by µ ∗ p . Theorem 4.3, proven in Appendix A.1, connects Definition 4.1 and Definition 4.2. The formal definitions of uniform convergence, strong convexity and Lipschitz continuity are given in Appendix A.1.

Theorem 4.3. Consider the same settings as in Definitions 4.1 and 4.2. Suppose H satisfies the uniform convergence property with function m H ( · , · ) . Assume ℓ is b -strongly convex and Risk( h ; µ c ) is ρ -Lipschitz continuous with respect to model parameters for some b, ρ > 0 . Let ˆ h ∗ p = argmin h ∈H ∑ ( x ,y ) ∈S c ∪S ∗ p ℓ ( h ; x , y ) and h ∗ p = argmin h ∈H { L ( h ; µ c ) + δ ∗ · L ( h ; µ ∗ p ) } . For any ϵ ′ , δ ′ ∈ (0 , 1) , if |S c | ≥ m H ( ϵ ′ , δ ′ ) , then with probability at least 1 -δ ′ ,

Remark 4.4 . Theorem 4.3 assumes three regularity conditions to ensure the finite-sample optimal poisoning attack is a consistent estimator of the distributional optimal one (i.e., insights on poisoning from distributional settings can transfer to finite-sample settings): the uniform convergence property of H that guarantees empirical minimization of surrogate loss returns a good hypothesis, the strong convexity condition that ensures a unique loss minimizer, and the Lipschitz condition that translates the closeness of model parameters to the closeness of risk. These conditions hold for most (properly regularized) convex problems and input distributions with bounded densities. The asymptotic convergence rate is determined by the function m H , which depends on the complexity of the hypothesis class H and the surrogate loss ℓ . For instance, if we choose λ carefully, sample complexity of the linear hypothesis class for a bounded hinge loss is Ω(1 / ( ϵ ′ ) 2 ) , where ϵ ′ is the error bound parameter for specifying the uniform convergence property (see Definition A.3) and other problemdependent parameters are hidden in the bigΩ notation (see Section 15 of [43] for details). We note the generalization of optimal poisoning attack for linear case is related to agnostic learning of halfspaces [23], which also imposes assumptions on the underlying distribution such as anti-concentration assumption [13, 17] similar to the Lipschitz continuity condition assumed in Theorem 4.3. The theorem applies to any loss function that is strongly convex (e.g., general convex losses with the ℓ 2 -or elastic net regularizers), not just the hinge loss.

Moreover, we note that δ ∗ represents the ratio of injected poisoned data that achieves the optimal attack performance. In general, δ ∗ can be any value in [0 , ϵ ] , but we show in Theorem 4.5, proven in Appendix A.2, that optimal poisoning can always be achieved with ϵ -poisoning under mild conditions.

Theorem 4.5. The optimal poisoning attack performance defined in Definition 4.2 can always be achieved by choosing ϵ as the poisoning ratio, if either of the following conditions is satisfied:

The support of the clean distribution supp( µ c ) ⊆ C .

H is a convex hypothesis class, and for any h θ ∈ H , there always exists a distribution µ such that supp( µ ) ⊆ C and ∂ ∂ θ L ( h θ ; µ ) = 0 .

Remark 4.6 . Theorem 4.5 characterizes the conditions under which the optimal performance is guaranteed to be achieved with the maximum poisoning ratio ϵ . Note that the first condition supp( µ c ) ⊆ C is mild because it typically holds for poisoning attacks against undefended classifiers. When attacking classifiers that employ some defenses such as data sanitization, the condition supp( µ c ) ⊆ C might not hold, due to the fact that the proposed defense may falsely reject some clean data points as outliers (i.e., related to false positive rates). The second condition complements the first one in that it does not require the victim model to be undefended, however, it requires H to be convex. We prove in Appendix A.3 that for linear hypothesis with hinge loss, such a µ can be easily constructed. The theorem enables us to conveniently characterize the optimal poisoning attacks in Section 5.1 by directly using ϵ . When the required conditions are satisfied, this theorem also provides a simple sanity check on whether a poisoning attack is optimal. In particular, if a candidate attack is optimal, the risk of the induced model is monotonically non-decreasing with respect to the poisoning ratio. Note that the theorem above applies to any strongly convex loss function.

5 Characterizing Optimal Poisoning Attacks

This section characterizes the distributional optimal poisoning attacks with respect to linear hypothesis class. We first consider a theoretical 1-dimensional Gaussian mixture model and exactly characterize optimal poisoning attack and then discuss the implications on the underlying factors that potentially cause the inherent vulnerabilities to poisoning attacks for general high-dimensional distributions.

5.1 One-Dimensional Gaussian Mixtures

Consider binary classification tasks using hinge loss with one-dimensional inputs, where X = R and Y = {-1 , +1 } . Let µ c be the underlying clean data distribution, where each example ( x, y ) is assumed to be i.i.d. sampled according to the following Gaussian mixture model:

where σ 1 , σ 2 > 0 and p ∈ (0 , 1) . Without loss of generality, we assume γ 1 ≤ γ 2 . Following our threat model, we let ϵ ≥ 0 be the maximum poisoning ratio and C := [ -u, u ] × Y for some u > 0 be the constraint set. Let H L = { h w,b : w ∈ {-1 , 1 } , b ∈ R } be the linear hypothesis class with normalized weights. Note that we consider a simplified setting where the weight parameter w ∈ {-1 , 1 } . 1 Since ∥ w ∥ 2 is fixed, we also set λ = 0 in the hinge loss function (1).

For clarify in presentation, we outline the connection between the definitions and the main theorem below, which will also show the high-level proof sketch of Theorem 5.3. We first prove that in order to understand the optimal poisoning attacks, it is sufficient to study the family of two-point distributions (Definition 5.1), instead of any possible distributions, as the poisoned data distribution. Based on this reduction and a specification of weight flipping condition (Definition 5.2), we then rigorously characterize the optimal attack performance with respect to different configurations of task-related parameters (Theorem 5.3). Similar conclusions may also be made with logistic loss.

Definition 5.1 (Two-point Distribution) . For any α ∈ [0 , 1] , ν α is defined as a two-point distribution , if for any ( x, y ) sampled from ν α ,

Definition 5.2 (Weight-Flipping Condition) . Consider the assumed Gaussian mixture model (2) and the linear hypothesis class H L . Let g be an auxiliary function such that for any b ∈ R ,

where Φ is the cumulative distribution function (CDF) of standard Gaussian N (0 , 1) . Let ϵ > 0 be the poisoning budget and g -1 be the inverse of g , then the weight-flipping condition is defined as:

Now we are ready to present our main theoretical results. The following theorem rigorously characterizes the behavior of the distributional optimal poisoning adversary A opt under the Gaussian mixture model (2) and the corresponding optimal attack performance:

Theorem 5.3. Suppose the clean distribution µ c follows the Gaussian mixture model (2) with p = 1 / 2 , γ 1 ≤ γ 2 , and σ 1 = σ 2 = σ . Assume u ≥ 1 and | γ 1 + γ 2 | ≤ 2( u -1) . There always exists some α ∈ [0 , 1] such that the optimal attack performance defined in Definition 4.2 is achieved with δ = ϵ and µ p = ν α , where ν α is defined by (3) . More specifically, if h ∗ p = argmin h ∈H L { L ( h ; µ c )+ ϵ · L ( h ; ν α ) } denotes the induced hypothesis with optimal poisoning, then

where s = max { g -1 ( ϵ ) -g -1 (0) , g -1 (0) -g -1 ( -ϵ ) } and g ( · ) is defined in Definition 5.2.

Remark 5.4 . The proof of Theorem 5.3 is given in Appendix B.1. We considered same variance of σ 1 = σ 2 = σ simply for the convenience in proof without involving minor but tedious calculations, and the same conclusion can be made using different variances. Theorem 5.3 characterizes the exact behavior of A opt for typical combinations of hyperparameters under the considered model, including distribution related parameters such as γ 1 , γ 2 , σ and poisoning related parameters such as ϵ , u . We use ϵ directly because the poisoning setup in the 1-D Gaussian distribution satisfies the second condition in Theorem 4.5. Hence, the best performance can be achieved with the maximum poisoning ratio ϵ . For u , a larger of it suggests the weight-flipping condition (4) is more likely to be satisfied, as an attacker can generate poisoned data with larger hinge loss to flip the weight parameter w . Class separability | γ 1 -γ 2 | and within-class variance σ also play an important role in affecting the optimal attack performance. If the ratio | γ 1 -γ 2 | /σ is large, then we know the initial risk Risk( h c ; µ c ) = Φ( γ 1 -γ 2 2 σ ) will be small. Consider the case where condition (4) is satisfied. Note that Φ( γ 2 -γ 1 2 σ ) = 1 -Φ( γ 1 -γ 2 2 σ ) implies an improved performance of optimal poisoning attack thus an higher inherent vulnerabilities to data poisoning attacks. However, it is worth noting that there is an implicit assumption in condition (4) that the weight parameter can be flipped from w = 1 to w = -1 . A large value of | γ 1 -γ 2 | /σ also implies that flipping the weight parameter becomes more difficult, since the gap between the hinge loss with respect to µ c for a hypothesis with w = -1 and that with w = 1 becomes larger. Moreover, if condition (4) cannot be satisfied, then a larger ratio of | γ 1 -γ 2 | /σ suggests that it is more difficult to move the decision boundary to incur an increase in test error, because of the number of correctly classified boundary points will increase at a faster rate. In summary, Theorem 5.3 suggests that a smaller value of u and a larger ratio of | γ 1 -γ 2 | /σ increases the inherent robustness to indiscriminate poisoning for typical configurations under our model (2). Empirical verification of the above theoretical results is given in Appendix G.

5.2 General Distributions

Recall that we have identified several key factors (i.e., u , | γ 1 -γ 2 | and σ ) for 1-D Gaussian distributions in Section 5.1 which are highly related to the performance of an optimal distributional poisoning adversary A opt . In this section, we demonstrate how to generalize the definition of these factors to high-dimensional datasets and illustrate how they affect an inherent robustness upper bound to indiscriminate poisoning attacks for linear learners. In particular, we project the clean distribution µ c and the constraint set C onto some vector w , then compute those factors based on the projections.

Definition 5.5 (Projected Constraint Size) . Let C ⊆ X × Y be the constraint set for poisoning. For any w ∈ R n , the projected constraint size of C with respect to w is defined as:

According to Definition 5.5, Size w ( C ) characterizes the size of the constraint set C when projected onto the (normalized) projection vector w / ∥ w ∥ 2 then scaled by ∥ w ∥ 2 , the ℓ 2 -norm of w . In theory, the constraint sets conditioned on y = -1 and y = +1 can be different, but for simplicity and practical considerations, we simply assume they are the same in the following discussions.

Definition 5.6 (Projected Separability and Standard Deviation) . Let X ⊆ R n , Y = {-1 , +1 } , and µ c be the underlying distribution. Let µ -and µ + be the input distributions with labels of -1 and +1 respectively. For any w ∈ R n , the projected separability of µ c with respect to w is defined as:

In addition, the projected standard deviation of µ c with respect to w is defined as:

For finite-sample settings, we simply replace the input distributions with their empirical counterparts to compute the sample statistics of Sep w ( µ c ) and SD w ( µ c ) . Note that the above definitions are specifically for linear models, but out of curiosity, we also provide initial thoughts on how to extend these metrics to neural networks (see Appendix F for preliminary results). Below, we provide justifications on how the three factors are related to the optimal poisoning attacks. Theorem 5.7 and the techniques used in its proof in Appendix B.2 are inspired by the design of Min-Max Attack [45].

Theorem 5.7. Consider input space X ⊆ R n , label space Y , clean distribution µ c and linear hypothesis class H . For any h w ,b ∈ H , x ∈ X and y ∈ Y , let ℓ ( h w ,b ; x , y ) = ℓ M ( -y ( w ⊤ x + b )) be a margin-based loss adopted by the victim, where ℓ M is convex and non-decreasing. Let C ⊆ X × Y be the constraint set and ϵ > 0 be the poisoning budget. Suppose h c = argmin h ∈H L ( h ; µ c ) has weight w c and h ∗ p is the poisoned model induced by optimal adversary A opt , then we have

Remark 5.8 . Theorem 5.7 proves an upper bound on the inherent vulnerability to indiscriminate poisoning for linear learners, which can be regarded as a necessary condition for the optimal poisoning attack. A smaller upper bound likely suggests a higher inherent robustness to poisoning attacks. In particular, the right hand side of (5) consists of two terms: the clean population loss of h c and a term related to the projected constraint size. Intuitively, the projected separability and standard deviation metrics highly affect the first term, since data distribution with a higher Sep w c ( µ c ) and a lower SD w c ( µ c ) implies a larger averaged margin with respect to h c , which further suggests a smaller L ( h c ; µ c ) . The second term is determined by the poisoning budget ϵ and the projected constraint size, or more precisely, a larger ϵ and a larger Size w c ( C ) indicate a higher upper bound on Risk( h ∗ p ; µ c ) . In addition, we set h = h c and the projection vector as w c for the last inequality of (5), because h c achieves the smallest population surrogate loss with respect to the clean data distribution µ c . However, choosing h = h c may not always produce a tighter upper bound on Risk( h ∗ p ; µ c ) since there is no guarantee that the projected constraint size Size w c ( C ) is small. An interesting future direction is to select a more appropriate projection vector that returns a tight, if not the tightest, upper bound on Risk( h ∗ p ; µ c ) for any clean distribution µ c (see Appendix D.2 for preliminary experiments). We also note that the results above apply to any (non-decreasing and strongly-convex) margin-based losses.

6 Experiments

Recall from Theorem 4.3 and Remark 4.4 that the finite-sample optimal poisoning attack is a consistent estimator of the distributional one for linear learners. In this section, we demonstrate the theoretical insights gained from Section 5, despite proven only for the distributional optimal attacks, still appear to largely explain the empirical performance of best attacks across benchmark datasets.

Given a clean training data S c , we empirically estimate the three distributional metrics defined in Section 5.2 on the clean test data with respect to the weight w c . Since ∥ w c ∥ 2 may vary across different datasets while the predictions of linear models (i.e., the classification error) are invariant to the scaling of ∥ w c ∥ 2 , we use ratios to make their metrics comparable: Sep w c ( µ c ) / SD w c ( µ c ) (denoted as Sep/SD in Table 1) and Sep w c ( µ c ) / Size w c ( C ) (Sep/Size). According to our theoretical results, we expect datasets that are less vulnerable to poisoning have higher values for both metrics.

Table 1 summarizes the results where Error Increase is produced by the best attacks from the stateof-the-art attacks mentioned in Section 3. The Sep/SD and Sep/Size metrics are highly correlated to the poisoning effectiveness (as measured by error increase). Datasets such as MNIST 1-7 and MNIST 6-9 are harder to poison than others, which correlates with their increased separability and reduced impact from the poisoning points. In contrast, datasets such as Enron and Filtered Enron are highly vulnerable and this is strongly correlated to their small separability and the large impact from the admissible poisoning points. The results of Filtered Enron (low base error, high increased error) and Adult (high base error, low increased error) demonstrate that poisoning vulnerability, measured by the error increase, cannot be trivially inferred from the initial base error.

When the base error is small, which is the case for all tested benchmark datasets except Adult, the empirical metrics are highly correlated to the error increase and also the final poisoned error. However, when the base error becomes high as it is for Adult, the empirical metrics are highly correlated to the final poisoned error, but not the error increase, if the metrics are computed on the entire (clean) test data. For the error increase, computing the metrics on the clean and correctly classified (by w c ) test points is more informative as it (roughly) accounts for the (newly) induced misclassification from poisoning. Therefore, we report metrics based on correctly-classified test points in Table 1 and defer results of the whole test data to Appendix D.2. For datasets except Adult, both ways of computing the metrics produce similar results. The Adult dataset is very interesting in that it is robust to poisoning (i.e., small error increase) despite having a very high base error. As a preliminary experiment, we also extended our analysis to neural networks (Appendix F) and find the identified factors are correlated to the varied poisoning effectiveness when different datasets are compared under similar learners.

Our current results only show the correlation of the identified factors to the varied (empirical) poisoning effectiveness, not causality. However, we also speculate that these factors may cause different poisoning effectiveness because: 1) results in Section 5 characterize the relationship of these factors to the performance of optimal attacks exactly for 1-D Gaussian distributions and partially for the general distributions, and 2) preliminary results on MNIST 1-7 shows that gradual changes in one factor (with the other fixed) also causes gradual changes in the vulnerability to poisoning (Appendix D.2.6). We leave the detailed exploration on their causal relationship as future work.

7 Discussion on Future Implications

Our results imply future defenses by explaining why candidate defenses work and motivating defenses to improve separability and reduce projected constraint size. We present two ideas-using data filtering might reduce projected constraint size and using better features might improve separability.

Reduced projected constraint size. The popular data sanitization defense works by filtering out bad points. We speculate it works because the defense may be effectively limiting the projected constraint size of C . To test this, we picked the combination of Sphere and Slab defenses considered in prior works [25, 45] to protect the vulnerable Enron dataset. We find that with the defense, the test error is increased from 3.2% to 28.8% while without the defense the error can be increased from 2.9% to 34.8%. Although limited in effectiveness, the reduced Error Increase with the defense is highly correlated to the significantly reduced projected constraint size Size w c ( C ) (and hence a significantly higher Sep/Size)-the Sep/Size metric jumps from 0.01 without defense to 0.11 with defense while

Sep/SD remains almost the same at 1.18 with and without defense. Similar conclusions can also be drawn for MNIST 1-7 and Dogfish (detailed experimental results are in Appendix D.2.3).

Better feature representation. We consider a transfer learning scenario where the victim trains a linear model on a clean pretrained model. As a preliminary experiment, we train LeNet and ResNet18

models on the CIFAR10 dataset till convergence, but record the intermediate models of ResNet18 to produce models with different feature extractors (RX denotes ResNet18 trained for X epochs). We then use the feature extraction layers of these models (including LeNet) as the pretrained models and obtain features of CIFAR10 images with labels 'Truck' and 'Ship', and train linear models on them.

We evaluate the robustness of this dataset against poisoning attacks and set C as dimension-wise box-constraints, whose values are the minimum and maximum values of the clean data points for each dimension when fed to the feature extractors. This way of configuring C corresponds to the practical scenario where the victim has

4

5

6

Sep/SD

0.15

0.20

0.25

0.30

0.35

Sep/Size

LeNet

R-1

R-50

R-90

R-100

R-120

0

1

2

3

4

5

6

Error Increase (%)

access to some small number of clean samples so that they can deploy a simple defense of filtering out inputs that do not fall into a dimension-wise box constraint that is computed from the available clean samples of the victim. Figure 2 shows that as the feature extractor becomes better (using deeper models and training for more epochs), both the Sep/SD and Sep/Size metrics increase, leading to reduced error increase. Quantitatively, the Pearson correlation coefficient between the error increase and each of the two factors is -0 . 98 (a strong negative correlation). This result demonstrates the possibility that better feature representations free from poisoning might be leveraged to improve downstream resilience against indiscriminate poisoning attacks. We also simulated a more practical scenario where the training of the pretrained models never sees any data (i.e., data from classes of 'Truck" and 'Ship") used in the downstream analysis and the conclusions are similar (Appendix D.2.7).

8 Limitations

Our work also has several limitations. We only characterize the optimal poisoning attacks for theoretical distributions under linear models. Even for the linear models, the identified metrics cannot quantify the actual error increase from optimal poisoning attacks, which is an interesting future work, and one possible approach might be to tighten the upper bound in Theorem 5.7 using better optimization methods. The metrics identified in this paper are learner dependent, depending on the properties of the learning algorithm, dataset and domain constraints (mainly reflected through C ). In certain applications, one might be interested in understanding the impact of learner agnostic dataset properties on poisoning effectiveness-a desired dataset has such properties that any reasonable learners trained on the dataset can be robust to poisoning attacks. One likely application scenario is when the released data will be used by many different learners in various applications, each of which may be prone to poisoning. We also did not systematically investigate how to compare the vulnerabilities of different datasets under different learning algorithms. Identifying properties specific to the underlying learner that affect the performance of (optimal) data poisoning attacks is challenging but interesting future work.

Although we focus on indiscriminate data poisoning attacks in this paper, we are optimistic that our results will generalize to subpopulation or targeted poisoning settings. In particular, the specific learning task properties identified in this paper may still be highly correlated, but now additional factors related to the relative positions of the subpopulations/individual test samples to the rest of the population under the clean decision boundary will also play important roles.

References

Amazon, Inc. Amazon SageMaker. https://aws.amazon.com/sagemaker/ . Accessed: 2023-05-01.

Daniel Arp, Michael Spreitzenbarth, Malte Hubner, Hugo Gascon, Konrad Rieck, and CERT Siemens. Drebin: Effective and explainable detection of android malware in your pocket. In Network and Distributed System Security , 2014.

Battista Biggio, Blaine Nelson, and Pavel Laskov. Support Vector Machines Under Adversarial Label Noise. In Asian Conference on Machine Learning , 2011.

Battista Biggio, Blaine Nelson, and Pavel Laskov. Poisoning Attacks against Support Vector Machines. In International Conference on Machine Learning , 2012.

Mustain Billah, Adnan Anwar, Ziaur Rahman, and Syed Md Galib. Bi-level poisoning attack model and countermeasure for appliance consumption data of smart homes. Energies , 14(13):3887, 2021.

Nicholas Carlini, Matthew Jagielski, Christopher A Choquette-Choo, Daniel Paleka, Will Pearce, Hyrum Anderson, Andreas Terzis, Kurt Thomas, and Florian Tramèr. Poisoning web-scale training datasets is practical. arXiv preprint arXiv:2302.10149 , 2023.

Yizheng Chen, Zhoujie Ding, and David Wagner. Continuous learning for Android malware detection. arXiv preprint arXiv:2302.04332 , 2023.

Yizheng Chen, Shiqi Wang, Yue Qin, Xiaojing Liao, Suman Jana, and David Wagner. Learning security classifiers with verified global robustness properties. In ACM Conference on Computer and Communications Security , 2021.

Antonio Emanuele Cinà, Sebastiano Vascon, Ambra Demontis, Battista Biggio, Fabio Roli, and Marcello Pelillo. The hammer and the nut: Is bilevel optimization really needed to poison linear classifiers? In International Joint Conference on Neural Networks , 2021.

Ambra Demontis, Marco Melis, Maura Pintor, Matthew Jagielski, Battista Biggio, Alina Oprea, Cristina Nita-Rotaru, and Fabio Roli. Why Do adversarial attacks transfer? Explaining transferability of evasion and poisoning attacks. In USENIX Security Symposium , 2019.

Ambra Demontis, Paolo Russu, Battista Biggio, Giorgio Fumera, and Fabio Roli. On security and sparsity of linear classifiers for adversarial settings. In Structural, Syntactic, and Statistical Pattern Recognition: Joint IAPR International Workshop . Springer, 2016.

Ilias Diakonikolas, Gautam Kamath, Daniel Kane, Jerry Li, Jacob Steinhardt, and Alistair Stewart. Sever: A robust meta-algorithm for stochastic optimization. In International Conference on Machine Learning , 2019.

Ilias Diakonikolas, Daniel Kane, and Nikos Zarifis. Near-optimal SQ lower bounds for agnostically learning halfspaces and ReLUs under Gaussian marginals. Advances in Neural Information Processing Systems , 2020.

Dheeru Dua and Casey Graff. UCI Machine Learning Repository, 2017.

Maurizio Ferrari Dacrema, Paolo Cremonesi, and Dietmar Jannach. Are we really making much progress? a worrying analysis of recent neural recommendation approaches. In ACM Conference on Recommender Systems , 2019.

Liam Fowl, Micah Goldblum, Ping-yeh Chiang, Jonas Geiping, Wojciech Czaja, and Tom Goldstein. Adversarial examples make strong poisons. Advances in Neural Information Processing Systems , 34:30339-30351, 2021.

Spencer Frei, Yuan Cao, and Quanquan Gu. Agnostic learning of halfspaces with gradient descent via soft margins. In International Conference on Machine Learning , 2021.

Jonas Geiping, Liam Fowl, W Ronny Huang, Wojciech Czaja, Gavin Taylor, Michael Moeller, and Tom Goldstein. Witches' Brew: Industrial Scale Data Poisoning via Gradient Matching. In International Conference on Learning Representations , 2021.

Hanxun Huang, Xingjun Ma, Sarah Monazam Erfani, James Bailey, and Yisen Wang. Unlearnable examples: Making personal data unexploitable. arXiv preprint arXiv:2101.04898 , 2021.

Ling Huang, Anthony D Joseph, Blaine Nelson, Benjamin IP Rubinstein, and J Doug Tygar. Adversarial Machine Learning. In ACM Workshop on Security and Artificial Intelligence , 2011.

WRonny Huang, Jonas Geiping, Liam Fowl, Gavin Taylor, and Tom Goldstein. MetaPoison: Practical general-purpose clean-label data poisoning. In Advances in Neural Information Processing Systems , 2020.

Matthew Jagielski, Giorgio Severi, Niklas Pousette Harger, and Alina Oprea. Subpopulation data poisoning attacks. In Proceedings of the 2021 ACM SIGSAC Conference on Computer and Communications Security , pages 3104-3122, 2021.

Adam Tauman Kalai, Adam R Klivans, Yishay Mansour, and Rocco A Servedio. Agnostically learning halfspaces. SIAM Journal on Computing , 37(6):1777-1805, 2008.

Pang Wei Koh and Percy Liang. Understanding black-box predictions via influence functions. In International Conference on Machine Learning , 2017.

Pang Wei Koh, Jacob Steinhardt, and Percy Liang. Stronger data poisoning attacks break data sanitization defenses. Machine Learning , 111(1):1-47, 2022.

Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.

Yann LeCun. The MNIST database of handwritten digits, 1998.

Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE , 86(11):2278-2324, 1998.

Alexander Levine and Soheil Feizi. Deep partition aggregation: Provable defense against general poisoning attacks. arXiv preprint arXiv:2006.14768 , 2020.

Yue Liu, Chakkrit Tantithamthavorn, Li Li, and Yepang Liu. Explainable AI for Android malware detection: Towards understanding why the models perform so well? In International Symposium on Software Reliability Engineering , 2022.

Yiwei Lu, Gautam Kamath, and Yaoliang Yu. Indiscriminate data poisoning attacks on Neural Networks. arXiv preprint arXiv:2204.09092 , 2022.

Yiwei Lu, Gautam Kamth, and Yaoliang Yu. Exploring the limits of indiscriminate data poisoning attacks. arXiv preprint arXiv:2303.03592 , 2023.

Andrew Maas, Raymond E Daly, Peter T Pham, Dan Huang, Andrew Y Ng, and Christopher Potts. Learning word vectors for sentiment analysis. In Proceedings of the 49th annual meeting of the association for computational linguistics: Human language technologies , pages 142-150, 2011.

Shike Mei and Xiaojin Zhu. The Security of Latent Dirichlet Allocation. In Artificial Intelligence and Statistics , 2015.

Shike Mei and Xiaojin Zhu. Using Machine Teaching to Identify Optimal Training-Set Attacks on Machine Learners. In AAAI Conference on Artificial Intelligence , 2015.

Vangelis Metsis, Ion Androutsopoulos, and Georgios Paliouras. Spam filtering with Naive Bayes-Which Naive Bayes? In Third Conference on Email and Anti-Spam , 2006.

Blaine Nelson, Marco Barreno, Fuching Jack Chi, Anthony D Joseph, Benjamin IP Rubinstein, Udam Saini, Charles A Sutton, J Doug Tygar, and Kai Xia. Exploiting machine learning to subvert your spam filter. In USENIX Workshop on Large Scale Exploits and Emergent Threats , 2008.

F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research , 12:2825-2830, 2011.

Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. 'Why should I trust you?' Explaining the predictions of any classifier. In ACM SIGKDD International Conference on Knowledge Discovery and Data Mining , 2016.

Peter J Rousseeuw, Frank R Hampel, Elvezio M Ronchetti, and Werner A Stahel. Robust statistics: the approach based on influence functions . John Wiley & Sons, 2011.

Ahmad Salah, Eman Shalabi, and Walid Khedr. A lightweight Android malware classifier using novel feature selection methods. Symmetry , 12(5):858, 2020.

Ali Shafahi, W Ronny Huang, Mahyar Najibi, Octavian Suciu, Christoph Studer, Tudor Dumitras, and Tom Goldstein. Poison Frogs! Targeted clean-label poisoning attacks on Neural Networks. In Advances in Neural Information Processing Systems , 2018.

Shai Shalev-Shwartz and Shai Ben-David. Understanding machine learning: From theory to algorithms . Cambridge University Press, 2014.

Nedim Šrndic and Pavel Laskov. Detection of malicious PDF files based on hierarchical document structure. In Network and Distributed System Security , 2013.

Jacob Steinhardt, Pang Wei W Koh, and Percy S Liang. Certified defenses for data poisoning attacks. Advances in Neural Information Processing Systems , 2017.

Fnu Suya, Saeed Mahloujifar, Anshuman Suri, David Evans, and Yuan Tian. Model-targeted poisoning attacks with provable convergence. In International Conference on Machine Learning , 2021.

Lue Tao, Lei Feng, Hongxin Wei, Jinfeng Yi, Sheng-Jun Huang, and Songcan Chen. Can adversarial training be manipulated by non-robust features? Advances in Neural Information Processing Systems , 35:26504-26518, 2022.

Lue Tao, Lei Feng, Jinfeng Yi, Sheng-Jun Huang, and Songcan Chen. Better safe than sorry: Preventing delusive adversaries with adversarial training. Advances in Neural Information Processing Systems , 34:16209-16225, 2021.

Florian Tramèr and Dan Boneh. Differentially private learning needs better features (or much more data). International Conference on Learning Representations , 2021.

Wenxiao Wang, Alexander Levine, and Soheil Feizi. Lethal Dose Conjecture on data poisoning. In 36th Conference on Neural Information Processing Systems , 2022.

Wenxiao Wang, Alexander J Levine, and Soheil Feizi. Improved certified defenses against data poisoning with (deterministic) finite aggregation. In International Conference on Machine Learning , pages 22769-22783. PMLR, 2022.

Yaqing Wang, Quanming Yao, James T Kwok, and Lionel M Ni. Generalizing from a few examples: A survey on few-shot learning. ACM computing surveys (csur) , 53(3):1-34, 2020.

Eric Wong and Zico Kolter. Provable defenses against adversarial examples via the convex outer adversarial polytope. In International Conference on Machine Learning , 2018.

Han Xiao, Huang Xiao, and Claudia Eckert. Adversarial Label Flips Attack on Support Vector Machines. In European Conference on Artificial Intelligence , 2012.

Da Yu, Huishuai Zhang, Wei Chen, Jian Yin, and Tie-Yan Liu. Indiscriminate poisoning attacks are shortcuts. 2021.

Chen Zhu, W Ronny Huang, Ali Shafahi, Hengduo Li, Gavin Taylor, Christoph Studer, and Tom Goldstein. Transferable clean-label poisoning attacks on Deep Neural Nets. In International Conference on Machine Learning , 2019.

A Proofs of Main Results in Section 4

A.1 Proof of Theorem 4.3

We first introduce the formal definitions of strong convexity and Lipschitz continuity conditions with respect to a function, and the uniform convergence property respect to a hypothesis class. These definitions are necessary for the proof of Theorem 4.3.

Definition A.1 (Strong Convexity) . A function f : R n → R is b -strongly convex for some b > 0 , if f ( x 1 ) ≥ f ( x 2 ) + ∇ f ( x 2 ) ⊤ ( x 1 -x 2 ) + b 2 ∥ x 1 -x 2 ∥ 2 2 for any x 1 , x 2 ∈ R n .

Definition A.2 (Lipschitz Continuity) . A function f : R n → R is ρ -Lipschitz for some ρ > 0 , if | f ( x 1 ) -f ( x 2 ) | ≤ ρ ∥ x 1 -x 2 ∥ 2 for any x 1 , x 2 ∈ R n .

Definition A.3 (Uniform Convergence) . Let H be a hypothesis class. We say that H satisfies the uniform convergence property with a loss function ℓ , if there exists a function m H : (0 , 1) 2 → N such that for every ϵ ′ , δ ′ ∈ (0 , 1) and for every probability distribution µ , if S is a set of examples with m ≥ m H ( ϵ ′ , δ ′ ) samples drawn i.i.d. from µ , then

Such a uniform convergence property, which can be achieved using the VC dimension or the Rademacher complexity of H , guarantees that the learning rule specified by empirical risk minimization always returns a good hypothesis with high probability [43]. Similar to PAC learning, the function m H measures the minimal sample complexity requirement that ensures uniform convergence.

Now, we are ready to prove Theorem 4.3

Proof of Theorem 4.3. First, we introduce the following notations to simplify the proof. For any S p , µ p and δ ≥ 0 , let

According to the definitions of ˆ h ∗ p and h ∗ p , we know ˆ h ∗ p = ˆ g ( S ∗ p , S c ) and h ∗ p = g ( δ ∗ , µ ∗ p , µ c ) .

Now we are ready to prove Theorem 4.3. For any S c sampled from µ c , consider the empirical loss minimizer ˆ h ∗ p = ˆ g ( S ∗ p , S c ) and the population loss minimizer g ( δ S ∗ p , ˆ µ S ∗ p , µ c ) , where δ S ∗ p = |S ∗ p | / |S c | . Then S ∗ p ∪ S c can be regarded as the i.i.d. sample set from ( µ c + δ S ∗ p · ˆ µ S ∗ p ) / (1 + δ S ∗ p ) . According to Definition A.3, since H satisfies the uniform convergence property with respect to ℓ , we immediately know that the empirical loss minimization is close to the population loss minimization if the sample size is large enough (see Lemma 4.2 in [43]). To be more specific, for any ϵ ′ , δ ′ ∈ (0 , 1) , if |S c | ≥ m H ( ϵ ′ , δ ′ ) , then with probability at least 1 -δ ′ , we have

In addition, since the surrogate loss ℓ is b -strongly convex and the population risk is ρ -Lipschitz, we further know the clean risk of ˆ g ( S ∗ p , S c ) and g ( δ S ∗ p , ˆ µ S ∗ p , µ c ) is guaranteed to be close. Namely, with probability at least 1 -δ ′ , we have

Note that δ S ∗ p ∈ [0 , ϵ ] and supp(ˆ µ S ∗ p ) ⊆ C . Thus, according to the definition of h ∗ p = g ( δ ∗ , µ ∗ p , µ c ) , we further have

So far, we have proven one direction of the asymptotic for results Theorem 4.3.

On the other hand, we can always construct a subset ˜ S p with size | ˜ S p | = δ ∗ · |S c | by i.i.d. sampling from µ ∗ p . Consider the empirical risk minimizer ˆ g ( ˜ S p , S c ) and the population risk minimizer h ∗ p = g ( δ ∗ , µ ∗ p , µ c ) . Similarly, since H satisfies the uniform convergence property, if |S c | ≥ m H ( ϵ ′ , δ ′ ) , then with probability at least 1 -δ ′ ,we have

According to the strong convexity of ℓ and the Lipschitz continuity of the population risk, we further have

Note that ˜ S p ⊆ C and | ˜ S p | = δ ∗ · |S c | ≤ ϵ · |S c | . Thus according to the definition of ˆ h ∗ p = ˆ g ( S ∗ p , S c ) , we have

Combining (6) and (7), we complete the proof of Theorem 4.3.

A.2 Proof of Theorem 4.5

Proof of Theorem 4.5. We prove Theorem 4.5 by construction.

We start with the first condition supp( µ c ) ⊆ C . Suppose δ ∗ < ϵ , since the theorem trivially holds if δ ∗ = ϵ . To simplify notations, define h p ( δ, µ p ) = argmin h ∈H { L ( h ; µ c ) + δ · L ( h ; µ p ) } for any δ and µ p . To prove the statement in Theorem 4.5, it is sufficient to show that there exists some µ ( ϵ ) p based on the first condition such that

The above equality means we can always achieve the same maximum risk after poisoning with the full poisoning budget ϵ . To proceed with the proof, we construct µ ( ϵ ) p based on µ c and µ ∗ p as follows:

We can easily check that µ ( ϵ ) p is a valid probability distribution and supp( µ ( ϵ ) p ) ⊆ C . In addition, we can show that

where the third equality holds because of the construction of µ ( ϵ ) p . Therefore, we have proven (8), which further implies the optimal attack performance can always be achieved with ϵ -poisoning as long as the first condition is satisfied.

Next, we turn to the second condition of Theorem 4.5. Similarly, it is sufficient to construct some µ ( ϵ ) p for the setting where δ ∗ < ϵ such that

We construct µ ( ϵ ) p based on µ ∗ p and the assumed data distribution µ . More specifically, we construct

By construction, we know µ ( ϵ ) p is a valid probability distribution. In addition, according to the assumption of supp( µ ) ⊆ C , we have supp( µ ( ϵ ) p ) ⊆ C . According to the assumption that for any θ , there exists a µ such that ∂ ∂ θ L ( h θ ; µ ) = 0 , we know for any possible weight parameter θ ∗ p of h p ( δ ∗ , µ ∗ p ) , there also exists a corresponding µ such that the gradient is 0 and therefore, we have

where the last equality is based on the first-order optimality condition of h p ( δ ∗ , µ ∗ p ) for convex losses. For simplicity, we also assumed h p ( δ ∗ , µ ∗ p ) is obtained by minimizing the loss on µ c + δ ∗ · µ ∗ p while in the case of supp( µ c ) ⊈ C , the victim usually minimizes the loss on ¯ µ c + δ ∗ · µ ∗ p , where ¯ µ c is the 'truncated' version of µ c such that supp(¯ µ c ) ⊆ C . To conclude, we know h p ( ϵ, µ ( ϵ ) p ) = h p ( δ ∗ , µ ∗ p ) holds for any possible h p ( δ ∗ , µ ∗ p ) and we complete the proof of Theorem 4.5.

A.3 Proofs of the Statement about Linear Models in Remark 4.6

Proof. We provide the construction of µ with respect to the second condition of Theorem 4.5 for linear models and hinge loss. Since for any h w ,b ∈ H L and any ( x , y ) ∈ X × Y , we have

Let θ = ( w , b ) , then the gradient with respect to w can be written as:

Similarly, the gradient with respect to b can be written as:

Therefore, for large input space X × Y , we can simply construct µ by constructing a two-point distribution (with equal probabilities of label +1 and -1) that cancels out each other's gradient (for both w and b ), so that y ( w ⊤ x + b ) ≤ 1 and -y · x + λ w = 0 .

We may also generalize the construction of µ from linear models with hinge loss to general convex models if the victim minimizes the loss on ¯ µ c + δ ∗ · µ ∗ p to obtain h p ( δ ∗ , µ ∗ p ) , which is common in practice. In this case, we can simply set µ = ¯ µ c + δ ∗ · µ ∗ p , which guarantees

B Proofs of Main Results in Section 5

B.1 Proof of Theorem 5.3

To prove Theorem 5.3, we need to make use of the following three auxiliary lemmas, which are related to the maximum population hinge loss with w = 1 (Lemma B.1), the weight-flipping condition (Lemma B.2) and the risk behaviour of any linear hypothesis under (2) (Lemma B.3). For the sake of completeness, we present the full statements of Lemma B.1 and Lemma B.2 as follows. In particular, Lemma B.1, proven in Appendix C.1, characterizes the maximum achievable hinge loss with respect to the underlying clean distribution µ c and some poisoned distribution µ p conditioned on w = 1 .

Lemma B.1. Suppose the underlying clean distribution µ c follows the Gaussian mixture model (2) with p = 1 / 2 and σ 1 = σ 2 = σ . Assume | γ 1 + γ 2 | ≤ 2 u . For any ϵ ≥ 0 , consider the following maximization problem:

where b p = argmin b ∈ R [ L ( h 1 ,b ; µ c ) + ϵ · L ( h 1 ,b ; µ p )] . There exists some α ∈ [0 , 1] such that the optimal value of (10) is achieved with µ p = ν α , where ν α is a two-point distribution with some parameter α ∈ [0 , 1] defined according to (3) .

Lemma B.1 suggests that it is sufficient to study the extreme two-point distributions ν α with α ∈ [0 , 1] to understand the maximum achievable population hinge loss conditioned on w = 1 . Lemma B.2, proven in Appendix C.2, characterizes the sufficient and necessary conditions in terms of ϵ , u and µ c , under which there exists a linear hypothesis with w = -1 that achieves the minimal value of population hinge loss with respect to µ c and some µ p .

Lemma B.2. Suppose the underlying clean distribution µ c follows the Gaussian mixture model (2) with p = 1 / 2 and σ 1 = σ 2 = σ . Assume | γ 1 + γ 2 | ≤ 2( u -1) for some u ≥ 1 . Let g be an auxiliary function such that for any b ∈ R ,

where Φ is the CDF of standard Gaussian. For any ϵ > 0 , there exists some µ p ∈ Q ( u ) such that argmin h w,b ∈H L [ L ( h w,b ; µ c ) + ϵ · L ( h w,b ; µ p )] outputs a hypothesis with w = -1 , if and only if

where ∆( s ) = L ( h 1 ,g -1 ( s ) ; µ c ) -min b ∈ R L ( h -1 ,b ; µ c ) + ϵ (1 + u ) -s · g -1 ( s ) , and g -1 denotes the inverse of g .

Lemma B.2 identifies sufficient and necessary conditions when a linear hypothesis with flipped weight parameter is possible. Note that we assume γ 1 ≤ γ 2 , thus flipping the weight parameter of the induced model from w = 1 to w = -1 is always favorable from an attacker's perspective. In particular, if the population hinge loss with respect to µ c and some µ p achieved by the loss minimizer conditioned on w = 1 is higher than that achieved by the loss minimizer with w = -1 , then we immediately know that flipping the weight parameter is possible, which further suggests the optimal poisoning attack performance must be achieved by some poisoned victim model with w = -1 .

Finally, we introduce Lemma B.3, proven in Appendix C.3, which characterizes the risk behavior of any linear hypothesis with respect to the assumed Gaussian mixture model (2).

Lemma B.3. Let µ c be the clean data distribution, where each example is sampled i.i.d. according to the data generating process specified in (2) . For any linear hypothesis h w,b ∈ H L , we have

where Φ denotes the CDF of standard Gaussian distribution N (0 , 1) .

Now we are ready to prove Theorem 5.3 using Lemmas B.1, B.2 and B.3.

Proof of Theorem 5.3. According to Theorem 4.5 and Remark 4.6, we note that the optimal poisoning performance in Definition 4.2 is always achieved with δ = ϵ . Therefore, we will only consider δ = ϵ in the following discussions.

Since the optimal poisoning performance is defined with respect to clean risk, it will be useful to understand the properties of Risk( h w,b ; µ c ) such as monotonicity and range. According to Lemma B.3, for any h w,b ∈ H L , we have

To understand the monotonicity of risk, we compute its derivative with respect to b :

If w = 1 , then Risk( h w,b ; µ c ) is monotonically decreasing when b ∈ ( -∞ , -γ 1 + γ 2 2 ) and monotonically increasing when b ∈ ( -γ 1 + γ 2 2 , ∞ ) , suggesting that minimum is achieved at b = -γ 1 + γ 2 2 and maximum is achieved when b goes to infinity. To be more specific, Risk( h 1 ,b ; µ c ) ∈ [Φ( γ 1 -γ 2 2 σ ) , 1 2 ] . On the other hand, if w = -1 , then Risk( h w,b ; µ c ) is monotonically increasing when b ∈ ( -∞ , γ 1 + γ 2 2 ) and monotonically decreasing when b ∈ ( γ 1 + γ 2 2 , ∞ ) , suggesting that maximum is achieved at b = γ 1 + γ 2 2 and minimum is achieved when b goes to infinity. Thus, Risk( h -1 ,b ; µ c ) ∈ [ 1 2 , Φ( γ 2 -γ 1 2 σ )] .

Based on the monotonicity analysis of Risk( h w,b ; µ c ) , we have the following two observations:

If there exists some feasible µ p such that h -1 ,b p = argmin h ∈H L { L ( h ; µ c ) + ϵL ( h ; µ p ) } can be achieved, then the optimal poisoning performance is achieved with w = -1 and b close to γ 1 + γ 2 2 as much as possible.

If there does not exist any feasible µ p that induces h -1 ,b p by minimizing the population hinge loss, then the optimal poisoning performance is achieved with w = 1 and b far from -γ 1 + γ 2 2 as much as possible (conditioned that the variance σ is the same for the two classes).

Recall that we prove in Lemma B.2 specifies a sufficient and necessary condition for the existence of such h -1 ,b p , which is equivalent to the condition (4) presented in Lemma B.2. Note that according to Lemma C.1, b = γ 1 + γ 2 2 also yields the population loss minimizer with respect to µ c conditioned on w = -1 . Thus, if condition (4) is satisfied, then we know there exists some α ∈ [0 , 1] such that the optimal poisoning performance can be achieved with µ p = ν α . This follows from the assumption | γ 1 + γ 2 | ≤ 2( u -1) , which suggests that for any ( x, y ) ∼ ν α , the individual hinge loss at ( x, y ) will be zero. In addition, we know that the poisoned hypothesis induced by A opt is h -1 , γ 1 + γ 2 2 , which maximizes risk with respect to µ c .

On the other hand, if condition (4) is not satisfied, we know that the poisoned hypothesis induced by any feasible µ p has weight parameter w = 1 . Based on our second observation, this further suggests that the optimal poisoning performance will always be achieved with either µ p = ν 0 or µ p = ν 1 . According to the first-order optimality condition and Lemma C.1, we can compute the closed-form solution regarding the optimal poisoning performance. Thus, we complete the proof.

B.2 Proof of Theorem 5.7

Proof of Theorem 5.7. Consider linear hypothesis class H and the poisoned distribution µ ∗ p generated by the optimal poisoning adversary A opt in Definition 4.1. Given clean distribution µ c , poisoning ratio ϵ and constraint set C , the inherent vulnerability to indiscriminate poisoning is captured by the optimal attack performance Risk ( h ∗ p ; µ c ) , where h ∗ p denotes the poisoned linear model induced by µ ∗ p . For any h ∈ H , we have

where the first inequality holds because the surrogate loss is defined to be not smaller than the 0-1 loss, the second inequality holds because the surrogate loss is always non-negative, and the third inequality holds because h ∗ p minimizes the population loss with respect to both clean distribution

µ and optimally generated poisoned distribution µ ∗ p . Consider h c = argmin h ∈H L ( h ; µ c ) (with weight parameter w c and bias parameter b c ), which is the linear model learned from the clean data. Therefore, plugging h = h c into the right hand side of (11), we further obtain

where the last inequality holds because for any poisoned data point ( x , y ) ∼ µ ∗ p , the surrogate loss at ( x , y ) with respect to h c is ℓ M ( y · ( w ⊤ c x + b c ) ) , and y · ( w ⊤ c x + b c ) ≤ max ( x ,y ) ∈C | w ⊤ c x + b c | . Under the condition that min ( x ,y ) ∈C w ⊤ c x ≤ -b c ≤ max ( x ,y ) ∈C w ⊤ c x which means the decision boundary of h c falls into the constraint set C when projected on to the direction of w c , we further have max ( x ,y ) ∈C | w ⊤ c x + b c | ≤ Size w c ( C ) , which implies the validity of (12). We remark that the condition min ( x ,y ) ∈C w ⊤ c x ≤ -b c ≤ max ( x ,y ) ∈C w ⊤ c x typically holds for margin-based loss in practice, since the support of the clean training data belongs to the constraint set for poisoning inputs (for either undefended victim models or models that employ some unsupervised data sanitization defense). Therefore, we leave this condition out in the statement of Theorem 5.7 for simplicity.

C Proofs of Technical Lemmas used in Appendix B.1

C.1 Proof of Lemma B.1

To prove Lemma B.1, we need to make use of the following general lemma which characterizes the population hinge loss and its derivative with respect to clean data distribution µ c . For the sake of completeness, we provide the proof of Lemma C.1 in Appendix C.4.

Lemma C.1. Let µ c be data distribution generated according to (2) . For any h w,b ∈ H L , the population hinge loss is:

and its gradient with respect to b is:

where φ and Φ denote the PDF and CDF of standard Gaussian distribution N (0 , 1) , respectively.

Next, let us summarize several key observations based on Lemma C.1 (specifically for the setting considered in Lemma B.1). For any w ∈ {-1 , 1 } , ∂ ∂b L ( h w,b ; µ c ) is a monotonically increasing with b , which achieves minimum -1 2 when b goes to -∞ and achieves maximum 1 2 when b goes to ∞ . If w = +1 , then L ( h w,b ; µ c ) is monotonically decreasing when b ∈ ( -∞ , -γ 1 + γ 2 2 ) and monotonically increasing when b ∈ ( -γ 1 + γ 2 2 , ∞ ) , reaching the minimum at b = b ∗ c (1) := -γ 1 + γ 2 2 . On the other hand, if w = -1 , then L ( h w,b ; µ c ) is monotonically decreasing when b ∈ ( -∞ , γ 1 + γ 2 2 ) and monotonically increasing when b ∈ ( γ 1 + γ 2 2 , ∞ ) , reaching the minimum at b = b ∗ c ( -1) := γ 1 + γ 2 2 .

As for the clean loss minimizer conditioned on w = 1 , we have

whereas as for the clean loss minimizer conditioned on w = -1 , we have

Let f ( t ) = t · Φ( t σ ) + σ √ 2 π · exp( -t 2 2 σ 2 ) , we know L ( h 1 ,b ∗ c (1) ; µ c ) = f ( γ 1 -γ 2 +2 2 ) and L ( h -1 ,b ∗ c ( -1) ; µ c ) = f ( γ 2 -γ 1 +2 2 ) . We can compute the derivative of f ( t ) : f ′ ( t ) = Φ( t σ ) ≥ 0 , which suggests that L ( h 1 ,b ∗ c (1) ; µ c ) ≤ L ( h -1 ,b ∗ c ( -1) ; µ c ) .

Now we are ready to prove Lemma B.1.

Proof of Lemma B.1. First, we prove the following claim: for any possible b p , linear hypothesis h 1 ,b p can always be achieved by minimizing the population hinge loss with respect to µ c and µ p = ν α with some carefully-chosen α ∈ [0 , 1] based on b p .

For any µ p ∈ Q ( u ) , according to the first-order optimality condition with respect to b p , we have

where the last inequality follows from ∂ ∂b ℓ ( h w,b ; x, y ) ∈ [ -1 , 1] for any ( x, y ) . Let B p be the set of any possible bias parameters b p . According to (13), we have

Let b ∗ c (1) = argmin b ∈ R L ( h 1 ,b ; µ c ) be the clean loss minimizer conditioned on w = 1 . According to Lemma C.1 and the assumption | γ 1 + γ 2 | ≤ 2 u , we know b ∗ c (1) = γ 1 + γ 2 2 ∈ [ -u, u ] . For any b p ∈ B p , we can always choose

where ν α is defined according to (3). This follows from the first-order optimality condition for convex function and the closed-form solution for the derivative of hinge loss with respect to ν α :

Thus, we have proven the claimed presented at the beginning of the proof of Lemma B.1.

Next, we show that for any b p ∈ B p , among all the possible choices of poisoned distribution µ p that induces b p , choosing µ p = ν α with α defined according to (14) is the optimal choice in terms of the maximization objective in (10). Let µ p ∈ Q ( u ) be any poisoned distribution that satisfies the following condition:

According to the aforementioned analysis, we know that by setting α according to (14), ν α also yields b p . Namely,

Since the population losses with respect to µ c are the same at the induced bias b = b p , it remains to prove ν α achieves a larger population loss with respect to the poisoned distribution than that of µ p , i.e., L ( h 1 ,b p ; ν α ) ≥ L ( h 1 ,b p ; µ p ) .

such that

Consider the following two probabilities with respect to b p and µ p :

Note that the derivative of hinge loss with respect to the bias parameter is ∂ ∂b ℓ ( h w,b ; x, y ) ∈ {-1 , 0 , 1 } , thus we have

Moreover, according to the first-order optimality of b p with respect to µ p , we have

If we measure the sum of the probability of input having negative gradient and half of the probability of having zero gradient, we have:

Therefore, we can construct a mapping g that maps µ p to ν α : by moving any ( x, y ) ∼ µ p that contributes p 1 (negative derivative) and any ( x, y ) ∼ µ p that contributes p 2 (positive derivative) to extreme locations ( -u, +1) and ( u, -1) , respectively, and move the remaining ( x, y ) that has zero derivative to ( -u, +1) and ( u, -1) with equal probabilities (i.e., 1 -p 1 -p 2 2 ), and we can easily verify that the gradient of b p with respect to µ p is the same as ν α .

In addition, note that hinge loss is monotonically increasing with respect to the ℓ 2 distance of misclassified examples to the decision hyperplane, and the initial clean loss minimizer b ∗ c (1) ∈ [ -u, u ] , we can verify that the constructed mapping g will not reduce the individual hinge loss. Namely, ℓ ( h 1 ,b p ; x, y ) ≤ ℓ ( h 1 ,b p ; g ( x, y )) holds for any ( x, y ) ∼ µ p . Therefore, we have proven Lemma B.1.

C.2 Proof of Lemma B.2

Proof of Lemma B.2. First, we introduce the following notations. For any µ p ∈ Q ( u ) and any w ∈ {-1 , 1 } , let

According to Lemma B.1, we know that the maximum population hinge loss conditioned on w = 1 is achieved when µ p = ν α for some α ∈ [0 , 1] . To prove the sufficient and necessary condition specified in Lemma B.2, we also need to consider w = -1 . Note that different from w = 1 , we want to specify the minimum loss that can be achieved with some µ p for w = -1 . For any µ p ∈ Q ( u ) , we have

According to Lemma C.1, we know b ∗ c ( -1) = γ 1 + γ 2 2 , which achieves the minimum clean loss conditioned on w = -1 . Since we assume γ 1 + γ 2 2 ∈ [ -u +1 , u -1] , according to the first-order optimality condition, the equality in (15) can be attained as long as µ p only consists of correctly classified data that also incurs zero hinge loss with respect to b ∗ c ( -1) (not all correctly classified instances incur zero hinge loss). It can be easily checked that choosing µ p = ν α based on (3) with any α ∈ [0 , 1] satisfies this condition, which suggests that as long as the poisoned distribution µ p is given in the form of ν α and if the w = -1 is achievable (conditions on when this can be achieved will be discussed shortly), then the bias term that minimizes the distributional loss is equal to b ∗ c ( -1) , and is the minimum compared to other choices of b p ( -1; µ p ) . According to Lemma B.1, it further implies the following statement: there exists some α ∈ [0 , 1] such that

For simplicity, let us denote by ∆ L ( µ p ; ϵ, u, µ c ) the maximization objective regarding the population loss difference between w = 1 and w = -1 . Thus, a necessary and sufficient condition such that there exists a h -1 ,b p ( -1; µ p ) as the loss minimizer is that max α ∈ [0 , 1] ∆ L ( ν α ; ϵ, u, µ c ) ≥ 0 . This requires us to characterize the maximal value of loss difference for any possible configurations of ϵ, u and µ c . According to Lemma C.1 and the definition of ν α , for any α ∈ [0 , 1] , we denote the above loss difference as

The second term I 2 is fixed (and the loss on ν α is zero conditioned on w = -1 ), thus it remains to characterize the maximum value of I 1 with respect to α for different configurations. Consider auxiliary function

We know g ( b ) ∈ [ -1 2 , 1 2 ] is a monotonically increasing function by checking with derivative to b . Let g -1 be the inverse function of g . Note that according to Lemma C.1 and the first-order optimality condition of b p (1; ν α ) , we have

where the first equality follows from Lemma C.1, the second equality follows from the first-order optimality condition and the last equality is based on the definition of ν α . This suggests that b p (1; ν α ) = g -1 ( ϵ · (2 α -1) ) for any α ∈ [0 , 1] .

Consider the following two configurations for the term I 1 : 0 ̸∈ [ g -1 ( -ϵ ) , g -1 ( ϵ )] and 0 ∈ [ g -1 ( -ϵ ) , g -1 ( ϵ )] . Consider the first configuration, which is also equivalent to g (0) / ∈ [ -ϵ, ϵ ] . We can prove that if γ 1 + γ 2 < 0 meaning that b ∗ c (1) > 0 , choosing α = 0 achieves the maximal value of I 1 ; whereas if γ 1 + γ 2 > 0 , choosing α = 1 achieves the maximum. Note that it is not possible for γ 1 + γ 2 = 0 under this scenario. The proof is straightforward, since we have

where t = g -1 (2 ϵα -ϵ ) ∈ [ g -1 ( ϵ ) , g -1 ( ϵ )] . In addition, we can compute the derivative of I 1 with respect to t :

which suggests that I 1 is a concave function with respect to t . If 0 ∈ [ g -1 ( -ϵ ) , g -1 ( ϵ )] , we achieve the global maximum of I 1 at t = 0 by carefully picking α 0 = 1 2 + 1 2 ϵ · g (0) . If not (i.e., g -1 ( -ϵ ) > 0 or g -1 ( ϵ ) < 0 ), then we pick t that is closer to 0 , which is either g ( -ϵ ) or g ( ϵ ) by setting α = 0 or α = 1 respectively. Therefore, we can specify the sufficient and necessary conditions when the weight vector w can be flipped from 1 to -1 :

When g (0) ̸∈ [ -ϵ, ϵ ] , the condition is

When g (0) ∈ [ -ϵ, ϵ ] , the condition is

Plugging in the definition of g and ∆ L , we complete the proof of Lemma B.2.

C.3 Proof of Lemma B.3

Proof of Lemma B.3. Let µ 1 , µ 2 be the probability measures of the positive and negative examples assumed in (2), respectively. Let φ ( z ; γ, σ ) be the PDF of Gaussian distribution N ( γ, σ 2 ) . For simplicity, we simply write φ ( z ) = φ ( z ; 0 , 1) for standard Gaussian. For any h w,b ∈ H L , we know

w can be either 1 or -1 . First, let's consider the case where w = 1 . According to the definition of risk and the data generating process of µ c , we have

Similarly, when w = -1 , we have

Combining the two cases, we complete the proof.

C.4 Proof of Lemma C.1

Proof of Lemma C.1. We use similar notations such as µ 1 , µ 2 and φ as in Lemma B.3. For any h w,b ∈ H L with w = 1 , then according to the definition of population hinge loss, we have

Taking the derivative with respect to parameter b and using simple algebra, we have

Similarly, for any h w,b ∈ H L with w = -1 , we have

D Additional Experimental Results and Details

In this section, we provide details on our experimental setup (Appendix D.1) and then provide additional results (Appendix D.2).

D.1 Details on Experimental Setup in Section 3

Details on datasets and training configurations. In the main paper, we used different public benchmark datasets including MNIST [27] digit pairs (i.e., 1-7. 6-9, 4-9) and also the Enron dataset, which is created by Metsis et al. [36], Dogfish [24] and Adult [14], which are all used in the evaluations of prior works except MNIST 6-9 and MNIST 4-9. In the appendix, we additionally present the results of the IMDB dataset [33], which has also been used in prior evaluations [24, 25]. We did not include the IMDB results in the main paper because we could not run the existing state-of-the-art poisoning attacks on IMDB because the computation time is extremely slow. Instead, we directly quote the poisoned error of SVM from Koh et al. [25] and then present the computed metrics. For the Dogfish and Enron dataset, we construct the constraint set C in the no defense setting by finding the minimum ( u i min ) and maximum ( u i max ) values occurred in each feature dimension i for both the training and test data, which then forms a box constraint [ u i min , u i max ] for each dimension. This way of construction is also used in the prior work [25]. Because we consider linear models, the training [38] of linear models and the attacks on them are stable (i.e., less randomness involved in the process) and so, we get almost identical results when feeding different random seeds. Therefore, we did not report error bars in the results. The regularization parameter λ for training the linear models (SVM and LR) are configured as follows: λ = 0 . 09 for MNIST digit pairs, Adult, Dogfish, SVM for Enron; λ = 0 . 01 for IMDB, LR for Enron. Overall, the results and conclusions in this paper are insensitive to the choice of λ . The computation of the metrics in this paper are extremely fast and can be done on any laptop. The poisoning attacks can also be done on a laptop, except the Influence Attack [25], whose computation can be accelerated using GPUs.

Attack details. The KKT, MTA and Min-Max attacks evaluated in Section 3 require a target model as input. This target model is typically generated using some label-flipping heuristics [25, 46]. In practice, these attacks are first run on a set of carefully-chosen target models, then the best attack performance achieved by these target models is reported. We generate target models using the improved procedure described in Suya et al. [46] because their method is able to generate better target models that induce victim models with a higher test error compared with the method proposed in Koh et al. [25]. We generate target models with different error rates, ranging from 5% to 70% using the label-flipping heuristics, and then pick the best performing attack induced by these target models.

Following the prior practice [25], we consider adversaries that have access to both the clean training and test data, and therefore, adversaries can design attacks that can perform better on the test data. This generally holds true for the Enron and MNIST 1-7 datasets, but for Dogfish, we find in our experiments that the attack 'overfits' to the test data heavily due to the small number of training and test data and also the high dimensionality. More specifically, we find that the poisoned model tends to incur significantly higher error rates on the clean test data compared to the clean training data. Since this high error cannot fully reflect the distributional risk, when we report the results in Section 3 we report the errors on both the training and the testing data to give a better empirical sense of what the distributional risk may look like. This also emphasizes the need to be cautious on the potential for 'overfitting' behavior when designing poisoning attacks. Table 2 shows the drastic differences between the errors of the clean training and test data after poisoning.

D.2 Supplemental Results

In this section, we provide additional results results that did not fit into the main paper, but further support the observations and claims made in the main paper. We first show the results of IMDB and the metrics computed on the whole clean test data in Appendix D.2.2 to complement Table 1 in the main paper, then include the complete results of the impact of data sanitization defenses in Appendix D.2.3 to complement the last paragraph in Section 7. Next, we provide the metrics computed on selective benchmark datasets using a different projection vector from the clean model weight in Appendix D.2.4 to support the results in Table 1 in the main paper. Lastly, we show the performance of different poisoning attacks at various poisoning ratios in Appendix D.2.5, complementing Figure 1 in the main paper.

D.2.1 IMDB results

Table 1 in the main paper presents the metrics that are computed on the correctly classified test samples by the clean model w c . In Table 3, we additionally include the IMDB results to the Table 1 in the main paper. From the table, we can see that IMDB is still highly vulnerable to poisoning because its separability is low compared to datasets that are moderately vulnerable or robust, and impacted the most by the poisoning points compared to all other benchmark datasets. Note that, the increased error from IMDB is directly quoted from Koh et al. [25], which considers data sanitization

defenses. Therefore, we expect the attack effectiveness might be further improved when we do not consider any defenses, as in our paper.

D.2.2 Metrics computed using all test data

Table 4 shows the results when the metrics are computed on the full test data set (including misclassified ones), rather than just on examples that were classified correctly by the clean model. The metrics are mostly similar to Table 3 when the initial errors are not high. For datasets with high initial error such as Adult, the computed metrics are more aligned with the final poisoned error, not the error increase.

D.2.3 Explaining the impact of data sanitization defenses

We then provide additional results for explaining the effectiveness of the data sanitization defenses in improving dataset robustness, which is discussed in section 7. On top of the Enron result shown in the paper, which is attacked at 3% poisoning ratio, we also provide attack results of MNIST 1-7 dataset. We report the attack results when ϵ = 10% and we considered a significantly higher poisoning ratio because at the original 3% poisoning, the dataset can well resist existing attacks and hence there is no point in protecting the dataset with sanitization defenses. This attack setting is just for illustration purpose and attackers in practice may be able to manipulate such a high number of poisoning points. Following the main result in the paper, we still compute the metrics based on the correctly classified samples in the clean test set, so as to better depict the relationship between the increased errors and the computed metrics. The results are summarized in Table 5 and we can see that existing data sanitization defenses improve the robustness to poisoning by majorly limiting Size w c ( C ) . For MNIST 1-7, equipping with data sanitization defense will make the dataset even more robust (robust even at the high 10% poisoning rate), which is consistent with the findings in prior work [45].

D.2.4 Using different projection vectors

In the main paper, we used the weight of the clean model as the projection vector and found that the computed metrics are highly correlated with the empirical attack effectiveness observed for different benchmark datasets. However, there can also be other projection vectors that can be used for explaining the different vulnerabilities, as mentioned in Remark 5.8.

We conducted experiments that use the projection vector that minimizes the upper bound on optimal poisoning attacks, given in Equation 5. The upper-bound minimization corresponds to a min-max optimization problem. We solve it using the online gradient descent algorithm (alternatively updating the poisoning points and model weight), adopting an approach similar to the one used by Koh et al. for the i-Min-Max attack [25]. We run the min-max optimization for 30,000 iterations with learning rate of 0.03 for the weight vector update, and pick the weight vector that results in the lowest upper bound in Equation 5.

The results on two of the benchmark datasets, MNIST 1-7 and Dogfish, are summarized in Table 6. From the table, we can see that, compared to the clean model, the new projection vector reduces the projected constraint size (increases Sep/Size), which probably indicates the weight vector obtained from minimizing the upper bound focuses more on minimizing the term ℓ M (Size w c ( C )) in Equation 5. Nevertheless, projecting onto the new weight vector can still well explain the difference between the benchmark datasets.

Using the obtained vector w U (denote the corresponding hypothesis as h U ) above, we can also compute an (approximate) upper bound on the test error of the optimal indiscriminate poisoning attack, which is proposed by Steinhardt et al. [45] and is shown in (17). The upper bound is an approximate upper bound to the test error because max S p ⊆C L ( ˆ h p ; S c ) only upper bounds the maximum training error (of poisoned model ˆ h p ) on S c from poisoning and it is assumed that, with proper regularization, the training and test errors are roughly the same. We can view h U as the numerical approximation of the hypothesis that leads to the smallest upper bound on the (approximate) test error. Next, in Table 7, we compare the computed (approximate) upper bound and the lower bound on poisoning effectiveness achieved by best known attacks, and we can easily observe that, although the gap between the lower and upper bounds are relatively large (all datasets are vulnerable at 3% poisoning ratio when we directly check the upper bounds), the upper bounds still vary between datasets. We also speculate that the actual performance of the best attacks may not be as high as indicated by the upper bound, because when computing the upper bound, we used the surrogate loss to upper bound the 0-1 loss (exists some non-negligible gap) and maximized the surrogate loss in the poisoning setting, which potentially introduces an even larger gap between the surrogate loss and the 0-1 loss. Hence, we motivate future research to shrink the possibly large gap.

Filtered Enron

MNIST6-9

Dogfish

MNIST1-7

Enron

MNIST4-9

Adult

0

10

20

30

Test Error (%)

Trained on

c

Trained on

c

p

(a) MNIST 1-7

(b) Dogfish

(c) Enron

D.2.5 Performance of different attacks are similar

In this section, we first present the best attack results on different benchmark datasets on the LR model in Figure 3. Similar to the case of linear SVM as shown in Figure 1, the performance of best known attacks vary drastically across the benchmark datasets. Although we only report the performance of the best known attack from the candidate state-of-the-art poisoning attacks, the difference among these canndidate attacks do not vary much in many settings. In Figure 4 we show the attack performance of different attacks on the selected benchmark datasets of MNIST 1-7, Dogfish, and Enron and the main observation is that different attacks perform mostly similarly for a given dataset (but their performance varies a lot across datasets). Also, from the attack results on Enron (Figure 4c), we can see that several of the attacks perform worse at higher poisoning ratios. Although there is a chance that the attack performance can be improved by careful hyperparameter tuning, it also suggests that these attacks are suboptimal. Optimal poisoning attacks should never get less effective as the poisoning ratio increases, according to Theorem 4.5.

D.2.6 Possible causal relationship of identified factors to poisoning vulnerability

In this section, we describe our preliminary experiment on showing the possible causal relationship from the identified two factors (i.e., Sep/SD and Sep/Size) to the empirically observed vulnerability against poisoning attacks. Given a dataset, we aim to modify it such that we can gradually change one factor of the dataset while (ideally) keeping the other one fixed, and then measure the changes in its empirical poisoning vulnerability. To cause changes to the factor of 'Sep/SD", we need to modify the original data points in the dataset. This modification may make it hard to keep the factor of 'Sep/Size" (or 'Size") fixed, as the computation of 'Size" depends on the clean boundary w c , which is again dependent on the (modified) clean distribution. Therefore, even if we use the same constraint set C for different modified datasets, the 'Size" or 'Sep/Size" can be different. We conducted preliminary experiments on MNIST 1-7 and the details are given below.

Change Sep/Size. , In order to measure the impact of Sep/Size without impacting Sep/SD, we can choose C that covers even larger space so that the resulting constraint set will not filter out any clean points from the original training data, which then implies both the w c and the corresponding metric of Sep/SD will not change. We performed a hypothetical experiment on MNIST 1-7 where we scale the original box-constraint from [0 , 1] to [0 , c ] , c > = 1 , where c is the scale factor and c = 1 denotes the original setting for MNIST 1-7 experiment. The experiments are hypothetical because, in practice, a box-constraint of [0 , c ] with c > 1 will result in invalid images. We experimented with c = 2 , 3 , 5 , 7 , 10 and the results are summarized in Table 8. From the table, we see that as c becomes larger, the Size increases and correspondingly the Sep/Size decreases. These changes are associated with a gradual increase in the 'Error Increase" from poisoning.

Change Sep/SD. As mentioned above, in order to change Sep/SD, we need to modify the original data points, and this modification makes it hard for us to keep Sep/Size fixed for MNIST 1-7 even when using the same C . We leave the exploration of better ways to change data distributions with the Sep/Size fixed as future work. To modify the original data points, we first train a linear SVM on the original MNIST 1-7 dataset. Then, we pick 3% of all (training + test data) correctly classified points that are closest to the clean boundary. These points, once repeated x times, replace x × 3% of correctly classified training and testing data points that are furthest from the boundary. This process can be viewed as oversampling in the near-boundary region and undersampling in the region far from the boundary, and aim to create a dataset with a low separability ratio. We tested with x = 1 , 2 , 4 , 6 , 8 , 10 . We only deal with correctly classified points because we want to ensure the modified dataset still has high clean accuracy. The results are summarized in Table 9 and we can see that as we increase x , the Sep/SD gradually decreases, and the attack effectiveness (measured by the

3.2

3.4

3.6

3.8

Sep/SD

0.06

0.08

0.10

0.12

0.14

0.16

0.18

0.20

Sep/Size

LeNet

R-1

R-25

R-50

R-75

R-90

R-100

R-120

0.0

2.5

5.0

7.5

10.0

12.5

15.0

17.5

20.0

22.5

Error Increase (%)

error increase) increases. Interestingly, the Sep/Size also decreases even though it is computed with the same C . Although the preliminary results above are not sufficient to prove a causation claim, they still show the possibility with the empirical evidence on the MNIST 1-7 dataset.

D.2.7 Evaluating feature extractors with downstream data removed

The experiments in Figure 2 uses the entire CIFAR10 training data to train the pretrained feature extractor, which also includes the data of class 'Truck" and 'Ship" that are used in the downstream training and analysis. Although our original experiment aims to show how the victim can gain improved resistance to poisoning by using better feature extractors, for the particular experiment in Figure 2, a more realistic setting is to exclude the downstream training data used to train the (upstream) pretrained feature extractors. Therefore, we repeated the same experiment, but now removed the data points from the classes of "Ship" and "Truck" from the upstream training data and then train the (pretrained) models. The results are summarized in Figure 5 and the overall finding is similar to the one in Figure 2 that when we adopt a deep architecture and train it for more epochs, the quality of the obtained feature extractor increases and so, it appears to improve the resistance of downstream classifiers to poisoning attacks. The increased quality of the feature extractor is in general correlated to the increased Sep/SD and Sep/Size. To quantitatively measure the correlation of the error increase and each of the two factors, we show the Pearson correlation coefficient of the relevant metrics and the results are summarized in Table 10. From the table, we can see that when we directly measure the Pearson correlation with each individual factor, the correlation between the two factors and error increase is still very strong as the lowest (absolute) correlation coefficient of -0.66 still indicates a strong negative correlation. And if we further restrict ourselves to the ResNet-18 models only, then the (absolute) correlation coefficient is even higher, which again confirms the two factors are strongly (and negatively) correlated with the error increase.

In Table 10, we also compare the new experiment with the original experiment in Figure 2 and observe that the (absolute) Pearson correlation is reduced slightly, which is due to the fact that when the pretrained model does not involve any downstream data, the two factors may not consistently change in the same direction and undermines the Pearson correlation of each individual factor with

the error increase. However, if we compute the Pearson correlation between the number of epochs for training the ResNet-18 model (roughly incorporates the combined effects of both factors) and the error increase from poisoning, we observe an even stronger negative correlation (-0.99) compared to the original experiment (-0.95), which again supports the claim that training the deep architecture for more epochs seems to be one possible approach to improve resistance to poisoning. The slight increase in correlation in the new experiment results from the slightly higher covariance between the error increase and the number of epochs, but this difference is not significant since we only consider a limited number of samples when computing the correlation.

Developing general methods for learning meaningful feature extractors such that the two factors can be improved simultaneously in a controlled way would be an interesting research direction, which we plan to pursue in the future. However, we also note that this is a challenging task from the practical perspective since the goal is to extract useful feature representations for downstream applications, so the base error should also be considered in addition to the error increase from poisoning, especially when no downstream data are used in the upstream training. Our initial thought is to consider an intermediate few-shot learning setup [52], where a small number of (clean) downstream training data are allowed to be incorporated in training the feature extractor. This few-shot learning setup might correspond to an intermediate setting between the original experiment with full knowledge of the (clean) downstream training data and the new experiment with almost zero knowledge but still reflects certain real-world application scenarios. To conclude, all results in both the original and new experiments support our main claim that it is possible for the victim to choose better feature extractors to improve the resistance of downstream classifiers to indiscriminate poisoning attacks.

E Comparison to LDC and Aggregation Defenses

We first provide a more thorough discussion on the differences between our work and the Lethal Dose Conjecture (LDC) [50] from NeurIPS 2022, which had similar goals in understanding the inherent vulnerabilities of datasets but focused on targeted poisoning attacks (Appendix E.1). Then, we discuss how our results can also be related to aggregation based defenses whose asymptotic optimality on robustness against targeted poisoning attacks is implied by the LDC conjecture (Appendix E.2).

E.1 Relation to LDC

As discussed in Section 1, LDC is a more general result and covers broader poisoning attack goals (including indiscriminate poisoning) and is agnostic to the learning algorithm, dataset and also the poisoning generation setup. However, this general result may give overly pessimistic estimates on the power of optimal injection-only poisoning attacks in the indiscriminate setting we consider. We first briefly mention the main conjecture in LDC and then explain why the LDC conjecture overestimated the power of indiscriminate poisoning attacks, followed by a discussion on the relations of the identified vulnerability factors in this paper and the key quantity in LDC.

The main conjecture in LDC. LDC conjectures that, given a (potentially poisoned) dataset of size N , the tolerable sample size for targeted poisoning attacks (through insertion and/or deletion) by any defenses and learners, while still predicting a known test sample correctly, is an asymptotic guarantee of Θ( N/n ) , where n < N is the sample complexity of the most data-efficient learner (i.e., a learner that uses smallest number of clean training samples to make correct prediction). Although it is a conjecture on the asymptotic robustness guarantee, it is rigorously proven for cases of bijection uncovering and instance memorization, and the general implication of LDC is leveraged to improve existing aggregation based certified defenses against targeted poisoning attacks.

Overestimating the power of indiscriminate poisoning. LDC conjectures the robustness against targeted poisoning attacks, but the same conjecture can also be used in indiscriminate setting straightforwardly by taking the expectation over the tolerable samples for each of the test samples to get the expected tolerable poisoning size for the entire distribution (as mentioned in the original LDC paper) or by choosing the lowest value to give a worst case certified robustness for the entire distribution. The underlying assumption in the reasoning above is that individual samples are independently impacted by their corresponding poisoning points while in the indiscriminate setting, the entire distribution is impacted simultaneously by the same poisoning set. The assumption on the independence of the poisoning sets corresponding to different test samples might overestimate the power of indiscriminate

poisoning attacks as it might be impossible to simultaneously impact different test samples (e.g., test samples with disjoint poisoning sets) using the same poisoning set. In addition, the poisoning generation setup also greatly impacts the attack effectiveness-injection only attacks can be much weaker than attacks that modify existing points, but LDC provides guarantees against this worst case poisoning generation of modifying points. These general and worst-case assumptions mean that LDC might overestimate the power of injection-only indiscriminate poisoning attacks considered in this paper.

In practice, insights from LDC can be used to enhance existing aggregation based defenses. If we treat the fraction of (independently) certifiable test samples by the enhanced DPA [29] in Figure 2(d) (using k = 500 partitions) in the LDC paper as the certified accuracy (CA) for the entire test set in the indiscriminate setting, the CA against indiscriminate poisoning attack is 0% at the poisoning ratio of ϵ = 0 . 5% (250 / 50000) . In contrast, the best indiscriminate poisoning attacks [32] on CIFAR10 dataset reduces the model accuracy from 95% to 81% at the much higher ϵ = 3% poisoning ratio using standard training (i.e., using k = 1 partition). Note that using k = 1 partition is a far less optimal choice than k = 500 as k = 1 will always result in 0% CA for aggregation based defenses. Our work explicitly considers injection only indiscriminate poisoning attacks so as to better understand its effectiveness.

While it is possible that current indiscriminate attacks for neural networks are far from optimal and there may exist a very strong (but currently unknown) poisoning attack that can reduce the neural network accuracy on CIFAR10 to 0% at a 0.5% poisoning ratio, we speculate such likelihood might be low. This is because, neural networks are found to be harder to poison than linear models [31, 32] while our empirical findings in the most extensively studied linear models in Section 3 indicate some datasets might be inherently more robust to poisoning attacks.

Providing finer analysis on the vulnerability factors. As mentioned above, LDC might overestimate the power of indiscriminate poisoning attacks. In addition, the key quantity n is usually unknown and hard to estimate accurately in practice and the robustness guarantee is asymptotic while the constants in asymptotic guarantees can make a big difference in practice. However, the generic metric n still offers critical insights in understanding the robustness against indiscriminate poisoning attacks. In particular, our findings on projected separability and standard deviation can be interpreted as the first step towards understanding the dataset properties that can be related to the (more general) metric n (and maybe also the constant in Θ(1 /n ) ) in LDC for linear learners. Indeed, it is an interesting future work to identify the learning task properties that impact n at the finer-granularity.

As for the projected constraint size (Definition 5.5), we believe there can be situations where it may be independent from n . The main idea is that in cases where changing C arbitrarily will not impact the clean distribution (e.g., when the support set of the clean distribution is a strict subset of C , arbitrarily enlarging C will still not impact the clean distribution), the outcomes of learners trained on clean samples from the distribution will not change (including the most data-efficient learner) and hence n will remain the same for different permissible choices of C , indicating that the vulnerability of the same dataset remains the same even when C changes drastically without impacting the clean distribution. However, changes in C (and subsequently changes in the projected constraint size) will directly impact the attack effectiveness, as a larger C is likely to admit stronger poisoning attacks.

To illustrate how much the attack power can change as C changes, we conduct experiments on MNIST 1-7 and show that scaling up the original dimension-wise box-constraint from [0 , 1] to [0 , c ] (where c > 1 is the scale factor) can significantly boost attack effectiveness. Table 11 summarizes the results and we can observe that, as the scale factor c increases (enlarged C , increased projected constraint size and reduced Sep/Size), the attack effectiveness also increases significantly. Note that this experiment is an existence proof and MNIST 1-7 is used as a hypothetical example. In practice, for normalized images, the box constraint cannot be scaled beyond [0,1] as it will result in invalid images.

E.2 Relation to Aggregation-based Defenses

Aggregation-based (provable) defenses, whose asymptotic optimality is implied by the LDC, work by partitioning the potentially poisoned data into k partitions, training a base learner on each partition and using majority voting to obtain the final predictions. These defenses provide certified robustness to poisoning attacks by giving the maximum number of poisoning points that can be tolerated to

correctly predict a known test point, which can also be straightforwardly applied to indiscriminate setting by treating different test samples independently, as mentioned in the discussion of LDC.

Because no data filtering is used for each partition of the defenses, at the partition level, our results (i.e., the computed metrics) obtained in each poisoned partition may still be similar to the results obtained on the whole data without partition (i.e., standard training, as in this paper), as the clean data and C (reflected through the poisoning points assigned in each partition) may be more or less the same. At the aggregation level, similar to the discussion on LDC, these aggregation defenses may still result in overly pessimistic estimates on the effectiveness of injection only indiscriminate poisoning attacks as the certified accuracy at a particular poisoning ratio can be very loose, and the two possible reasons are: 1) straightforwardly applying results in targeted poisoning to indiscriminate poisoning might lead to overestimation and 2) considering the worst case adversary of modifying points might overestimate the power of injection only attacks in each poisoned partition. Therefore, our work can be related to aggregation defenses via reason 2), as it might be interpreted as the first step towards identifying factors that impact the attack effectiveness of injection only indiscriminate attacks in each poisoned partition, which may not always be highly detrimental depending on the learning task properties in each partition, while these aggregation defenses assume the existence of a single poisoning point in a partition can make the model in that partition successfully poisoned.

Loose certified accuracy in indiscriminate setting. Given a poisoning budget ϵ , aggregation based defenses give a certified accuracy against indiscriminate poisoning attacks by first computing the tolerable fraction of poisoning points for each test sample and all the test samples with tolerable fraction smaller than or equal to ϵ are certifiably robust. Then, the fraction of those test samples to the total test samples gives the certified accuracy for the test set. Similar to the result of CIFAR10 shown in LDC, here, we provide an additional result of certified accuracy for neural networks trained on the MNIST dataset: the state-of-the-art finite aggregation (FA) method [51] gives a certified accuracy of 0% at 1% poisoning ratio (600/60,000) using k = 1200 partitions while at the much higher 3% poisoning ratio, the current state-of-the-art indiscriminate poisoning attack [32] can only reduce the accuracy of the neural network trained on MNIST without partitioning (i.e., k = 1 , a far less optimal choice from the perspective of aggregation defenses) from over 99% to only around 90%.

F Extension to Multi-class Settings and Non-linear Learners

In this section, we first provide the high-level idea of extending the metric computation from binary setting to multi-class setting and then provide empirical results on multi-class linear models and show that these metrics can still well-explain the observations in multi-class linear classifiers (Appendix F.1). Then, we provide the idea of extending the metrics from linear models to neural networks (NNs) and also the accompanying experimental results (Appendix F.2). In particular, we find that, for the same learner (e.g., same or similar NN architecture), our metrics may still be able to explain the different dataset vulnerabilities. However, the extended metrics cannot explain the vulnerabilities of datasets that are under different learners (e.g., NN with significantly different architectures), whose investigation is a very interesting future work, but is out of the scope of this paper.

F.1 Extension to Multi-class Linear Learners

Multi-class classifications are very common in practice and therefore, it is important to extend the computation of the metrics from binary classification to multi-class classification. For linear models, k -class classification ( k > 2 ) is handled by treating it as binary classifications in the "one vs one"

mode that results k ( k -1) / 2 binary problems by enumerating over every pair of classes or in the "one vs rest" mode that results in k binary problems by picking one class as the positive class and the rest as the negative class. In practice, the "one vs rest" mode is preferred because it requires training smaller number of classifiers. In addition, the last classification layer of neural networks may also be roughly viewed as performing multi-class classification in 'one vs rest" mode. Therefore, we only discuss and experiment with multi-class linear models trained in 'one vs rest" mode in this section, consistent with the models in prior poisoning attacks [25, 32], but classifiers trained in 'one vs one" mode can also be handled similarly.

Computation of the metrics. Although we consider linear models in 'one vs rest" mode, when computing the metrics, we handle it in a way similar to the 'one vs one" mode - when computing the metrics, given a positive class, we do not treat all the remaining k-1 classes (constitute the negative class) as a whole, instead, for each class in the remaining classes, we treat it as a 'fake" negative class and compute the metrics as in the binary classification setting. Then from the k -1 metrics computed, we pick the positive and 'fake" negative pair with smallest separability metric and use it as the metric for the current positive and negative class (includes all remaining k -1 classes) 2 . Once we compute the metrics for all the k binary pairs, we report the lowest metrics obtained. The reasoning behind computing the metrics in (similar to) 'one vs one" mode is, for a given positive class, adversaries may target the most vulnerable pair from the total k -1 (positive, negative) pairs to cause more damage using the poisoning budget. Therefore, treating the remaining k -1 pairs as a whole when computing the metrics will obfuscate this observation and may not fully reflect the poisoning vulnerabilities of a dataset.

We provide a concrete example on how treating the remaining classes as a whole can lead to wrong estimates on the dataset separability: we first train simple CNN models on the full CIFAR-10 [26] and MNIST datasets and achieve models with test accuracies of ≈ 70% and > 99% respectively. When we feed the MNIST and CIFAR-10 test data through the model and inspect the feature representations, the t-SNE graph indicate that the CIFAR-10 dataset is far less separable than the MNIST, which is expected as CIFAR-10 has much lower test accuracy compared to MNIST. However, when we compute the separability metrics in our paper by considering all k -1 classes in the negative class, the separability of CIFAR-10 is similar to the separability of MNIST, which is inconsistent with drastic differences in the test accuracies of the respective CNN models. In contrast, if we treat each class in the remaining k -1 classes separately and pick the smallest value, we will again see the expected result that CIFAR-10 is far less separable than MNIST. Therefore, for the following experiments, we will compute the metrics by treating the remaining k -1 classes individually. We first provide the results of multi-class linear models for MNIST dataset below and then discuss our initial findings on the neural networks for CIFAR-10 and MNIST in Appendix F.2.

Results on multi-class linear learners. As explained above, when checking the k -binary pairs for a k -class problem, we report the lowest values for the Sep/SD and Sep/Size metrics. However, in some cases, the lowest values for the two metrics might be in two different pairs and in this case, we will report the results of both pairs. Table 12 summarizes the results, where the poisoned errors are directly quoted from the prior works-LR error is from Lu et al. [31] and the SVM error is from Koh et al. [25]. We can see that MNIST dataset is indeed more vulnerable than the selected MNIST 1-7 and MNIST 6-9 pairs because it is less separable and also impacted more by the poisoning points. We also note that the poisoned error of SVM is obtained in the presence of data sanitization defenses

and hence, the poisoned error may be further increased when there are no additional defenses. We also see that, for SVM, although the lowest values for Sep/SD and Sep/Size are in two different pairs, their results do not differ much, indicating that either of them can be used to represent the overall vulnerability of MNIST.

F.2 Extension to Multi-class Neural Networks

We first note that the insights regarding the separability and constraints set C can be general, as the the first metric measures the sensitivity of the dataset against misclassification when the decision boundary is perturbed slightly. The latter captures how much the decision boundary can be moved by the poisoning points once injected into the clean training data. The Sep/SD and Sep/Size metrics used in this paper are the concrete substantiations of the two metrics under linear models. Specific ways to compute the metrics in non-linear settings should still (approximately) reflect the high level idea above. Below, we use neural network (NN) as an example.

High level idea. We may partition the neural network into two parts of feature extractor and linear classification module and we may view the feature representations of the input data as a "new" data in the corresponding feature space, and so that we can convert the metric computations for non-linear neural network into metric computations (on feature space) for linear models. To be more concrete, we propose to use a fixed feature extractor, which can be extractors inherited from pretrained models (e.g., in transfer learning setting) or trained from scratch on the clean data, to map the input data to the feature space. Here, if the victim also uses the same pretrained feature extractor (as in the transfer learning setting), then our metrics can have higher correlation with the poisoned errors from existing attacks because the non-linear feature extractor is now independent from the poisoned points used in the victim's training. Below, we consider the from-scratch training case as it is more challenging.

Computation of the metrics. Although the feature extractor will also be impacted by the poisoning points now, in our preliminary experiment, we will still use the extractor trained on the clean data and leave the exploration of other better feature extractors as future work. Using the transformation from the clean feature extractor, the projected separability and standard deviation can be easily computed. But the computation of the projected constraint size can be tricky, because the set C after transforming through the feature extractor can be non-convex and sometimes, for complicated C , computing such transformation can be very challenging (can be a case even for linear models), but is a very interesting direction to explore. For the simple forms of C such as the dimension-wise box constraints considered in this paper, we may leverage the convex outer polytope method [53] to bound the activation in each layer till the final feature layer so that we can obtain a final transformed convex set C ′ using the feature extractor, which is a set that contains the original C . However, due to time limitation, when computing the projected constraint size in the following experiments, we simply set C as the dimension-wise box-constraints, whose minimum and maximum values are computed from the feature representations of the clean data points, similar to the transfer-learning experiment in Section 7.

Results under similar learners. For the experiments, we use the simple CNN models presented in Lu et al. [31] for MNIST and CIFAR-10 datasets (similar architecture). We directly quote the attack results of TGDA attack by Lu et al. [31] for both the CIFAR-10 and MNIST datasets. Note that, very recently, a stronger GC attack is also proposed by Lu et al. [32] and outperforms the TGDA attack. However, we could not include the newer result because the code is not published and the evaluation in the original paper also did not include the simple CNN for CIFAR-10 dataset. The results are shown in Table 13. From the table, we can see that CIFAR-10 tends to be more vulnerable than MNIST as the Sep/SD and Sep/Size metrics (in neural networks) are all much lower than those of MNIST. These significantly lower values of CIFAR-10 may also suggest that the poisoned error for CIFAR-10 with simple CNN maybe increased further (e.g., using the stronger attack in Lu et al. [32]).

Results under different learners. Above, we only showed results when the MNIST and CIFAR-10 datasets are compared under similar learners. However, in practical applications, one might use deeper architecture for CIFAR-10 and so, we computed the metrics for CIFAR-10 using ResNet18 model. Then we compare the metrics of MNIST under simple CNN and the metrics of CIFAR-10 under ResNet18 in Table 14, where the poisoned errors are quoted from the more recent GC attack [32] because the attack results are all available in the original paper. However, from the table we can see that, although MNIST is less separable and impacted more by the poisoning points, the error increase is still slightly smaller than CIFAR-10, which is not consistent with our metrics. If the current attacks are already performing well and the empirical poisoned errors are indeed good approximations to the inherent vulnerabilities, then we might have to systematically investigate the comparison of vulnerabilities of different datasets under different learners.

G Evaluation on Synthetic Datasets

In this section, we empirically test our theory on synthetic datasets that are sampled from the considered theoretical distributions in Section 5.1.

G.1 Synthetic Datasets

According to Remark 5.4, there are two important factors to be considered: (1) the ratio between class separability and within-class variance | γ 1 -γ 2 | /σ , denoted by β for simplicity; (2) the size of constraint set u . We conduct synthetic experiments in this section to study the effect of these factors on the performance of (optimal) data poisoning attacks.

More specifically, we generate 10,000 training and 10,000 testing data points according to the Gaussian mixture model (2) with negative center γ 1 = -10 and positive center γ 2 = 0 . Throughout our experiments, γ 1 and γ 2 are kept fixed, whereas we vary the variance parameter σ and the value of u . The default value of u is set as 20 if not specified. Evaluations of empirical poisoning attacks require training linear SVM models, where we choose λ = 0 . 01 . The poisoning ratio is still set as 3% , consistent with evaluations on the benchmark datasets.

Impact of β . First, we show how the optimal attack performance changes as we increase the value of β . We report the risk achieved by the OPT attack based on Theorem 5.3. Note that we can only obtain approximations of the inverse function g -1 using numerical methods, which may induce a small approximation error for evaluating the optimal attack performance. For the finite-sample setting, we also report the empirical test error of the poisoned models induced by the empirical OPT attack and the best current poisoning attack discussed in Section 3, where the latter is termed as Best Heuristic for simplicity. Since the poisoned models induced by these empirical attacks do not restrict w ∈ {-1 , 1 } , we normalize w to make the empirical results comparable with our theoretical results.

Figure 6 summarizes the attack performance when we vary β . As the ratio between class separability and within-class variance increases, the risk of the OPT attack and empirical test errors of empirical OPT and best heuristic attacks gradually decrease. This is consistent with our theoretical results discussed in Section 5.1. Note that there exists a minor difference between these attacks when the value of β is small, where the test error attained by the best current heuristic poisoning attack is slightly higher than that achieved by the empirical OPT attack. This is due to the small numerical error induced by approximating the inverse function g -1 .

Impact of u . Our theoretical results assume the setting where w ∈ {-1 , 1 } . However, this restriction makes the impact of the constraint set size u less significant, as it is only helpful in judging whether flipping the sign of w is feasible and becomes irrelevant to the maximum risk after poisoning when flipping is infeasible. In contrast, if w is not restricted, the impact of u will be more significant as larger u tends to reduce the value of w , which in turn makes the original clean data even closer to each other and slight changes in the decision boundary can induce higher risks (further discussions on this are found in Appendix G.2).

To illustrate the impact of u in a continuous way, we allow w to take real numbers. Since this relaxation violates the assumption of our theory, the maximum risk after poisoning can no longer be characterized based on Theorem 5.3. Instead, we use the poisoning attack inspired by our theory to get an empirical lower bound on the maximum risk. Since γ 1 + γ 2 < 0 , Theorem 5.3 suggests that optimal poisoning should place all poisoning points on u with label -1 when w ∈ { 1 , -1 } . We simply use this approach even when w can now take arbitrary values. We vary the value of u gradually and record the test error of the induced hypothesis, We repeat this procedure for different dataset configurations (i.e., fixing γ 1 , γ 2 and varying σ ).

The results are summarized in Figure 7. There are two key observations: (1) once w is no longer constrained, if u is large enough, the vulnerability of all the considered distributions gradually increases as we increase the value of u , and (2) datasets with smaller β are more vulnerable with the increased value of u compared to ones with larger β , which has larger increased test error under the same class separability and box constraint (choosing other values of β also reveals a similar trend). Although not backed by our theory, it makes sense as smaller β also means more points might be closer to the boundary (small margin) and hence small changes in the decision boundary can have significantly increased test errors.

G.2 Relationship Between Box Constraint Size and Model Weight

Our theory assumes that the weight vector of w can only take normalized value from {-1 , 1 } for one-dimensional case, while in practical machine learning applications, convex models are trained by optimizing the hinge loss with respect to both parameters w and b , which can result in w as a real number. And when w takes real numbers, the impact of u becomes smoother: when poisoning with larger u , the poisoning points generated can be very extreme and forces the poisoned model to have reduced w (compared to clean model w c ) in the norm so as to minimize the large loss introduced by the extreme points. Figure 8 plots the relationship between u and w of poisoned model, and supports the statement above. When the norm of w becomes smaller, the original clean data that are well-separated also becomes less separable so that slight movement in the decision boundary can cause significantly increased test errors. This makes the existence of datasets that have large risk gap before and after poisoning more likely, which is demonstrated in Figure 7.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper explores poisoning attacks on linear classifiers.  Specifically, the authors explore why poisoning attacks may have high efficacy on one dataset but low efficacy on another.  The authors relate this phenomenon to two potential factors: (1) (projected) constraint set size and separability. The authors theoretically characterize an optimal poisoning attack w.r.t. Gaussian mixture data.\n\nSTRENGTHS:\nPrevious work has repeatedly shown that poisoning attacks represent a real and serious threat to machine learning models.  Most work in the field focuses on how to design better attacks or better defenses.  An improved theoretical understanding of what mechanisms allow an attack to work could accelerate research for both attacks and defenses.  In my view, this represents a challenging, under-explored research topic. This paper makes an incremental step into improving understanding in this area.\n\nWEAKNESSES:\nI rate the paper as borderline reject and explain my rationale below. I want to make clear that with a thoughtful rebuttal and changes to different aspects, I could definitely see my score increasing.\n\nWhile the paper has many things I like, it is my view that the paper still needs work.  Perhaps the biggest problem is that the writing tends to overclaim results or draw conclusions that, as written, are not well supported.  I discuss a couple of cases below under \"Experimental Results.\" I also ask multiple questions about points I had concerns with in the \"Questions\" section below.\n\n* A more harmless example of what I perceived as overclaiming is on lines 42-43, which state, \"we consider indiscriminate poisoning attacks for linear models, *the most commonly studied victim models in the literature* (emphasis added). It is not my assessment that linear models are the most commonly studied victim model, particularly recently. Reading this sentence, it felt to me that the authors were stretching to over-justify the paper's restriction to linear models. This is despite the subsequent sentences being more than adequate to justify the choice, e.g., \"linear models are still not well understood\" and \"high interpretability.\"\n\n* A second example is on lines 35-36, which state \"the reasons why attacks are sometimes ineffective have not been previously studied.\"  I do not know this statement to be true and expect given other work (e.g., the lethal dose conjecture) that this is not true.\n\n**Experimental Results**\n\nMy biggest concerns are around the experimental results and the conclusions drawn from them. \n\n* Lines 343-344 state, \"Table 1 summarizes the results show that Sep/SD and Sep/Size can *largely* explain why datasets...are harder to poison than others.\" In my view, this statement is making primarily a causation claim. However, the results in the table provide primarily correlation results.  To make a causation claim (which I think is the appropriate type of claim for this paper to try to make), the experiments should externally vary the separability and measure how the performance changes.  I realize that is non-trivial, but I believe it is the right type of experiment here.  \n\n* Lines 390-392 state, \"Although limited in effectiveness, the defense still mitigates the poisoning to some degree *mostly by shrinking the projected constraint size*\" I do not see how the empirical results as presented can support such a strong claim.  As written, the text may barely support the claim that smaller projected constraint size *partially* helps (a weak, not that meaningful claim). To justify *mostly* a lot more is needed.\n\n* Section 7 Better Feature Representation: Lines 363-364 state, \"We train LeNet and ResNet18 on the CIFAR10 dataset...\" From this sentence, it is my understanding that your pretraining was done on all 10 classes.  If so, your feature encoder goes beyond the transfer learning setting, and I believe this approach does *not* correspond with \"the practical scenario where the victim has access to some *small number* of clean samples...\"\n\n* On lines 360-361, we present two ideas -- using better features might improve separability and using data filtering might reduce the projected constraint size.\" I would expect data filtering would both improve separability and reduce the constraint size.  The paper should explain how you are ensuring filtering is only affecting one criteria and not the other.  \n\n* I understand why the authors chose to lay out Figure 1 as they did.  However, I did not find the overlapping bars were the best format.  I was initially confused by the formatting, and it took me some time to figure out exactly what was being visualized since I expected multiple attacks, but only saw two bars.\n\n**Miscellaneous Comments**\n\n* Overall, I found section 5.1 notationally cumbersome and lacking a clear structure and flow. I had to re-read it twice to extract all the main points.  One possible fix is to provide more intuition about why each is being proposed and how the pieces relate.  It felt as though you gave a lot of disparate pieces of information to the reader without walking them through how they fit together as you went alone\n\n* Lines 40-41 state, \"An attack is considered ineffective if the increased risk from poisoning is roughly equal to or smaller than the injected poisoning rate.\"  This perspective makes sense, but it took me a minute to understand the definition as stated. I think this sentence could be written more intuitively.\n\nQUESTIONS:\n* Eq. (2) allows the two Gaussian distributions to have distinct variances (e.g., $\\sigma_1$ and $\\sigma_2$). In Theorem 5.3, you specify the two variances as equivalent. Is Theorem 5.3's equivariance specification for convenience in the proof or a necessity?\n\n* On Line 155, you wrote, \"*We chose 3% as the maximum poisoning rate.\" I found this confusing wording. I understanding \"maximum poisoning rate\" is terminology you use in the theoretical discussion.  When discussing experiments you perform, defining your poisoning rate w.r.t. a *maximum* implies to me you tested multiple poisoning rates. However, I do not get that indication elsewhere in the paragraph.  Please be precise concerning your experimental setup.\n\n* Figure 1 uses the \"Filtered Enron\" dataset.  I am not familiar with the filtered version of the Enron dataset. Is this filtering procedure something you performed or an existing dataset? How do the results change if you use the vanilla Enron dataset?  If you filtered the dataset yourself, I believe the paper should explain why that filtering was done, not just how it was performed.\n\n* Your preliminaries section specifies a few design choices. For example, you restrict consideration to primarily hinge loss. Moreover, you specify an L2 regularizer (Eq. (1)). As a reader, I am somewhat left to guess why these specific choices are made. For example, I assume you chose hinge loss because you want max margin.  Concerning regularizer, I was unsure if L2 was for convenience and whether a different convex regularizer (e.g., L1) could be used. If you are focusing on specific design choices, I believe it is better to explain why.\n    * I do not recall any of the theorems requiring specifically the hinge loss as opposed to alternate convex losses.\n\n* Lines 183-184 state, \"Definition 4.1 *suggests* that no poisoning strategy can achieve a better attack performance than that achieved by $\\mathcal{A}_{opt}$\" When reading this, I was confused by the choice of the word \"suggests,\" as given $\\epsilon$, I found Def. 4.1 clearly stated no alternate strategy achieves better attack performance.\n\n* One criterion for Theorem 4.5 to hold is that the support of $\\mu_c$ should be a (proper) subset of $\\mathcal{C}$.  I believe this criterion does not hold in the optimal Gaussian case given the definition of $\\mathcal{Q}(u)$.  If so, please explain how that affects your results (including the proof). Ideally, this should be more explicit in the paper.\n   * Searching through the paper, I did not see where $\\mathcal{Q}(u)$ was used beyond line 246. Perhaps I missed it. If it is just for that one point, I think it is worth removing.\n\nLIMITATIONS:\nBetter understanding what causes attacks to fail could lead to enable better attacks.  At the same time, it can lead to better defenses.  I see no serious ethical or societal risks for this work.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-15"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper delves into the impact and effectiveness of indiscriminate data poisoning attacks on machine learning models, particularly linear ones. It highlights significant variations in attack effectiveness across different datasets and seeks to understand whether this is due to inherent dataset robustness or shortcomings in current attack strategies. The paper presents optimal poisoning definitions, characterizes these attacks under a theoretical Gaussian mixture model, and identifies learning task properties that affect attack performance. It concludes with suggestions for designing better defenses against such attacks.\n\nSTRENGTHS:\n- The question the paper is trying to answer, of whether some datasets are inherently robust to poisoning attacks or just resilient to state-of-the-art attacks, is original and interesting.\n- The paper flows smoothly and is decently written, but contains some grammatical mistakes and typos (some of which I mention below). A careful proofread can increase the quality of writing quite a bit.\n- The paper is sound, and the claims are proved on 1D toy Gaussian mixture model then extended to more complex data distributions. \n- The empirical results support the theoretical results. Specifically, on various datasets, the results show that the datasets that are less vulnerable to poisoning have higher values for metrics defined by the paper (Sep/SD and Sep/Size).\n\nWEAKNESSES:\n- The contributions of the paper is not very significant as there is no clear practical implication of the findings of the paper. However, the paper investigates an interesting phenomenon of \"inherent robustness\" of some datasets which might inspire further investigations of this sort.\n- The paper analysis is restricted to linear models.\n\nQUESTIONS:\n- The format of figure out is confusing. \n  - Are the bars stacked? \n  - Caption mentions “Datasets are sorted from lowest to highest base error rate”. Does this mean the error rate of the SVM trained on $S_c$? \n  - Please change the color palette to more distinguishable colors. \n\n- The paper analysis is restricted to linear models. Would any of these insights hold for more complex models? Did the authors do any (empirical) investigation there? In particular, it would be interesting if the authors find that high Sep/SD and Sep/Size metrics are indicative of lower vulnerability to poisoning in more general neural networks and on datasets such as CIFAR-10 and ImageNet.\n\n- Are the percentage changes in the results paragraph in Line 157 absolute or relative? I believe it is absolute, but would be helpful to clarify that.\n\n### Minor\n\n- Line 123: What is a “theoretic game”?\n- Line 278: an higher-> a higher\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-06-26"}

comments: SUMMARY:
Inspired by the variant performance of indiscriminate data poisoning attacks against different datasets, this paper studies the inherent robustness of datasets against them. Using an optimal attack designed for a binary Gaussian dataset, the authors discover three properties, namely class separability, variation, and constraint size are important factors to determine the robustness of a dataset. Experiments further validate the theory on several datasets.

STRENGTHS:
(1) This paper studies an important problem, namely the dataset robustness against indiscriminate data poisoning attacks.

(2) The authors provide initial but valuable evidence on understanding the vulnerability of datasets by explicating the three properties. Although these characteristics match our intuitive understanding of the problem, they have not been systematically analyzed or studied.

(3) The empirical findings, especially the results in Section D, are nicely presented and very comprehensive. Moreover, I can find answers to most of my initial questions when reading the main paper in the appendix.

WEAKNESSES:
(1) I don't find the title a nice summarization of the entire paper as it is probably too general. I would suggest mentioning the dataset robustness in the title.

(2) I don't think comparing with LDC is necessary as I clearly see the drastic difference. I suggest the authors move this part to the appendix and present some more valuable content in the main paper, e.g., some experiments in Appendix D.

(3) Although it has been mentioned in the appendix, I think it is important to mention what attacks you use to generate Table 1. The authors should at least provide a pointer to the appendix.

QUESTIONS:
Questions are listed above.

LIMITATIONS:
Limitations are well addressed.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2023-07-06

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper explores the issue of data poisoning attacks on machine learning models, focusing on linear models. The authors find that the effectiveness of state-of-the-art poisoning strategies varies across different datasets. They introduce definitions of optimal poisoning attacks for finite-sample and distributional settings and prove performance convergence under certain conditions. They also characterize optimal poisoning attacks under a theoretical Gaussian mixture model and find that a larger projected constraint size is linked with higher vulnerability, while larger separability and smaller standard deviation provide less vulnerability. These findings explain the performance differences of indiscriminate poisoning attacks on linear models across various datasets and suggest ways to improve robustness against poisoning.\n\nSTRENGTHS:\n- The observation that the performance of state-of-the-art poisoning strategies significantly varies across different datasets is indeed insightful. It not only challenges the common understanding of indiscriminate poisoning attacks but also opens up new lines of inquiry into why these differences occur.\n- The theoretical analysis under specific data distributions is a major strength of this paper. It's robust, reliable, and underpins a comprehensive understanding of how and why different datasets might be more or less vulnerable to poisoning attacks. By tying the theoretical analysis back to empirical attack performance, the authors effectively bridge the gap between theory and practice, enhancing the practical relevance of their findings.\n\nWEAKNESSES:\n- It's not explicitly clear from the paper whether the observation that \"linear learners on some datasets are able to resist the best-known attacks even without any defenses\" is novel or previously reported. The authors could benefit from addressing this explicitly. \n\n- The authors' focus on linear classifiers is indeed a limitation. While the paper provides valuable insights into data poisoning attacks on linear models, these findings cannot be directly extrapolated to non-linear classifiers such as neural networks. Moreover, the paper does not provide enough information to definitively conclude whether linear classifiers are more robust than non-linear classifiers without additional defense strategies.\n\n-  The paper doesn't sufficiently distinguish its analysis from existing works on data poisoning that also provided theoretical analysis on Gaussian mixture distributions and linear classifiers, such as [1]. Could you elaborate on this?\n\n\n\n[1] Tao, Lue, et al. \"Can Adversarial Training Be Manipulated By Non-Robust Features?.\" Advances in Neural Information Processing Systems 35 (2022): 26504-26518.\n\nQUESTIONS:\n- Is the resistance of some linear learners to attacks, even without defenses, a new finding or reported in previous studies?\n\n- How applicable are these findings to non-linear classifiers like neural networks? Do they imply greater inherent robustness of linear classifiers against attacks?\n\nLIMITATIONS:\nIn summary, this paper makes a substantial contribution to the field. Although its primary limitation is its exclusive focus on linear classifiers, it effectively elucidates the observed phenomena within this scope.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}