SlowGuess/NeurIPS_2023_OCR huggingface dataset:
We evaluate the robustness of this dataset against poisoning attacks and set  $\mathcal{C}$  as dimension-wise box-constraints, whose values are the minimum and maximum values of the clean data points for each dimension when fed to the feature extractors. This way of configuring  $\mathcal{C}$  corresponds to the practical scenario where the victim has
Figure 2: Improving downstream robustness to poisoning through better feature extractors.
access to some small number of clean samples so that they can deploy a simple defense of filtering out inputs that do not fall into a dimension-wise box constraint that is computed from the available clean samples of the victim. Figure 2 shows that as the feature extractor becomes better (using deeper models and training for more epochs), both the Sep/SD and Sep/Size metrics increase, leading to reduced error increase. Quantitatively, the Pearson correlation coefficient between the error increase and each of the two factors is  $-0.98$  (a strong negative correlation). This result demonstrates the possibility that better feature representations free from poisoning might be leveraged to improve downstream resilience against indiscriminate poisoning attacks. We also simulated a more practical scenario where the training of the pretrained models never sees any data (i.e., data from classes of "Truck" and "Ship") used in the downstream analysis and the conclusions are similar (Appendix D.2.7).
Our work also has several limitations. We only characterize the optimal poisoning attacks for theoretical distributions under linear models. Even for the linear models, the identified metrics cannot quantify the actual error increase from optimal poisoning attacks, which is an interesting future work, and one possible approach might be to tighten the upper bound in Theorem 5.7 using better optimization methods. The metrics identified in this paper are learner dependent, depending on the properties of the learning algorithm, dataset and domain constraints (mainly reflected through  $\mathcal{C}$ ). In certain applications, one might be interested in understanding the impact of learner agnostic dataset properties on poisoning effectiveness—a desired dataset has such properties that any reasonable learners trained on the dataset can be robust to poisoning attacks. One likely application scenario is when the released data will be used by many different learners in various applications, each of which may be prone to poisoning. We also did not systematically investigate how to compare the vulnerabilities of different datasets under different learning algorithms. Identifying properties specific to the underlying learner that affect the performance of (optimal) data poisoning attacks is challenging but interesting future work.
Although we focus on indiscriminate data poisoning attacks in this paper, we are optimistic that our results will generalize to subpopulation or targeted poisoning settings. In particular, the specific learning task properties identified in this paper may still be highly correlated, but now additional factors related to the relative positions of the subpopulations/individual test samples to the rest of the population under the clean decision boundary will also play important roles.
We appreciate the comments from the anonymous reviewers. This work was partially funded by awards from the National Science Foundation (NSF) SaTC program (Center for Trustworthy Machine Learning, #1804603), the AI Institute for Agent-based Cyber Threat Intelligence and Operation (ACTION) (#2229876), NSF #2323105, and NSF #2325369. Any opinions, findings and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the National Science Foundation. Fnu Suya also acknowledges the financial support from the Maryland Cybersecurity Center (MC2) at the University of Maryland to attend the conference.
[1] Amazon, Inc. Amazon SageMaker. https://aws.amazon.com/sagemaker/. Accessed: 2023-05-01.
[2] Daniel Arp, Michael Spreitzenbarth, Malte Hubner, Hugo Gascon, Konrad Rieck, and CERT Siemens. Drebin: Effective and explainable detection of android malware in your pocket. In Network and Distributed System Security, 2014.
[3] Battista Biggio, Blaine Nelson, and Pavel Laskov. Support Vector Machines Under Adversarial Label Noise. In *Asian Conference on Machine Learning*, 2011.
[4] Battista Biggio, Blaine Nelson, and Pavel Laskov. Poisoning Attacks against Support Vector Machines. In International Conference on Machine Learning, 2012.
[5] Mustain Billah, Adnan Anwar, Ziaur Rahman, and Syed Md Galib. Bi-level poisoning attack model and countermeasure for appliance consumption data of smart homes. Energies, 14(13):3887, 2021.
[6] Nicholas Carlini, Matthew Jagielski, Christopher A Choquette-Choo, Daniel Paleka, Will Pearce, Hyrum Anderson, Andreas Terzis, Kurt Thomas, and Florian Tramér. Poisoning web-scale training datasets is practical. arXiv preprint arXiv:2302.10149, 2023.
[7] Yizheng Chen, Zhoujie Ding, and David Wagner. Continuous learning for Android malware detection. arXiv preprint arXiv:2302.04332, 2023.
[8] Yizheng Chen, Shiqi Wang, Yue Qin, Xiaojing Liao, Suman Jana, and David Wagner. Learning security classifiers with verified global robustness properties. In ACM Conference on Computer and Communications Security, 2021.
[9] Antonio Emanuele Cinà, Sebastiano Vascon, Ambra Demontis, Battista Biggio, Fabio Roli, and Marcello Pelillo. The hammer and the nut: Is bilevel optimization really needed to poison linear classifiers? In International Joint Conference on Neural Networks, 2021.
[10] Ambra Demontis, Marco Melis, Maura Pintor, Matthew Jagielski, Battista Biggio, Alina Oprea, Cristina Nita-Rotaru, and Fabio Roli. Why Do adversarial attacks transfer? Explaining transferability of evasion and poisoning attacks. In USENIX Security Symposium, 2019.
[11] Ambra Demontis, Paolo Russu, Battista Biggio, Giorgio Fumera, and Fabio Roli. On security and sparsity of linear classifiers for adversarial settings. In Structural, Syntactic, and Statistical Pattern Recognition: Joint IAPR International Workshop. Springer, 2016.
[12] Ilias Diakonikolas, Gautam Kamath, Daniel Kane, Jerry Li, Jacob Steinhardt, and Alistair Stewart. Sever: A robust meta-algorithm for stochastic optimization. In International Conference on Machine Learning, 2019.
[13] Ilias Diakonikolas, Daniel Kane, and Nikos Zarifis. Near-optimal SQ lower bounds for agnostically learning halfspaces and ReLUs under Gaussian marginals. Advances in Neural Information Processing Systems, 2020.
[14] Dheeru Dua and Casey Graff. UCI Machine Learning Repository, 2017.
[52] Yaqing Wang, Quanming Yao, James T Kwok, and Lionel M Ni. Generalizing from a few examples: A survey on few-shot learning. ACM computing surveys (csur), 53(3):1-34, 2020.
[53] Eric Wong and Zico Kolter. Provable defenses against adversarial examples via the convex outer adversarial polytope. In International Conference on Machine Learning, 2018.
[54] Han Xiao, Huang Xiao, and Claudia Eckert. Adversarial Label Flips Attack on Support Vector Machines. In European Conference on Artificial Intelligence, 2012.
[55] Da Yu, Huishuai Zhang, Wei Chen, Jian Yin, and Tie-Yan Liu. Indiscriminate poisoning attacks are shortcuts. 2021.
[56] Chen Zhu, W Ronny Huang, Ali Shafahi, Hengduo Li, Gavin Taylor, Christoph Studer, and Tom Goldstein. Transferable clean-label poisoning attacks on Deep Neural Nets. In International Conference on Machine Learning, 2019.
A Proofs of Main Results in Section 4
A.1 Proof of Theorem 4.3
We first introduce the formal definitions of strong convexity and Lipschitz continuity conditions with respect to a function, and the uniform convergence property respect to a hypothesis class. These definitions are necessary for the proof of Theorem 4.3.
Definition A.1 (Strong Convexity). A function  $f: \mathbb{R}^n \to \mathbb{R}$  is  $b$ -strongly convex for some  $b > 0$ , if  $f(\pmb{x}_1) \geq f(\pmb{x}_2) + \nabla f(\pmb{x}_2)^\top (\pmb{x}_1 - \pmb{x}_2) + \frac{b}{2} \| \pmb{x}_1 - \pmb{x}_2 \|_2^2$  for any  $\pmb{x}_1, \pmb{x}_2 \in \mathbb{R}^n$ .
Definition A.2 (Lipschitz Continuity). A function  $f: \mathbb{R}^n \to \mathbb{R}$  is  $\rho$ -Lipschitz for some  $\rho > 0$ , if  $|f(\pmb{x}_1) - f(\pmb{x}_2)| \leq \rho \| \pmb{x}_1 - \pmb{x}_2 \|_2$  for any  $\pmb{x}_1, \pmb{x}_2 \in \mathbb{R}^n$ .
Definition A.3 (Uniform Convergence). Let  $\mathcal{H}$  be a hypothesis class. We say that  $\mathcal{H}$  satisfies the uniform convergence property with a loss function  $\ell$ , if there exists a function  $m_{\mathcal{H}}:(0,1)^2\to \mathbb{N}$  such that for every  $\epsilon', \delta' \in (0,1)$  and for every probability distribution  $\mu$ , if  $\mathcal{S}$  is a set of examples with  $m\geq m_{\mathcal{H}}(\epsilon',\delta')$  samples drawn i.i.d. from  $\mu$ , then
$$
\mathbb {P} _ {\mathcal {S} \leftarrow \mu^ {m}} \left[ \sup  _ {h \in \mathcal {H}} \left| L (h; \hat {\mu} _ {\mathcal {S}}) - L (h; \mu) \right| \leq \epsilon^ {\prime} \right] \geq 1 - \delta^ {\prime}.
$$
Such a uniform convergence property, which can be achieved using the VC dimension or the Rademacher complexity of  $\mathcal{H}$ , guarantees that the learning rule specified by empirical risk minimization always returns a good hypothesis with high probability [43]. Similar to PAC learning, the function  $m_{\mathcal{H}}$  measures the minimal sample complexity requirement that ensures uniform convergence.
Now, we are ready to prove Theorem 4.3
Proof of Theorem 4.3. First, we introduce the following notations to simplify the proof. For any  $S_p$ ,  $\mu_p$  and  $\delta \geq 0$ , let
$$
\hat {g} (\mathcal {S} _ {p}, \mathcal {S} _ {c}) = \operatorname * {a r g m i n} _ {h \in \mathcal {H}} \sum_ {(\boldsymbol {x}, y) \in \mathcal {S} _ {c} \cup \mathcal {S} _ {p}} \ell (h; \boldsymbol {x}, y),
$$
$$
g(\delta ,\mu_{p},\mu_{c}) = \operatorname *{argmin}_{h\in \mathcal{H}}\left\{L(h;\mu_{c}) + \delta \cdot L(h;\mu_{p})\right\} .
$$
According to the definitions of  $\hat{h}_p^*$  and  $h_p^*$ , we know  $\hat{h}_p^* = \hat{g}(\mathcal{S}_p^*, \mathcal{S}_c)$  and  $h_p^* = g(\delta^*, \mu_p^*, \mu_c)$ .
Now we are ready to prove Theorem 4.3. For any  $S_{c}$  sampled from  $\mu_c$ , consider the empirical loss minimizer  $\hat{h}_p^* = \hat{g} (S_p^*,S_c)$  and the population loss minimizer  $g(\delta_{S_p^*},\hat{\mu}_{S_p^*},\mu_c)$ , where  $\delta_{S_p^*} = |S_p^*| / |S_c|$ . Then  $S_{p}^{*}\cup S_{c}$  can be regarded as the i.i.d. sample set from  $(\mu_c + \delta_{S_p^*}\cdot \hat{\mu}_{S_p^*}) / (1 + \delta_{S_p^*})$ . According to Definition A.3, since  $\mathcal{H}$  satisfies the uniform convergence property with respect to  $\ell$ , we immediately know that the empirical loss minimization is close to the population loss minimization if the sample size is large enough (see Lemma 4.2 in [43]). To be more specific, for any  $\epsilon',\delta'\in (0,1)$ , if  $|\mathcal{S}_c|\geq m_{\mathcal{H}}(\epsilon',\delta')$ , then with probability at least  $1 - \delta'$ , we have
$$
\begin{array}{l} L\big(\hat{g} (\mathcal{S}_{p}^{*},\mathcal{S}_{c});\mu_{c}\big) + \delta_{\mathcal{S}_{p}^{*}}\cdot L\big(\hat{g} (\mathcal{S}_{p}^{*},\mathcal{S}_{c});\hat{\mu}_{\mathcal{S}_{p}^{*}}\big)\leq \operatorname *{argmin}_{h\in \mathcal{H}}\left\{L(h;\mu_{c}) + \delta_{\mathcal{S}_{p}^{*}}\cdot L(h;\hat{\mu}_{\mathcal{S}_{p}^{*}})\right\} +2\epsilon^{\prime} \\ = L \left(g \left(\delta_ {\mathcal {S} _ {p} ^ {*}}, \hat {\mu} _ {\mathcal {S} _ {p} ^ {*}}, \mu_ {c}\right); \mu_ {c}\right) + \delta_ {\mathcal {S} _ {p} ^ {*}} \cdot L \left(g \left(\delta_ {\mathcal {S} _ {p} ^ {*}}, \hat {\mu} _ {\mathcal {S} _ {p} ^ {*}}, \mu_ {c}\right); \hat {\mu} _ {\mathcal {S} _ {p} ^ {*}}\right) + 2 \epsilon^ {\prime}. \\ \end{array}
$$
In addition, since the surrogate loss  $\ell$  is  $b$ -strongly convex and the population risk is  $\rho$ -Lipschitz, we further know the clean risk of  $\hat{g}(\mathcal{S}_p^*, \mathcal{S}_c)$  and  $g(\delta_{\mathcal{S}_p^*}, \hat{\mu}_{\mathcal{S}_p^*}, \mu_c)$  is guaranteed to be close. Namely, with probability at least  $1 - \delta'$ , we have
$$
\begin{array}{l} \left| \operatorname {R i s k} \left(\hat {g} \left(S _ {p} ^ {*}, S _ {c}\right); \mu_ {c}\right) - \operatorname {R i s k} \left(g \left(\delta_ {S _ {p} ^ {*}}, \hat {\mu} _ {S _ {p} ^ {*}}, \mu_ {c}\right); \mu_ {c}\right) \right| \leq \rho \cdot \left\| \hat {g} \left(S _ {p} ^ {*}, S _ {c}\right) - g \left(\delta_ {S _ {p} ^ {*}}, \hat {\mu} _ {S _ {p} ^ {*}}, \mu_ {c}\right) \right\| _ {2} \\ \leq 2 \rho \sqrt {\frac {\epsilon^ {\prime}}{b}}. \\ \end{array}
$$
Note that  $\delta_{\mathcal{S}_p^*} \in [0, \epsilon]$  and  $\mathrm{supp}(\hat{\mu}_{\mathcal{S}_p^*}) \subseteq \mathcal{C}$ . Thus, according to the definition of  $h_p^* = g(\delta^*, \mu_p^*, \mu_c)$ , we further have
$$
\begin{array}{l} \operatorname {R i s k} \left(h _ {p} ^ {*}; \mu_ {c}\right) \geq \operatorname {R i s k} \left(g \left(\delta_ {\mathcal {S} _ {p} ^ {*}}, \hat {\mu} _ {\mathcal {S} _ {p} ^ {*}}, \mu_ {c}\right); \mu_ {c}\right) \geq \operatorname {R i s k} \left(\hat {g} \left(\mathcal {S} _ {p} ^ {*}, \mathcal {S} _ {c}\right); \mu_ {c}\right) - 2 \rho \sqrt {\frac {\epsilon^ {\prime}}{b}} \\ = \operatorname {R i s k} \left(\hat {h} _ {p} ^ {*}; \mu_ {c}\right) - 2 \rho \sqrt {\frac {\epsilon^ {\prime}}{b}}. \tag {6} \\ \end{array}
$$
So far, we have proven one direction of the asymptotic for results Theorem 4.3.
On the other hand, we can always construct a subset  $\tilde{S}_p$  with size  $|\tilde{S}_p| = \delta^* \cdot |\mathcal{S}_c|$  by i.i.d. sampling from  $\mu_p^*$ . Consider the empirical risk minimizer  $\hat{g}(\tilde{S}_p, \mathcal{S}_c)$  and the population risk minimizer  $h_p^* = g(\delta^*, \mu_p^*, \mu_c)$ . Similarly, since  $\mathcal{H}$  satisfies the uniform convergence property, if  $|\mathcal{S}_c| \geq m_{\mathcal{H}}(\epsilon', \delta')$ , then with probability at least  $1 - \delta'$ , we have
$$
\begin{array}{l} L\bigl(\hat{g} (\tilde{\mathcal{S}}_{p},\mathcal{S}_{c});\mu_{c}\bigr) + \delta^{*}\cdot L\bigl(\hat{g} (\tilde{\mathcal{S}}_{p},\mathcal{S}_{c});\mu_{p}^{*}\bigr)\leq \operatorname *{argmin}_{h\in \mathcal{H}}\left\{L(h;\mu_{c}) + \delta^{*}\cdot L(h;\mu_{p}^{*})\right\} +2\epsilon^{\prime} \\ = L \left(g \left(\delta^ {*}, \mu_ {p} ^ {*}, \mu_ {c}\right); \mu_ {c}\right) + \delta^ {*} \cdot L \left(g \left(\delta^ {*}, \mu_ {p} ^ {*}, \mu_ {c}\right); \mu_ {p} ^ {*}\right) + 2 \epsilon^ {\prime}. \\ \end{array}
$$
According to the strong convexity of  $\ell$  and the Lipschitz continuity of the population risk, we further have
$$
\begin{array}{l} \left| \operatorname {R i s k} \left(\hat {g} \left(\tilde {\mathcal {S}} _ {p}, \mathcal {S} _ {c}\right); \mu_ {c}\right) - \operatorname {R i s k} \left(g \left(\delta^ {*}, \mu_ {p} ^ {*}, \mu_ {c}\right); \mu_ {c}\right) \right| \leq \rho \cdot \left\| \hat {g} \left(\tilde {\mathcal {S}} _ {p}, \mathcal {S} _ {c}\right) - g \left(\delta^ {*}, \mu_ {p} ^ {*}, \mu_ {c}\right) \right\| _ {2} \\ \leq 2 \rho \sqrt {\frac {\epsilon^ {\prime}}{b}}. \\ \end{array}
$$
Note that  $\tilde{S}_p\subseteq \mathcal{C}$  and  $|\tilde{S}_p| = \delta^*\cdot |\mathcal{S}_c|\leq \epsilon \cdot |\mathcal{S}_c|$ . Thus according to the definition of  $\hat{h}_p^* = \hat{g} (\mathcal{S}_p^*,\mathcal{S}_c)$ , we have
$$
\begin{array}{l} \operatorname {R i s k} (\hat {h} _ {p} ^ {*}; \mu_ {c}) \geq \operatorname {R i s k} \big (\hat {g} (\tilde {\mathcal {S}} _ {p}, \mathcal {S} _ {c}); \mu_ {c} \big) \geq \operatorname {R i s k} \big (g (\delta^ {*}, \mu_ {p} ^ {*}, \mu_ {c}); \mu_ {c} \big) - 2 \rho \sqrt {\frac {\epsilon^ {\prime}}{b}} \\ = \operatorname {R i s k} \left(h _ {p} ^ {*}; \mu_ {c}\right) - 2 \rho \sqrt {\frac {\epsilon^ {\prime}}{b}}. \tag {7} \\ \end{array}
$$
Combining (6) and (7), we complete the proof of Theorem 4.3.
A.2 Proof of Theorem 4.5
Proof of Theorem 4.5. We prove Theorem 4.5 by construction.
We start with the first condition  $\mathrm{supp}(\mu_c) \subseteq \mathcal{C}$ . Suppose  $\delta^* < \epsilon$ , since the theorem trivially holds if  $\delta^* = \epsilon$ . To simplify notations, define  $h_p(\delta, \mu_p) = \operatorname{argmin}_{h \in \mathcal{H}} \{L(h; \mu_c) + \delta \cdot L(h; \mu_p)\}$  for any  $\delta$  and  $\mu_p$ . To prove the statement in Theorem 4.5, it is sufficient to show that there exists some  $\mu_p^{(\epsilon)}$  based on the first condition such that
$$
\operatorname {R i s k} \left(h _ {p} \left(\epsilon , \mu_ {p} ^ {(\epsilon)}\right); \mu_ {c}\right) = \operatorname {R i s k} \left(h _ {p} \left(\delta^ {*}, \mu_ {p} ^ {*}\right); \mu_ {c}\right), \text {a n d} \operatorname {s u p p} \left(\mu_ {p} ^ {(\epsilon)}\right) \subseteq \mathcal {C}. \tag {8}
$$
The above equality means we can always achieve the same maximum risk after poisoning with the full poisoning budget  $\epsilon$ . To proceed with the proof, we construct  $\mu_p^{(\epsilon)}$  based on  $\mu_c$  and  $\mu_p^*$  as follows:
$$
\begin{array}{l} \mu_ {p} ^ {(\epsilon)} = \frac {\delta^ {*}}{\epsilon} \cdot \mu_ {p} ^ {*} + \frac {\epsilon - \delta^ {*}}{\epsilon (1 + \delta^ {*})} \cdot (\mu_ {c} + \delta^ {*} \cdot \mu_ {p} ^ {*}) \\ = \frac {\epsilon - \delta^ {*}}{\epsilon (1 + \delta^ {*})} \cdot \mu_ {c} + \frac {\delta^ {*} (1 + \epsilon)}{\epsilon (1 + \delta^ {*})} \cdot \mu_ {p} ^ {*}. \\ \end{array}
$$
We can easily check that  $\mu_p^{(\epsilon)}$  is a valid probability distribution and  $\mathrm{supp}(\mu_p^{(\epsilon)})\subseteq \mathcal{C}$ . In addition, we can show that
$$
\begin{array}{l} h_{p}(\epsilon ,\mu_{p}^{(\epsilon)}) = \operatorname *{argmin}_{h\in \mathcal{H}}\left\{L(h;\mu_{c}) + \epsilon \cdot L(h;\mu_{p}^{(\epsilon)})\right\} \\ = \operatorname * {a r g m i n} _ {h \in \mathcal {H}} \left\{\mathbb {E} _ {(\boldsymbol {x}, y) \sim \mu_ {c}} \ell (h; \boldsymbol {x}, y) + \epsilon \cdot \mathbb {E} _ {(\boldsymbol {x}, y) \sim \mu_ {p} ^ {(\epsilon)}} \ell (h; \boldsymbol {x}, y) \right\} \\ = \operatorname * {a r g m i n} _ {h \in \mathcal {H}} \left\{\frac {1 + \epsilon}{1 + \delta^ {*}} \cdot \left(\mathbb {E} _ {(\boldsymbol {x}, y) \sim \mu_ {c}} \ell (h; \boldsymbol {x}, y) + \delta^ {*} \cdot \mathbb {E} _ {(\boldsymbol {x}, y) \sim \mu_ {p} ^ {*}} \ell (h; \boldsymbol {x}, y)\right) \right\} \\ = h _ {p} \left(\delta^ {*}, \mu_ {p} ^ {*}\right) \\ \end{array}
$$
where the third equality holds because of the construction of  $\mu_p^{(\epsilon)}$ . Therefore, we have proven (8), which further implies the optimal attack performance can always be achieved with  $\epsilon$ -poisoning as long as the first condition is satisfied.
Next, we turn to the second condition of Theorem 4.5. Similarly, it is sufficient to construct some  $\mu_p^{(\epsilon)}$  for the setting where  $\delta^* < \epsilon$  such that
$$
\operatorname {R i s k} \big (h _ {p} (\epsilon , \mu_ {p} ^ {(\epsilon)}); \mu_ {c} \big) = \operatorname {R i s k} \big (h _ {p} (\delta^ {*}, \mu_ {p} ^ {*}); \mu_ {c} \big), \text {a n d} \operatorname {s u p p} (\mu_ {p} ^ {(\epsilon)}) \subseteq \mathcal {C}.
$$
We construct  $\mu_p^{(\epsilon)}$  based on  $\mu_p^*$  and the assumed data distribution  $\mu$ . More specifically, we construct
$$
\mu_ {p} ^ {(\epsilon)} = \frac {\delta^ {*}}{\epsilon} \cdot \mu_ {p} ^ {*} + \frac {\epsilon - \delta^ {*}}{\epsilon} \cdot \mu . \tag {9}
$$
By construction, we know  $\mu_p^{(\epsilon)}$  is a valid probability distribution. In addition, according to the assumption of  $\mathrm{supp}(\mu) \subseteq \mathcal{C}$ , we have  $\mathrm{supp}(\mu_p^{(\epsilon)}) \subseteq \mathcal{C}$ . According to the assumption that for any  $\pmb{\theta}$ , there exists a  $\mu$  such that  $\frac{\partial}{\partial_{\pmb{\theta}}} L(h_{\pmb{\theta}}; \mu) = \mathbf{0}$ , we know for any possible weight parameter  $\pmb{\theta}_p^*$  of  $h_p(\delta^*, \mu_p^*)$ , there also exists a corresponding  $\mu$  such that the gradient is  $\mathbf{0}$  and therefore, we have
$$
\begin{array}{l} \frac {\partial}{\partial \boldsymbol {\theta} _ {p} ^ {*}} \big (L (h _ {p} (\delta^ {*}, \mu_ {p} ^ {*}); \mu_ {c}) + \epsilon \cdot L (h _ {p} (\delta^ {*}, \mu_ {p} ^ {*}); \mu_ {p} ^ {(\epsilon)}) \big) \\ = \frac {\partial}{\partial_ {\boldsymbol {\theta} _ {p} ^ {*}}} \big (L (h _ {p} (\delta^ {*}, \mu_ {p} ^ {*}); \mu_ {c}) + \delta^ {*} \cdot L (h _ {p} (\delta^ {*}, \mu_ {p} ^ {*}; \mu_ {p} ^ {*}) \big) \\ = 0 \\ \end{array}
$$
where the last equality is based on the first-order optimality condition of  $h_p(\delta^*, \mu_p^*)$  for convex losses. For simplicity, we also assumed  $h_p(\delta^*, \mu_p^*)$  is obtained by minimizing the loss on  $\mu_c + \delta^* \cdot \mu_p^*$  while in the case of  $\mathrm{supp}(\mu_c) \not\subset \mathcal{C}$ , the victim usually minimizes the loss on  $\bar{\mu}_c + \delta^* \cdot \mu_p^*$ , where  $\bar{\mu}_c$  is the "truncated" version of  $\mu_c$  such that  $\mathrm{supp}(\bar{\mu}_c) \subseteq \mathcal{C}$ . To conclude, we know  $h_p(\epsilon, \mu_p^{(\epsilon)}) = h_p(\delta^*, \mu_p^*)$  holds for any possible  $h_p(\delta^*, \mu_p^*)$  and we complete the proof of Theorem 4.5.
A.3 Proofs of the Statement about Linear Models in Remark 4.6
Proof. We provide the construction of  $\mu$  with respect to the second condition of Theorem 4.5 for linear models and hinge loss. Since for any  $h_{\boldsymbol{w},b} \in \mathcal{H}_L$  and any  $(\boldsymbol{x},y) \in \mathcal{X} \times \mathcal{Y}$ , we have
$$
\ell (h _ {\boldsymbol {w}, b}; \boldsymbol {x}, y) = \max \{0, 1 - y (\boldsymbol {w} ^ {\top} \boldsymbol {x} + b) \} + \frac {\lambda}{2} \| \boldsymbol {w} \| _ {2} ^ {2}.
$$
Let  $\theta = (\pmb{w}, b)$ , then the gradient with respect to  $\pmb{w}$  can be written as:
$$
\frac {\partial}{\partial \boldsymbol {w}} \ell (h _ {\boldsymbol {w}, b}; \boldsymbol {x}, y) = \left\{ \begin{array}{l l} - y \cdot \boldsymbol {x} + \lambda \boldsymbol {w} & \text {i f} y (\boldsymbol {w} ^ {\top} \boldsymbol {x} + b) \leq 1, \\ \lambda \boldsymbol {w} & \text {o t h e r w i s e}. \end{array} \right.
$$
Similarly, the gradient with respect to  $b$  can be written as:
$$
\frac {\partial}{\partial_ {b}} \ell (h _ {\boldsymbol {w}, b}; \boldsymbol {x}, y) = \left\{ \begin{array}{l l} - y & \text {i f} y (\boldsymbol {w} ^ {\top} \boldsymbol {x} + b) \leq 1, \\ 0 & \text {o t h e r w i s e .} \end{array} \right.
$$
Therefore, for large input space  $\mathcal{X} \times \mathcal{Y}$ , we can simply construct  $\mu$  by constructing a two-point distribution (with equal probabilities of label  $+1$  and  $-1$ ) that cancels out each other's gradient (for both  $\boldsymbol{w}$  and  $b$ ), so that  $y(\boldsymbol{w}^\top \boldsymbol{x} + b) \leq 1$  and  $-y \cdot \boldsymbol{x} + \lambda \boldsymbol{w} = \mathbf{0}$ .
We may also generalize the construction of  $\mu$  from linear models with hinge loss to general convex models if the victim minimizes the loss on  $\bar{\mu}_c + \delta^* \cdot \mu_p^*$  to obtain  $h_p(\delta^*, \mu_p^*)$ , which is common in practice. In this case, we can simply set  $\mu = \bar{\mu}_c + \delta^* \cdot \mu_p^*$ , which guarantees
$$
\frac {\partial}{\partial_ {\boldsymbol {\theta} _ {p} ^ {*}}} L (h _ {p} (\delta^ {*}, \mu_ {p} ^ {*}); \mu) = \mathbf {0}.
$$
B Proofs of Main Results in Section 5
B.1 Proof of Theorem 5.3
To prove Theorem 5.3, we need to make use of the following three auxiliary lemmas, which are related to the maximum population hinge loss with  $w = 1$  (Lemma B.1), the weight-flipping condition (Lemma B.2) and the risk behaviour of any linear hypothesis under (2) (Lemma B.3). For the sake of completeness, we present the full statements of Lemma B.1 and Lemma B.2 as follows. In particular, Lemma B.1, proven in Appendix C.1, characterizes the maximum achievable hinge loss with respect to the underlying clean distribution  $\mu_{c}$  and some poisoned distribution  $\mu_{p}$  conditioned on  $w = 1$ .
Lemma B.1. Suppose the underlying clean distribution  $\mu_c$  follows the Gaussian mixture model (2) with  $p = 1/2$  and  $\sigma_1 = \sigma_2 = \sigma$ . Assume  $|\gamma_1 + \gamma_2| \leq 2u$ . For any  $\epsilon \geq 0$ , consider the following maximization problem: