SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Riccardo G. Margiotta* Sebastian Goldt Guido Sanguinetti
International School for Advanced Studies, Trieste, Italy
Machine learning models are famously vulnerable to adversarial attacks: small ad-hoc perturbations of the data that can catastrophically alter the model predictions. While a large literature has studied the case of test-time attacks on pre-trained models, the important case of attacks in an online learning setting has received little attention so far. In this work, we use a control-theoretical perspective to study the scenario where an attacker may perturb data labels to manipulate the learning dynamics of an online learner. We perform a theoretical analysis of the problem in a teacher-student setup, considering different attack strategies, and obtaining analytical results for the steady state of simple linear learners. These results enable us to prove that a discontinuous transition in the learner's accuracy occurs when the attack strength exceeds a critical threshold. We then study empirically attacks on learners with complex architectures using real data, confirming the insights of our theoretical analysis. Our findings show that greedy attacks can be extremely efficient, especially when data stream in small batches.
Adversarial attacks pose a remarkable threat to modern machine learning (ML), as models based on deep networks have proven highly vulnerable to such attacks. Understanding adversarial attacks is therefore of paramount importance, and much work has been done in this direction; see [1, 2] for reviews. The standard setup of adversarial attacks aims to change the predictions of a trained model by applying a minimal perturbation to its inputs. In the data poisoning scenario, instead, inputs and/or outputs of the training data are corrupted by an attacker whose goal is to force the learner to get as close as possible to a "nefarious" target model, for example to include a backdoor [3, 4]. Data poisoning has also received considerable attention that has focused on the offline setting, where models are trained on fixed datasets [5]. An increasing number of real-world applications instead require that machine learning models are continuously updated using streams of data, often relying on transfer-learning techniques. In many cases, the streaming data can be subject to adversarial manipulations. Examples include learning from user-generated data (GPT-based models) and collecting information from the environment, as in e-commerce applications. Federated learning constitutes yet another important example where machine learning models are updated over time, and where malicious nodes in the federation have private access to a subset of the data used for training.
In the online setting, the attacker intercepts the data stream and modifies it to move the learner towards the nefarious target model, see Fig. 1-A for a schematic where attacks change the labels. In this streaming scenario, the attacker needs to predict both the future data stream and model states, accounting for the effect of its own perturbations, so as to decide how to poison the current data batch. Finding the best possible attack policy for a given data batch and learner state can be formalized as a stochastic optimal control problem, with opposing costs given by the magnitude of perturbations
$^{*}$ R. G. Margiotta (rimargi@sissa.it) is the corresponding author.
and the distance between learner and attacker's target [6]. The ensuing dynamics have surprising features. Take for example Fig. 1-B, where we show the accuracy of an attacked model (a logistic regression classifier classifying MNIST images) as a function of the attack strength (we define this quantity in Sec. 2.2). Even in such a simple model, we observe a strong nonlinear behavior where the accuracy drops dramatically when crossing a critical attack strength. This observation raises several questions about the vulnerability of learning algorithms to online data poisoning. What is the effect of the perturbations on the learning dynamics and long-term behavior of the model under attack? Is there a minimum perturbation strength required for the attacker to achieve a predetermined goal? And how do different attack strategies compare in terms of efficacy? We investigate these questions by analyzing the attacker's control problem from a theoretical standpoint in the teacher-student setup, a popular framework for studying machine learning models in a controlled way [7-10]. We obtain analytical results characterizing the steady state of the attacked model in a linear regression setting, and we show that a simple greedy attack strategy can perform near optimally. This observation is empirically replicated across more complex learners in the teacher-student setup, and the fundamental aspects of our theoretical results are also reflected in real-data experiments.
Main contributions
- We present a theoretical analysis of online data poisoning in the teacher-student setup, providing analytical results for the linear regression case [RESULT 1-5]. In particular, we demonstrate that a phase transition takes place when the batch size approaches infinity, signaled by a discontinuity in the accuracy of the student against the attacker's strength [RESULT 2].
- We provide a quantitative comparison between different attack strategies. Surprisingly, we find that properly calibrated greedy attacks can be as effective as attacks with full knowledge of the incoming data stream. Greedy attacks are also computationally efficient, thus constituting a remarkable threat to online learners.
- We empirically study online data poisoning on real datasets (MNIST, CIFAR10), using architectures of varying complexities including LeNet, ResNet, and VGG. We observe qualitative features that are consistent with our theoretical predictions, providing validation for our findings.
1.1 Further related work
Data poisoning attacks have been the subject of a wide range of studies. Most of this research focused on the offline (or batch) setting, where the attacker interacts with the data only once before training starts [3, 11-14]. Mei et al. notably proposed an optimization approach for offline data poisoning [15]. Wang and Chaudhuri addressed a streaming-data setting, but in the idealized scenario where the attacker has access to the full (future) data stream (the so-called clairvoyant setting), which effectively reduces the online problem to an offline optimization [16]. So far, the case of online data poisoning has received only a few contributions with a genuine online perspective. In [17], Pang and coauthors investigated attacks on online learners with poisoned batches that aim for an immediate disruptive impact rather than manipulating the long-term learning dynamics. Zhang et al. proposed an optimal control formulation of online data poisoning by following a parallel line of research, that of online teaching [18, 19], focussing then on practical attack strategies that perturb the input [6]. Our optimal control perspective is similar to that of Zhang and co-authors, though our contribution differs from theirs in various aspects. First, we consider a supervised learning setting where attacks involve data labels, and not input data. We note that attacks on the labels have a reduced computational cost compared to high-dimensional adversarial perturbations of the inputs, which makes label poisoning more convenient in a streaming scenario. Second, we provide a theoretical analysis and explicit results for the dynamics of the attacked models, and we consider non-linear predictors in our experiments. We observe that our setup is reminiscent of continual learning, which has also recently been analyzed in a teacher-student setup [20]. However, in online data poisoning the labels are dynamically changed by the attacker to drive the learner towards a desired target. In continual learning there is no entity that actively modifies the labels, and the learner simply switches from one task to another. Finally, we note that standard (static, test-time) adversarial attacks have been studied in a teacher-student framework [21].
Reproducibility. The code and details for implementing our experiments are available here.
B.
Figure 1: Problem setup and key experimental observations. A: The environment (teacher) generates a sequence of clean data with  $\mu^{\mathrm{th}}$  input  $x_{\mu}$  and label  $y_{\mu}^{\mathrm{t}}$ . The attacker perturbs the labels towards a nefarious target  $y_{\mu}^{*} = \phi^{*}(x_{\mu})$ . At each training step, the predictions  $y_{\mu}^{\mathrm{s}}$  of the learner (student) and the perturbed labels  $y_{\mu}^{\dagger}$  are used to compute the loss and update the learner parameters. B: accuracy vs attack strength of a logistic regression classifying MNIST digits 1 and 7 under online label-flipping attacks. C: relative distance  $d_{\mu}$  of the logistic learner vs SGD step  $\mu$  during the clean training phase, and for training under attacks of increasing attack strength.
2.1 The teacher, the student, and the attacker
The problem setup is depicted in Fig. 1-A. There are three entities: the teacher, the student, and the attacker. The teacher is a model of the environment, which generates a stream of clean input-output data batches. The inputs  $x \in \mathbb{R}^D$  have elements sampled from a distribution  $\mathcal{P}_x$ , while clean labels are provided by the scalar teacher function:  $y^{\mathrm{t}} = \phi^{\mathrm{t}}(x)$ . The  $\mu^{\mathrm{th}}$  batch of the stream is  $B_{\mu}^{P} = \{(x_{\mu},y_{\mu}^{\mathrm{t}})_p\}_{p = 1}^{P}$ , with  $P$  the batch size. The attacker intercepts the clean batch  $B_{\mu}^{P}$  and perturbs the labels so as to make the student learn the target function  $\phi^{*}$ . The student  $\phi_{\mu}^{\mathrm{s}}$  represents the learning model, and it updates its weights  $\theta_{\mu}^{\mathrm{s}}$  on the stream of perturbed batches via SGD, using each (perturbed) batch only once. We will consider teacher, attacker's target, and student with the same architecture  $\phi (x;\theta)$ ; they only differ by the value of their parameters, respectively  $\theta^{\mathrm{t}},\theta^{*},\theta_{\mu}^{\mathrm{s}}$ . We will refer to this Teacher-Student-Attacker setup as the TSA problem.
If the student trained directly on the input-label pairs provided by the teacher, we would recover the standard teacher-student setup that has been analyzed extensively in the literature [7-10]. The key novelty of this paper is that we consider a setup where each new batch of data is instead poisoned by the attacker with the attempt of deviating the student towards a different target function.
2.2 The attack
The goal of the attacker is for the student to predict a target label  $y^{*} = \phi^{*}(x)$  for each given input  $x$ . Target label and clean label can coincide for specific inputs, however they won't generally do so. The attacker performs an action  $a_{\mu}$  that produces perturbed labels as
$$
y _ {\mu} ^ {\dagger} = y ^ {\mathrm {t}} \left(1 - a _ {\mu}\right) + y ^ {*} a _ {\mu}, \quad a _ {\mu} \in \left[ a _ {\min }, a _ {\max } \right]. \tag {1}
$$
Note that  $a_{\mu} = 0$  implies no perturbation, while for  $a_{\mu} = 1$  perturbed and target labels coincide. We allow  $a_{\mathrm{min}} < 0$  and  $a_{\mathrm{max}} > 1$  to model attackers that under/overshoot. The attacker's action  $a_{\mu}$  thus constitutes the control variable of the TSA problem. We primarily consider the case where the same value of  $a_{\mu}$  is used for all labels in each batch, but can be different for different batches. We also explore settings where this assumption is relaxed, including the case of sample-specific perturbations, and the scenario where only a fraction of the labels in each batch is perturbed.
The aim of the attacker is to sway the student towards its target, and so minimize the nefarious cost
$$
g _ {\mu} ^ {\mathrm {n e f}} = \frac {1}{2 P} \sum_ {p = 1} ^ {P} \left(\phi_ {\mu} ^ {\mathrm {s}} \left(x _ {\mu p}\right) - \phi^ {*} \left(x _ {\mu p}\right)\right) ^ {2}, \tag {2}
$$
using small perturbations. For this reason, it also tries to minimize the perturbation cost
$$
g _ {\mu} ^ {\text {p e r}} = \frac {1}{2} \tilde {C} a _ {\mu} ^ {2}, \quad \tilde {C} = \mathcal {E} \left(\phi^ {*}\right) C, \tag {3}
$$
with  $C$  a parameter expressing the cost of actions, and  $\mathcal{E}(\phi^{*}) = \mathbb{E}_{x}\left[(\phi^{*}(x) - \phi^{\mathrm{t}}(x))^{2}\right]$  the expected squared error of the attacker's target function;  $\mathbb{E}_x$  indicates the average over the input distribution. We use the pre-factor  $\mathcal{E}(\phi^{*})$  in  $g_{\mu}^{\mathrm{per}}$  to set aside the effect of the teacher-target distance on the balance between the two costs. Note that  $C$  is inversely proportional to the intuitive notion of attack strength: low  $C$  implies a low cost for the action and therefore results in larger perturbations; the  $x$ -axis in Fig. 1-B shows the inverse of  $C$ . The attacker's goal is to find the sequence of actions  $a_{\mu}^{\mathrm{opt}}$  that minimize the total expected cost:
$$
\left\{a _ {\mu} ^ {\text {o p t}} \right\} = \underset {\left\{a _ {\mu} \right\}} {\operatorname {a r g m i n}} \mathbb {E} _ {\text {f u t}} \left[ \lim  _ {T \rightarrow \infty} G \left(\left\{a _ {\mu} \right\}, T\right)\right], \quad G = \sum_ {\mu = 1} ^ {T} \gamma^ {\mu} \left(g _ {\mu} ^ {\text {p e r}} + g _ {\mu} ^ {\text {n e f}}\right), \tag {4}
$$
where  $\mathbb{E}_{\mathrm{fut}}$  represents the average over all possible realizations of the future data stream, and  $\gamma \in (0,1)$  is the future discounting factor. In relation to  $\gamma$ , we make the crucial assumption that  $G$  is dominated by the steady-state running costs; we use  $\gamma = 0.995$  in our experiments.
2.3 SGD dynamics under attack
The perturbed batch  $B_{\mu}^{P\dagger} = \{(x_{\mu},y_{\mu}^{\dagger})_p\}_{p = 1}^{P}$  is used to update the model parameters  $\theta_{\mu}^{\mathrm{s}}$  following the standard (stochastic) gradient descent algorithm using the MSE loss function:
$$
\theta_ {\mu + 1} ^ {\mathrm {s}} = \theta_ {\mu} ^ {\mathrm {s}} - \eta \nabla_ {\theta_ {\mu} ^ {\mathrm {s}}} \mathcal {L} _ {\mu} \left(B _ {\mu} ^ {P \dagger}\right), \quad \mathcal {L} _ {\mu} = \frac {1}{2 P} \sum_ {p = 1} ^ {P} \left(\phi_ {\mu} ^ {\mathrm {s}} \left(x _ {\mu p}\right) - y _ {\mu p} ^ {\dagger}\right) ^ {2}, \tag {5}
$$
where  $\eta$  is the learning rate. We characterize the system's dynamics in terms of the relative teacher-student distance, defined as
$$
d _ {\mu} (C, P) = \left(\mathcal {E} \left(\phi_ {\mu} ^ {\mathrm {s}}\right) / \mathcal {E} \left(\phi^ {*}\right)\right) ^ {1 / 2}. \tag {6}
$$
Note that  $d = 0$  (resp.  $d = 1$ ) for a student  $\phi^s$  that perfectly predicts the clean (resp. target) labels, on average. We will split the training process into two phases: first, the student trains on clean data, while the attacker calibrates its policy. Then, after convergence of the student, attacks start and training continues on perturbed data. A typical realization of the dynamics is depicted in Fig. 1-C, which shows  $d_{\mu}$  for a logistic regression classifying MNIST digits during the clean-labels phase, and under attacks of increasing strength (decreasing values of the cost  $C$ ). The sharp transition observed in Fig. 1-B occurs for  $d_{\mu \to \infty} \approx 0.5$ , when the student is equally distant from the teacher and target.
2.4 Attack strategies
Finding the optimal actions sequence that solves the control problem (4) involves performing the average  $\mathbb{E}_{\mathrm{fut}}$ , and so it is only possible if the attacker knows the data generating process. Even then, however, optimal solutions can be found only for simple cases, and realistic settings require approximate solutions. We consider the following approaches:
- Constant attacks. The action is observation-independent and remains constant for all SGD steps:  $a_{\mu} = a^{\mathrm{c}} \forall \mu$ . This simple baseline can be optimized by sampling data streams using past observations, simulating the dynamics via Eq. (5), and obtaining numerically the value of  $a^{\mathrm{c}}$  that minimizes the simulated total expected cost.
- Greedy attacks. A greedy attacker maximizes the immediate reward. Since the attacker actions have no instantaneous effect on the learner, the greedy objective has to include the next training step. The greedy action is then given by
$$
a _ {\mu} ^ {\mathrm {G}} \left(x _ {\mu}, \theta_ {\mu} ^ {\mathrm {s}}; \tilde {\gamma}\right) = \underset {a _ {\mu}} {\operatorname {a r g m i n}} \mathbb {E} _ {x _ {\mu + 1}} \left[ g _ {\mu} ^ {\text {p e r}} + \tilde {\gamma} g _ {\mu + 1} ^ {\text {n e f}} \right]. \tag {7}
$$
The average  $\mathbb{E}_{x_{\mu +1}}$  can be estimated with input data sampled from past observations, and using (5) to obtain the next student state. Similarly to constant attacks, the greedy future weight  $\tilde{\gamma}$  can be calibrated by minimizing the simulated total expected cost using past observations.
- Reinforcement learning attacks. The attacker can use a parametrized policy function  $a(x, \theta^s)$  and optimize it via reinforcement learning (RL). This is a model free approach, and it is suitable for grey box settings. Tuning deep policies is computationally costly, however, and it requires careful calibrations. We utilize TD3 [22], a powerful RL agent that improves the deep deterministic policy gradient algorithm [23]. We employ TD3 as implemented in Stable Baselines3 [24].
- Clairvoyant attacks. A clairvoyant attacker has full knowledge of the future data stream, so it simply looks for the sequence  $\{a_{\mu}\}$  that minimizes  $G(\{a_{\mu}\}, T \gg 1)$ . Although the clairvoyant setting is not realistically plausible, it provides an upper bound for the attacker's performance in our experiments. We use Opty [25] to cast the clairvoyant problem into a standard nonlinear program, which is then solved by an interior-point solver (IPOPT) [26].
The above attack strategies are summarized as pseudocode in Appendix A.
3 Theoretical analysis
We first analyze online label attacks with synthetic data to establish our theoretical results. We will confirm our findings with real data experiments in Sec. 4.2. Solving the stochastic optimal control problem (4) is very challenging, even for simple architectures  $\phi(x,\theta)$  and assuming that  $\mathcal{P}_x$  is known. We discuss an analytical treatment in two scenarios for the linear TSA problem with unbounded action space, where
$$
\phi (x; w) = w ^ {\mathrm {T}} x / \sqrt {D}. \tag {8}
$$
3.1 Large batch size: deterministic limit
In the limit of very large batches, batch averages of the nefarious cost  $g_{\mu}^{\mathrm{nef}}$  and loss  $\mathcal{L}_{\mu}$  in Eqs. (2, 5) can be approximated with averages over the input distribution. For standardized and i.i.d. input elements, the resulting equations are
$$
g _ {\mu} ^ {\mathrm {n e f}} = \frac {1}{2 D} \left| w _ {\mu} ^ {\mathrm {s}} - w ^ {*} \right| ^ {2}, \quad \nabla_ {w ^ {\mathrm {s}}} \mathcal {L} _ {\mu} = \frac {1}{D} \left(\left(w _ {\mu} ^ {\mathrm {s}} - w ^ {\mathrm {t}}\right) + \left(w _ {\mu} ^ {\mathrm {t}} - w ^ {*}\right) a _ {\mu}\right). \tag {9}
$$
In this setting, the attacker faces a deterministic optimal control problem, which can be solved using a Lagrangian-based method. Following this procedure, we obtain explicit solutions for the steady-state student weights  $w_{\infty}^{\mathrm{s}}$  and optimal action  $a_{\infty}^{\mathrm{opt}}$ . The relative distance  $d_{\infty}^{\mathrm{opt}}$  between student and teacher then follows from Eq. (6). We find
$$
[ \text {R E S U L T} 1 ] \quad w _ {\infty} ^ {\mathrm {s}} = w ^ {\mathrm {t}} + d _ {\infty} ^ {\mathrm {o p t}} \left(w ^ {*} - w ^ {\mathrm {t}}\right), \quad d _ {\infty} ^ {\mathrm {o p t}} = a _ {\infty} ^ {\mathrm {o p t}} = (C + 1) ^ {- 1}. \tag {10}
$$
The above result is intuitive: the student learns a function that approaches the attacker's target as the cost of actions  $C$  tends to zero, and  $d_{\infty}^{\mathrm{opt}}(C = 0) = 1$ . Vice versa, for increasing values of  $C$ , the attacker's optimal action decreases, and the student approaches the teacher function.
In the context of a classification task, we can characterize the system dynamics in terms of the accuracy of the student  $A_{\mu}(C,P) = \mathbb{E}_x[S_\mu (x)]$ , with  $S_{\mu}(x) = \big(\mathrm{sign}(\phi_{\mu}^{\mathrm{s}}(x)) + \mathrm{sign}(\phi^{\mathrm{t}}(x))\big) / 2$ . For label-flipping attacks, where  $\phi^{*} = -\phi^{\mathrm{t}}$ , a direct consequence of Eq. (10) is that the steady-state accuracy of the student exhibits a discontinuous transition in  $C = 1$ . More precisely, we find
$$
\left[ \text {R E S U L T} 2 \right] \quad A _ {\infty} (C) = 1 - H (C - 1), \tag {11}
$$
with  $H(\cdot)$  the Heaviside step function. This simple result explains the behaviour observed in Fig. 1-B, where the collapse in accuracy occurs for attack strength  $C^{-1} \approx 1$ . We refer to Appendix B for a detailed derivation of results (10, 11).
3.2 Finite batch size: greedy attacks
When dealing with batches of finite size, the attacker's optimal control problem is stochastic and has no straightforward solution, even for the simple linear TSA problem. To make progress, we consider greedy attacks, which are computationally convenient and analytically tractable. While an explicit solution to (7) is generally hard to find, it is possible for the linear case of Eq. (8).
C.
Figure 2: Greedy and clairvoyant attacks in the linear TSA problem. A: relative distance vs SGD step  $\mu$  for greedy attacks on linear models for two different batch sizes  $P$ . Solid lines display the average  $\bar{d}_{\mu}$ , shaded areas cover 1std of  $d_{\mu}$ . Dashed lines show the theoretical estimate of Eqs. (13). B: steady-state average accuracy of the student vs cost of action. The dashed line shows the theoretical prediction of Eq. (11). C: scatterplot of greedy (G) vs clairvoyant (CV) actions used on the same data stream with  $P = 1$ . D: student-teacher (s-t, green) and student-target (s-*, coral) overlap vs SGD step, for the clairvoyant (dotted line) and greedy (continuous line) attacks of panel C. Parameters:  $C = 1$ ,  $a \in [-2,3]$ ,  $D = 10$ ,  $\eta = 0.02 \times D$ . Input elements sampled i.i.d. from  $\mathcal{P}_x = \mathcal{N}(0,1)$ .
We find the greedy action policy
$$
a ^ {\mathrm {G}} = \frac {1}{\tilde {C} D P} \sum_ {p = 1} ^ {P} \left(w ^ {\mathrm {s}} - w ^ {*}\right) ^ {\mathrm {T}} x _ {p} \left(w ^ {\mathrm {t}} - w ^ {*}\right) ^ {\mathrm {T}} x _ {p} + O (\eta). \tag {12}
$$
Note that  $a^{\mathrm{G}}$  decreases with the cost of actions, and  $a^{\mathrm{G}} \to 0$  for  $\tilde{C} \to \infty$ . With a policy function at hand, we can easily investigate the steady state of the system by averaging over multiple realizations of the data stream. For input elements sampled i.i.d. from  $\mathcal{P}_x = \mathcal{N}(0,1)$ , we find the average steady-state weights  $\bar{w}_{\infty}^{\mathrm{s}}$  as
$$
[ \text {R E S U L T} 3 ] \quad \bar {w} _ {\infty} ^ {\mathrm {s}} = w ^ {\mathrm {t}} + \bar {d} _ {\infty} ^ {\mathrm {G}} \left(w ^ {*} - w ^ {\mathrm {t}}\right), \quad \bar {d} _ {\infty} ^ {\mathrm {G}} = \left(\left(\frac {P}{P + 2}\right) C + 1\right) ^ {- 1}, \tag {13}
$$