SlowGuess/NeurIPS_2023_OCR huggingface dataset:
where  $\bar{d}_{\infty}^{\mathrm{G}}$  is the distance (6) of the average steady-state student; we refer to Appendix C for the derivation of the above result. The expression of  $\bar{d}_{\infty}^{\mathrm{G}}$  includes a surprising batch-size effect: for a fixed value of the cost  $C$ , greedy attacks become more effective for batches of decreasing sizes, as the weights get closer to  $w^{*}$  while perturbations become smaller (see Eq. (45)). This effect is displayed in Fig. 2-A, which shows  $d_{\mu}$  vs SGD step for greedy attacks, and for  $P = 1, 10$ . Note that only the perturbed training phase is shown, so  $d_0 = 0$  as the student converges to the teacher function during the preceding clean training phase. The black dashed lines correspond to the steady-state values given by (13), which are in very good agreement with the empirical averages (up to a finite- $\eta$  effect) $^2$ . We also observe that  $d_{\mu}$  is characterized by fluctuations with amplitude that decreases as the batch size increases. A consequence of such fluctuations is that the average steady-state accuracy of the student undergoes a smooth transition. The transition becomes sharper as the batch size increases, and in the limit  $P \to \infty$  it converges to the prediction (11); see Fig. 2-B, which shows numerical evaluations of  $A_{\infty}(C, P)$  averaged over multiple data streams. In our experiments, we draw the teacher weights elements as i.i.d. samples from  $\mathcal{N}(0, 1)$  and normalize them so that  $|w^{\mathrm{t}}|^2 = D$ , while  $w^{*} = -w^{\mathrm{t}}$ .
Remark. In order to derive the result of Eq. (13), we have set  $\tilde{\gamma} = D / \eta$  so that  $\bar{d}_{\infty}^{\mathrm{G}} \to d_{\infty}^{\mathrm{opt}}$  for  $P \to \infty$  (see Appendix B.1). This guarantees the steady-state optimality of the greedy action policy in the large batch limit. Note that  $\tilde{\gamma}$  coincides with the timescale of the linear TSA problem, and it lends itself to a nice interpretation: in the greedy setting, where the horizon ends after the second training step, the future is optimally weighted by a factor proportional to the decorrelation time of the weights. Optimality for  $P$  finite is not guaranteed, though we observe numerically that  $\tilde{\gamma}$  approximately matches our choice even for  $P = 1, 10$ . We use again  $\tilde{\gamma} = D / \eta$  to obtain the theoretical results of the next two sections.
2We shall remark that the quantities  $\bar{d}_{\infty}^{\mathrm{G}}$  and  $\bar{d}_{\infty} = \lim_{\mu \to \infty}\langle d_{\mu}\rangle$ , the average over multiple data streams shown in Fig. 2-A, can, in general, differ, and they only coincide necessarily in the deterministic limit  $P\rightarrow \infty$ .
3.2.1 Sample-specific perturbations
The batch-size effect observed for greedy attacks can be explained as a signature of the batch average appearing in the policy function (12). When dealing with a single data point at a time, the attacker has precise control over the data labels, designing sample-specific perturbations. For batches with multiple samples, instead, perturbations are designed to perform well on average and do not exploit the fluctuations within the data stream. As a result, attacks become less effective, increasing in magnitude and leading to a smaller average steady-state distance between the student and the teacher function. A straightforward solution to this problem consists of applying sample-specific perturbations, thus using a multi-dimensional control  $a \in \mathbb{R}^P$ , which, however, results in a higher computational cost. This can be achieved by using the policy function (12) independently for each sample, so that the  $p$ -th element of  $a$  is given by
$$
a _ {p} ^ {\mathrm {G}} = \frac {1}{\tilde {C} D} \left(w ^ {\mathrm {s}} - w ^ {*}\right) ^ {\mathrm {T}} x _ {p} \left(w ^ {\mathrm {t}} - w ^ {*}\right) ^ {\mathrm {T}} x _ {p}. \tag {14}
$$
In Appendix C.1, we show that this strategy coincides with the optimal greedy solution for multidimensional, sample-specific control, up to corrections of order  $\eta$ . We also show that the resulting average steady state reached by the student, for normal and i.i.d. input data, is
$$
[ \text {R E S U L T} 4 ] \quad \bar {w} _ {\infty} ^ {\mathrm {s}} = w ^ {\mathrm {t}} - \bar {d} _ {\infty} ^ {\mathrm {G}} \Delta w ^ {\mathrm {t *}}, \quad \bar {d} _ {\infty} ^ {\mathrm {G}} = (C / 3 + 1) ^ {- 1}. \tag {15}
$$
Remarkably, there is no batch-size dependence in the above solution, and the average steady-state distance coincides with (13) for  $P = 1$ . This result demonstrates that precise, sample-specific greedy attacks remain effective independently of the batch size of the streaming data.
3.2.2 Mixing clean and perturbed data
In the previous section, we addressed a case of enhanced control, where batch perturbations are replaced by sample-specific ones. Here, instead, we consider a case of reduced control, where the attacker can apply the same perturbation to a fraction of the samples in each batch. This scenario can arise when the attacker faces computational limitations or environmental constraints. For example, the streaming frequency may be too high for the attacker to contaminate all samples at each time step, or, in a poisoned federated learning setting, the central server could have access to a source of clean data, beyond the reach of the attackers. Concretely, we assume that the attacker has access to a fraction  $\rho$  of the batch samples only. Therefore, training involves  $\rho P$  perturbed and  $(1 - \rho)P$  clean samples. Perturbations follow the policy function
$$
a _ {\mu} ^ {\mathrm {G}} = \frac {1}{\tilde {C} D \rho P} \sum_ {p = 1} ^ {\rho P} \Delta w _ {\mu} ^ {\mathrm {s * T}} x _ {\mu p} \Delta w ^ {\mathrm {t * T}} x _ {\mu p} + O (\eta). \tag {16}
$$
In Appendix C.2, we find the following average steady-state solution for normal and i.i.d. input data:
$$
[ \text {R E S U L T} 5 ] \quad \bar {w} _ {\infty} ^ {\mathrm {s}} = w ^ {\mathrm {t}} - \bar {d} _ {\infty} ^ {\mathrm {G}} \Delta w ^ {\mathrm {t *}}, \quad \bar {d} _ {\infty} ^ {\mathrm {G}} = \left(\left(\frac {P}{\rho P + 2}\right) C + 1\right) ^ {- 1}; \tag {17}
$$
see Fig. 7 for a comparison with empirical evaluations. Note that for large batches  $\bar{d}_{\infty}^{\mathrm{G}} \simeq (C / \rho + 1)^{-1}$ , indicating that the cost of actions effectively increases by a factor  $\rho^{-1}$ . We remark that the above result is derived under the assumption that the attacker ignores the number of clean samples used during training, so it cannot compensate for the reduced effectiveness of the attacks.
4 Empirical results
Having derived analytical results for the infinite batch and finite batch linear TSA problem, we now present empirical evidence as to how different strategies compare on different architectures. As a first question, we investigate the efficacy of greedy attacks for data streaming in batches of small size. By choosing an appropriate future weight  $\tilde{\gamma}$ , we know the greedy solution is optimal as the batch size tends to infinity. To investigate optimality in the finite batch setting, we compare greedy actions with clairvoyant actions on the same data stream, optimizing  $\tilde{\gamma}$  over a grid of possible values. By
A. Linear regression
B. Sigmoidal perceptron
C. 2-layer NN
Figure 3: Empirical results for the TSA problem with synthetic data. A, top panel: average steady-state distance vs cost of action  $C$  for the linear TSA problem and greedy attacks. The orange area represents the range of solutions of  $\bar{d}_{\infty}^{\mathrm{G}}$  (13) for  $P \in [1,10]$ . Bottom panel: average steady-state running cost of different attack strategies relative to the largest one (constant attacks), for  $P = 1$ , and  $C = 1$ . B, C: same quantities as shown in A for the perceptron and NN architectures;  $\mathrm{RL}^*$  indicates an agent that sees the final layer only. Averages were performed over 10 data streams of  $10^{5}$  batches and over the last  $10^{3}$  steps for each stream. Parameters:  $D = 10$ ,  $\eta = 0.02 \times D$ ,  $a \in [-2,3]$  (A, B),  $M = 100$ ,  $\eta = 0.02 \times D \times \sqrt{M}$ ,  $a \in [0,1]$  (C). Input elements sampled i.i.d. from  $\mathcal{P}_x = \mathcal{N}(0,1)$ .
definition, clairvoyant attacks have access to all the future information and therefore produce the best actions sequence. The scatterplot in Fig. 2-C shows greedy vs clairvoyant actions, respectively  $a^{\mathrm{G}}$  and  $a^{\mathrm{CV}}$ , on the same stream. Note that greedy actions are typically larger than their clairvoyant counterparts, except when  $a^{\mathrm{CV}}$  is very large. This is a distinctive feature of greedy strategies as they look for local optima. The clairvoyant vision allows the attacker to allocate its resources more wisely, for example by investing and paying extra perturbation costs for higher future rewards. This is not possible for the greedy attacker, as its vision is limited to the following training step. Substantially, though, the scattered points in Fig. 2-C lie along the diagonal, indicating that greedy attacks are nearly clairvoyant efficient, even for  $P = 1$ . The match between  $a^{\mathrm{G}}$  and  $a^{\mathrm{CV}}$  corresponds to almost identical greedy and clairvoyant trajectories of the student-teacher and student-target weights overlap, defined respectively as  $O_{\mu}^{\mathrm{st}} = w_{\mu}^{\mathrm{sT}}w^{\mathrm{t}} / D$  and  $O_{\mu}^{\mathrm{s*}} = w_{\mu}^{\mathrm{sT}}w^{*} / D$  (see Fig. 2-D).
Next, we investigate the steady state reached by the student under greedy attacks and the performance of the various attack strategies for three different architectures: the linear regression of Eq. (8), the sigmoidal perceptron  $\phi(x; w) = \operatorname{Erf}(z / \sqrt{2})$ , with  $\operatorname{Erf}(\cdot)$  the error function and  $z = w^{\mathrm{T}}x / \sqrt{D}$ , and the 2-layer neural network (NN)
$$
\phi (x; v, w) = \frac {1}{\sqrt {M}} \sum_ {m = 1} ^ {M} v _ {m} \operatorname {E r f} \left(z _ {m} / \sqrt {2}\right), \quad z _ {m} = w _ {m} ^ {\mathrm {T}} x / \sqrt {D}, \tag {18}
$$
with parameters  $w_{m} \in \mathbb{R}^{D}$  and  $v \in \mathbb{R}^{M}$ . We continue to draw all teacher weights from the standard normal distribution, normalizing them to have unitary self-overlap. For the perceptron we set the target weights as  $w^{*} = -w^{\mathrm{t}}$ , while for the neural network  $w_{m}^{*} = w_{m}^{\mathrm{t}}$  and  $v^{*} = -v^{\mathrm{t}}$ . The top row in Fig. 3 shows  $\bar{d}_{\infty} = \lim_{\mu \to \infty}\langle d_{\mu}\rangle$ , i.e. the average steady-state distance reached by the student versus the cost of actions  $C$  for greedy attacks. The solid lines show the results obtained from simulations<sup>3</sup>. The yellow shaded area shows  $\bar{d}_{\infty}^{\mathrm{G}}$  form Eq. (13) comprised between values of batch size  $P = 1$  (upper contour) and  $P = 10$  (lower contour). Note that the three cases show very similar behavior of  $\bar{d}_{\infty}(C,P)$  and are in excellent agreement with the linear TSA prediction.
Practically, this involves solving numerically the minimization problem in Eq. (7) on a discrete action grid in the range  $[a_{\mathrm{min}}, a_{\mathrm{max}}]$ . This is done at each SGD step  $\mu$ , and it requires simulating the transition  $\mu \rightarrow \mu + 1$  to estimate the average nefarious cost  $\mathbb{E}_x[g_{\mu + 1}^{\mathrm{ref}}]$  sampling input data from a buffer of past observations.
In order to compare the efficacy of all the attack strategies presented in Sec. 2.4, we consider the running cost, defined as  $g_{\mu} = g_{\mu}^{\mathrm{per}} + g_{\mu}^{\mathrm{nef}}$ . More precisely, we compute the steady-state average  $\bar{g}_{\infty}$  over multiple data streams. The bottom row in Fig. 3 shows this quantity for the corresponding architecture of the top row. The evaluation of the clairvoyant attack is missing for the NN architecture, as this approach becomes unfeasible for complex models due to the high non-convexity of the associated minimization problem. Similarly, reinforcement learning attacks are impractical for large architectures. This is because the observation space of the RL agent, given by the input  $x$  and the parameters of the model under attack, becomes too large to handle for neural networks. Therefore, for the NN model we used a TD3 agent that only observes the read-out weights  $v$  of the student network; hence the asterisk in  $\mathrm{RL^{*}}$ . We observe that the constant and clairvoyant strategies have respectively the highest and lowest running costs, as expected. Greedy attacks represent the second-best strategy, with an average running cost just above that of clairvoyant attacks. While we do not explicitly show it, it should also be emphasized that the computational costs of the different strategies vary widely; in particular, while greedy attacks are relatively cheap, RL attacks require longer data streams and larger computational resources, since they ignore the learning dynamics. We refer to Fig. 8, 9 in Appendix D for a comparison between greedy and RL action policies, and for a visual representation of the convergence in performance of the RL algorithm vs number of training episodes.
4.2 Experiments on real data
In this section, we explore the behaviour of online data poisoning in an uncontrolled scenario, where input data elements are not i.i.d. and are correlated with the teacher weights. Specifically, we consider two binary image classification tasks, using classes '1' and '7' from MNIST, and 'cats' and 'dogs' from CIFAR10, while attacks are performed using the greedy algorithm. We explore two different setups. First, we experiment on the MNIST task with simple architectures where all parameters are updated during the perturbed training phase. For this purpose, we consider a logistic regression (LogReg) and a simple convolutional neural network, the LeNet of [27]. Then, we address the case of transfer learning for more complex image-recognition architectures, where only the last layer is trained during the attacks, which constitutes a typical application of large pre-trained models. Specifically, we consider the visual geometry group (VGG11) [28] and residual networks (ResNet18) [29], with weights trained on the ImageNet-1k classification task. For all architectures, we fix the last linear layer to have 10 input and 1 output features, followed by the Erf activation function.
Preprocessing. For LogReg, we reduce the dimensionality of input data to 10 via PCA projection. For LeNet, images are resized from  $28 \times 28$  to  $32 \times 32$  pixels, while for VGG11 and ResNet18 images are resized from  $32 \times 32$  to  $224 \times 224$  pixels. Features are normalized to have zero mean and unit variance, and small random rotations of maximum 10 degrees are applied to each image, thus providing new samples at every step of training.
Learning. The training process follows the TSA paradigm: we train a teacher network on the given classification task, and we then use it to provide (soft) labels to a student network with the same architecture. As before, we consider two learning phases: first, the attacker is silent, and it calibrates  $\tilde{\gamma}$ . Then attacks start and the student receives perturbed data. For consistency with our teacher-student analysis, we consider the SGD optimizer with MSE loss; see Fig. 10 in Appendix D for a comparison between SGD and Adam-based learning dynamics. To keep the VGG11 and ResNet18 models in a regime where online learning is possible, we use the optimizer with positive weight decay.
The findings are presented in Fig. 4, where panels A and B represent the outcomes of the full training and transfer learning experiments, respectively. Each panel consists of two rows: the top row displays the dependence of the relative student-teacher distance on the cost of action  $C$  and batch size  $P$ . The orange area depicts  $\bar{d}_{\infty}^{\mathrm{G}}$  from Eq. (13) for  $P \in [1,10]$ , which we included for a visual comparison. The bottom row shows the corresponding average steady-state accuracy of the student. In general, all learners exhibit comparable behavior to the synthetic teacher-student setup in terms of the relationship between the average steady-state distance  $\bar{d}_{\infty}$  and  $C$ . Additionally, all experiments demonstrate a catastrophic transition in classification accuracy when the value of  $C$  surpasses a critical threshold. The impact of batch size  $P$  on the  $\bar{d}_{\infty}$  is also evident across all experiments. The effect of the weight decay used in the transfer learning setting is also visible: the value of  $\bar{d}_{\infty}$  remains larger than zero even when  $C$  is very large (and attacks are effectively absent). Similarly, for very low values of  $C$ , the relative distance remains below 1. We note that calibrating  $\tilde{\gamma}$  takes a long time for complex models updating all parameters, so we provide single-run evaluations for LeNet.
A. Full training - MNIST
B. Transfer learning - CIFAR10
Figure 4: Empirical results for the TSA problem with real data. A, top row: average steady-state distance vs cost of action  $C$ , for greedy attacks on fully trained logistic regression (LogReg) and LeNet architectures. The orange area represents the range of solutions of  $d_{\infty}^{\mathrm{G}}$  (13) for  $P \in [1, 10]$ . Bottom row: average steady-state accuracy vs  $C$  for the architectures of the top row. B: same quantities as shown in A for transfer learning experiments on VGG11 and ResNet18. Averages were performed over 10 data streams of  $10^{5}$  batches and over the last  $10^{3}$  steps for each data stream. Parameters:  $D = 10$ ,  $\eta = 0.02 \times D$  (LogReg, VGG11, ResNet18),  $\eta = 0.01$  (LeNet),  $a \in [0, 1]$ .
Understanding the robustness of learning algorithms is key to their safe deployment, and the online learning case, which we analyzed here, is of fundamental relevance for any interactive application. In this paper, we explored the case of online attacks that target data labels. Using the teacher-student framework, we derived analytical insights that shed light on the behavior of learning models exposed to such attacks. In particular, we proved that a discontinuous transition in the model's accuracy occurs when the strength of attacks exceeds a critical value, as observed consistently in all of our experiments. Our analysis also revealed that greedy attack strategies can be highly efficient in this context, especially when applying sample-specific perturbations, which are straightforward to implement when data stream in small batches.
A key assumption of our investigation is that the attacker tries to manipulate the long-run behavior of the learner. We note that, for complex architectures, this assumption may represent a computational challenge, as the attacker's action induces a slowdown in the dynamics. Moreover, depending on the context, attacks may be subject to time constraints and therefore involve transient dynamics. While we have not covered such a scenario in our analysis, the optimal control approach that we proposed could be recast in a finite-time horizon setting. We also note that our treatment of online data poisoning did not involve defense mechanisms implemented by the learning model. Designing optimal control attacks that can evade detection of defense strategies represents an important avenue for future research.
Further questions are suggested by our results. While the qualitative features of our theoretical predictions are replicated across all real-data experiments that we performed, we observed differences in robustness between different algorithms. For standard, test-time adversarial attacks, model complexity can aggravate vulnerability [30, 31], and whether this happens in the context of online attacks represents an important open question. Another topic of interest is the interaction of the poisoning dynamics with the structure of the data, and with the complexity of the task. Finally, we remark that our analysis involved sequences of i.i.d. data samples: taking into account temporal correlations and covariate shifts within the data stream constitutes yet another intriguing challenge.
Compute. We used a single NVIDIA Quadro RTX 4000 graphics card for all our experiments.
SG and GS acknowledge co-funding from Next Generation EU, in the context of the National Recovery and Resilience Plan, Investment PE1 – Project FAIR “Future Artificial Intelligence Research”. This resource was co-financed by the Next Generation EU [DM 1555 del 11.10.22].
23. Lillicrap, T. P. et al. Continuous control with deep reinforcement learning. 4th International Conference on Learning Representations, ICLR (2016).
24. Raffin, A. et al. Stable-Baselines3: Reliable Reinforcement Learning Implementations. Journal of Machine Learning Research, 1-8 (2021).
25. Moore, J. K. & van Bogert, A. D. Opty: Software for trajectory optimization and parameter identification using direct collocation. Journal of Open Source Software (2018).
26. Wächter, A. & Biegler, L. T. On the Implementation of an Interior-Point Filter Line-Search Algorithm for Large-Scale Nonlinear Programming. Mathematical Programming, 25-57 (2006).
27. LeCun, Y., Bottou, L., Bengio, Y. & Haffner, P. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 2278-2324 (1998).
28. Simonyan, K. & Zisserman, A. Very Deep Convolutional Networks for Large-Scale Image Recognition. 3rd International Conference on Learning Representations, ICLR (eds Bengio, Y. & LeCun, Y.) (2015).
29. He, K., Zhang, X., Ren, S. & Sun, J. Deep Residual Learning for Image Recognition. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 770-778 (2016).
30. Tsipras, D., Santurkar, S., Engstrom, L., Turner, A. & Madry, A. Robustness may be at odds with accuracy. Preprint, ArXiv:1805.12152 (2018).
31. Carbone, G. et al. Robustness of bayesian neural networks to gradient-based attacks. Advances in Neural Information Processing Systems, 15602-15613 (2020).
32. Bressan, A. & Piccoli, B. Introduction to the mathematical theory of control (2007).
A Attack strategies: algorithms
Table 1 summarizes the attack strategies introduced in Sec. 2.4. For clarity, we present the algorithms for data streaming in batches of size  $P = 1$ . We recall that the action value  $a^c$  of constant attacks and the greedy future weight  $\tilde{\gamma}$  can be optimized by sampling future data streams from past observations, and using the SGD update rule (5) to simulate the future dynamics. Similarly, the attacker can sample past observations to calibrate the the RL policy  $a^{\mathrm{RL}}(x,\theta)$ .