SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Algorithm 1 Online label attacks (batch size  $P = 1$ )
Constant attacks   
1: optimize  $a^{\mathrm{c}}$    
2: repeat   
3:  $a_{\mu}\gets a^{\mathrm{c}}$    
4:  $y_{\mu}^{\dagger}\leftarrow y_{\mu}^{\mathrm{t}} + (y_{\mu}^{*} - y_{\mu}^{\mathrm{t}})a_{\mu}$    
5:  $y_{\mu}^{\mathrm{s}}\gets \phi^{\mathrm{s}}(x_{\mu};\theta_{\mu}^{\mathrm{s}})$    
6:  $\theta_{\mu +1}^{\mathrm{s}}\gets \theta_{\mu}^{\mathrm{s}} - \eta \nabla_{\theta_{\mu}^{\mathrm{s}}}\mathcal{L}(y_{\mu}^{\mathrm{s}},y_{\mu}^{\dagger})$    
7: until end of stream   
Greedy attacks   
1: optimize  $\tilde{\gamma}$    
2: repeat   
3: observe clean batch  $(x,y^{\mathrm{t}})_{\mu}$  , state  $\theta_{\mu}^{\mathrm{s}}$    
4:  $a_{\mu}\gets \mathrm{argmin}\langle g_{\mu}^{\mathrm{per}} + \tilde{\gamma} g_{\mu +1}^{\mathrm{ref}}\rangle$    
5:  $y_{\mu}^{\dagger}\leftarrow y_{\mu}^{\mathrm{t}} + (y_{\mu}^{*} - y_{\mu}^{\mathrm{t}})a_{\mu}$    
6:  $y_{\mu}^{\mathrm{s}}\gets \phi^{\mathrm{s}}(x_{\mu};\theta_{\mu}^{\mathrm{s}})$    
7:  $\theta_{\mu +1}^{\mathrm{s}}\leftarrow \theta_{\mu}^{\mathrm{s}} - \eta \nabla_{\theta_{\mu}^{\mathrm{s}}}\mathcal{L}(y_{\mu}^{\mathrm{s}},y_{\mu}^{\dagger})$    
8: until end of stream   
Clairvoyant attacks   
1: observe state  $\theta_1^s$  , stream  $S = \{(x,y^{\mathrm{t}})_\mu \}_{\mu = 1}^T$    
2: find  $\{a_\mu \}_{\mu = 1}^T = \operatorname {argmin}G(S,\theta_1^s)$    
3: repeat   
4:  $y_{\mu}^{\dagger}\leftarrow y_{\mu}^{\mathrm{t}} + (y_{\mu}^{*} - y_{\mu}^{\mathrm{t}})a_{\mu}$    
5:  $y_{\mu}^{\mathrm{s}}\gets \phi^{\mathrm{s}}(x_{\mu};\theta_{\mu}^{\mathrm{s}})$    
7:  $\theta_{\mu +1}^{\mathrm{s}}\leftarrow \theta_{\mu}^{\mathrm{s}} - \eta \nabla_{\theta_{\mu}^{\mathrm{s}}}\mathcal{L}(y_{\boldsymbol{\mu}}^{\mathrm{s}},y_{\boldsymbol{\mu}}^{\dagger})$    
8: until end of stream
B Linear TSA problem: optimal solution
In the limit of very large batches, batch averages of the nefarious cost  $g_{\mu}^{\mathrm{nef}}$  and loss  $\mathcal{L}_{\mu}$  in Eqs. (2, 5) can be approximated with averages over the input distribution. For a general architecture  $\phi$ , and indicating with t, s, and * the teacher, student, and attacker's target, respectively, we can write
$$
g _ {\mu} ^ {\mathrm {n e f}} = \frac {1}{2} \left(I _ {\mu} (\mathrm {s}, \mathrm {s}) - 2 I _ {\mu} (\mathrm {s}, *) + I _ {\mu} (*, *)\right), \tag {19}
$$
$$
\nabla_ {\theta^ {s}} \mathcal {L} _ {\mu} = \frac {1}{2} I _ {\mu} ^ {\prime} (s, s) - I _ {\mu} ^ {\prime} (s, t) (1 - a _ {\mu}) + I _ {\mu} ^ {\prime} (s, *) a _ {\mu}, \tag {20}
$$
where  $I(a,b) = \mathbb{E}_x\left[\phi^a (x)\phi^b (x)\right]$  for two models,  $a$  and  $b$ , with the same architecture but different parameters  $\theta^a,\theta^b$ , and  $I^{\prime} = \nabla_{\theta^{\mathrm{s}}}I$ . In this case, the optimal control problem (4) is deterministic, and we can write it in continuous time  $(\eta \rightarrow 0)$  as
$$
a _ {\mu} ^ {\text {o p t}} = \underset {a _ {\mu}} {\operatorname {a r g m i n}} \int_ {0} ^ {\infty} \mathrm {d} \mu \gamma^ {\mu} \left(g _ {\mu} ^ {\text {p e r}} + g _ {\mu} ^ {\text {n e f}}\right), \tag {21}
$$
where  $g_{\mu}^{\mathrm{per}} = \tilde{C} a_{\mu}^{2} / 2$ . Note that  $\mu$  now is a continuous time index, and the above minimization problem considers all functions  $a_{\mu}\in [a_{\mathrm{min}},a_{\mathrm{max}}]$  for  $\mu \in [0,\infty)$ . In the following, we will assume that the action space is unbounded, i.e.  $a_{\mu}\in (-\infty ,\infty)$ , and that the elements of  $x$  are i.i.d. samples with zero mean and variance  $\sigma^2$ . In the linear TSA problem, where  $\phi (x;w) = w^{\mathrm{T}}x / \sqrt{D}$ , the above Eqs. (19, 20) simplify to
$$
g _ {\mu} ^ {\mathrm {n e f}} = \frac {\sigma^ {2}}{2 D} \left| \Delta w _ {\mu} ^ {\mathrm {s} *} \right| ^ {2}, \quad \nabla_ {w ^ {\mathrm {s}}} \mathcal {L} _ {\mu} = \frac {\sigma^ {2}}{D} \left(\Delta w _ {\mu} ^ {\mathrm {s t}} + \Delta w ^ {\mathrm {t} *} a _ {\mu}\right), \tag {22}
$$
where  $\Delta w^{ab} = w^a -w^b$  , and  $|\cdot |^2$  the squared 2-norm. We also find
$$
\tilde {C} = \frac {\sigma^ {2}}{D} \left| \Delta w ^ {\mathrm {t} *} \right| ^ {2} C. \tag {23}
$$
We shall recall that in the TSA problem each batch is used only once during training. This assumption guarantees that inputs  $x_{\mu}$  and weights  $w_{\mu}^{\mathrm{s}}$  are uncorrelated, a necessary condition to derive the above expressions. We solve this problem with a Lagrangian-based approach: the Lagrangian  $L$  is obtained by imposing the constraint  $\dot{w}_{\mu}^{\mathrm{s}} = -\nabla_{w^{\mathrm{s}}} \mathcal{L}_{\mu}$ , at all times, with the co-state variables  $\lambda_{\mu} \in \mathbb{R}^{D}$ :
$$
L = \int_ {0} ^ {\infty} \mathrm {d} \mu \gamma^ {\mu} \left(g _ {\mu} ^ {\text {p e r}} + g _ {\mu} ^ {\text {n e f}}\right) - \lambda_ {\mu} ^ {\mathrm {T}} \left(\dot {w} _ {\mu} ^ {\mathrm {s}} + \sigma^ {2} \left(\Delta w _ {\mu} ^ {\mathrm {s t}} + \Delta w ^ {\mathrm {t *}} a _ {\mu}\right) / D\right). \tag {24}
$$
The Pontryagin's necessary conditions for optimality  $\partial_{a_{\mu}}L = 0$ ,  $[\partial_{w_{\mu}^{\mathrm{s}}} - \partial_{\mu}\partial_{\dot{w}_{\mu}^{\mathrm{s}}}]L = 0$ , and  $\partial_{\lambda_{\mu}}L = 0$  give the following set of equations coupling  $a_{\mu}, w_{\mu}^{\mathrm{s}}$ , and  $\lambda_{\mu}$ :
$$
\gamma^ {\mu} \tilde {C} a _ {\mu} - \lambda_ {\mu} ^ {\mathrm {T}} \sigma^ {2} \Delta \omega^ {\mathrm {t *}} = 0,
$$
$$
\gamma^ {\mu} \sigma^ {2} \Delta \omega_ {\mu} ^ {\mathrm {s *}} / D + \dot {\lambda} _ {\mu} - \sigma^ {2} \lambda_ {\mu} = 0,
$$
$$
\dot {w} _ {\mu} ^ {\mathrm {s}} + \sigma^ {2} \left(\Delta w _ {\mu} ^ {\mathrm {s t}} + \Delta w ^ {\mathrm {t} *} a _ {\mu}\right) / D = 0. \tag {25}
$$
The above system of ODEs can be solved by specifying the initial condition  $w_0^{\mathrm{s}}$  and using the transversality condition  $\lim_{\mu \to \infty} \lambda_{\mu} = 0$ . At steady state, we obtain the following equation for  $w_{\infty}^{\mathrm{s}}$ :
$$
\Delta w _ {\infty} ^ {\mathrm {s t}} + \sigma^ {2} \Delta w _ {\infty} ^ {\mathrm {s * T}} \Delta w ^ {\mathrm {t *}} \Delta w ^ {\mathrm {t *}} / \tilde {C D} = 0, \tag {26}
$$
which admits only one (optimal) solution. We find it as a linear combination of  $w^{\mathrm{t}}$  and  $w^{*}$ :
$$
w _ {\infty} ^ {\mathrm {s}} = w ^ {\mathrm {t}} - a _ {\infty} ^ {\mathrm {o p t}} \Delta w ^ {\mathrm {t *}}, \quad a _ {\infty} ^ {\mathrm {o p t}} = (C + 1) ^ {- 1}. \tag {27}
$$
From the definition of  $d_{\mu}$  (6), it follows that
$$
d _ {\infty} ^ {\text {o p t}} = a _ {\infty} ^ {\text {o p t}}. \tag {28}
$$
As a final remark, we observe that the same procedure can in principle be applied to TSA problems with non-linear architectures, as long as an explicit expression for  $I(a,b)$  is available. However, one may find multiple steady-state solutions, and obtaining guarantees for optimality is often hard [32].
B.1 Optimal greedy solution
For the linear TSA problem, and in the limit of large batches, the discrete-time equation (7) for greedy actions simplifies to
$$
a _ {\mu} ^ {\mathrm {G}} = \underset {a _ {\mu}} {\operatorname {a r g m i n}} \left[ \frac {1}{2} \tilde {C} a _ {\mu} ^ {2} + \tilde {\gamma} \frac {\sigma^ {2}}{2 D} \left| w _ {\mu + 1} ^ {\mathrm {s}} \left(a _ {\mu}\right) - w ^ {*} \right| ^ {2} \right]. \tag {29}
$$
The expression of  $w_{\mu + 1}^{\mathrm{s}}(a_{\mu})$  is given by the update rule, which can be conveniently written as
$$
w _ {\mu + 1} ^ {s} = v _ {1} + v _ {2} a _ {\mu}, \tag {30}
$$
where  $v_{1} = w_{\mu}^{\mathrm{s}} - \eta \sigma^{2}\Delta w_{\mu}^{\mathrm{st}} / D$ , and  $v_{2} = -\eta \sigma^{2}\Delta w^{\mathrm{t*}} / D$ . The solution of (29) gives the greedy policy function
$$
a _ {\mu} ^ {\mathrm {G}} = \frac {w ^ {\mathrm {T}} v _ {2} - v _ {1} ^ {\mathrm {T}} v _ {2}}{\frac {\tilde {C} D}{\tilde {\gamma}} + | v _ {2} | ^ {2}}. \tag {31}
$$
The right-hand side of the above equation depends on the future weighting factor  $\tilde{\gamma}$ . Requiring that the (greedy) update rule  $w_{\mu + 1}^{\mathrm{s}} = v_1 + v_2 a_\mu^{\mathrm{G}}$  gives the optimal steady-state solution of Eq. (27), and using the explicit expression of  $\tilde{C}$  (23), one finds the optimal weight
$$
\tilde {\gamma} = D / \sigma^ {2} \eta . \tag {32}
$$
B.2 Accuracy vs cost of action
In Sec. 3.1, we addressed the case of attacks in a classification context, with class labels given by the sign of  $\phi^{\mathrm{t}}(x)$ , and where the attacker aims to swap the labels, so  $\phi^{*} = -\phi^{\mathrm{t}}$ . A pointwise estimate of the accuracy is given by
$$
S _ {\mu} (x) = \frac {1}{2} \left(\operatorname {s i g n} \left(\phi_ {\mu} ^ {\mathrm {s}} (x)\right) + \operatorname {s i g n} \left(\phi^ {\mathrm {t}} (x)\right)\right), \tag {33}
$$
and the system dynamics can be characterized by the accuracy of the student, defined as  $A_{\mu}(C,P) = \mathbb{E}_x[S_\mu (x)]$ , with  $\mathbb{E}_x$  the average over the input distribution. For the linear TSA problem, we have  $w^{*} = -w^{\mathrm{t}}$ , and, consistently with the definition of  $d_{\mu}$  (6),
$$
w _ {\mu} ^ {\mathrm {s}} = w ^ {\mathrm {t}} + d _ {\mu} \left(w ^ {*} - w ^ {\mathrm {t}}\right) = \left(1 - 2 d _ {\mu}\right) w ^ {\mathrm {t}}. \tag {34}
$$
It follows that  $S_{\mu}(x) = 1$  for  $d_{\mu} \leq 0.5$ , and  $S_{\mu}(x) = 0$  otherwise. In the deterministic limit of large batches, the optimal steady-state solution (27) gives
$$
S _ {\infty} (x) = \frac {1}{2} \left(\operatorname {s i g n} \left((C - 1) w ^ {t ^ {T}} x\right) + \operatorname {s i g n} \left(w ^ {t ^ {T}} x\right)\right). \tag {35}
$$
We then have  $S_{\infty} = 0$  for  $C < 1$ , and  $S_{\infty} = 0$  otherwise, regardless of the value of  $x$ , giving
$$
A _ {\infty} (C) = 1 - H (C - 1), \tag {36}
$$
with  $H(\cdot)$  the Heaviside step function.
C Stochastic control with greedy attacks
When the batch size  $P$  is finite, the attacker's optimal control problem is stochastic and has no straightforward solution, even for the simplest case of the linear TSA problem. It is however possible to find the average steady-state solution for greedy attacks, as we show in this Appendix. We derived the greedy policy in B.1 for  $P \to \infty$ , and for i.i.d. input elements with zero mean and variance  $\sigma^2$ . For a general  $P$ , the same expression (31) holds, with  $v_{1}$  and  $v_{2}$  now given by
$$
v _ {1} = w _ {\mu} ^ {\mathrm {s}} - \eta \frac {1}{D P} \sum_ {p = 1} ^ {P} \left(\Delta w _ {\mu} ^ {\mathrm {s t T}} x _ {\mu p}\right) x _ {\mu p},
$$
$$
v _ {2} = - \eta \frac {1}{D P} \sum_ {p = 1} ^ {P} \left(\Delta w ^ {\mathrm {t} * ^ {\mathrm {T}}} x _ {\mu p}\right) x _ {\mu p}, \tag {37}
$$
where  $\Delta w^{ab} = w^a -w^b$ . The first-order expansion of  $a_{\mu}^{\mathrm{G}}$  in powers of  $\eta$  reads
$$
a _ {\mu} ^ {\mathrm {G}} = \frac {1}{\tilde {C} D P} \sum_ {p = 1} ^ {P} \Delta w _ {\mu} ^ {\mathrm {s * T}} x _ {\mu p} \Delta w ^ {\mathrm {t * T}} x _ {\mu p} + O (\eta), \tag {38}
$$
where we have used  $\tilde{\gamma} = D / \sigma^2\eta$ , which guarantees that in the limit  $P\to \infty$  we recover the optimal solution (27). We can use the above action policy in the SGD update rule (30) and take the average over data streams to get the steady-state equation
$$
\Delta \bar {w} _ {\infty l} ^ {\mathrm {s t}} = \frac {1}{\tilde {C D P} ^ {2}} \left(P T _ {1 l} + P (P - 1) T _ {\infty l}\right), \tag {39}
$$
where we have imposed  $\bar{w}_{\mu}^{\mathrm{s}} = \bar{w}_{\mu +1}^{\mathrm{s}} = \bar{w}_{\infty}^{\mathrm{s}}$  for  $\mu \to \infty$ , and where
$$
T _ {1 l} = \mathbb {E} _ {x} \left[ \left(\Delta \bar {w} _ {\infty} ^ {\mathrm {s} * \mathrm {T}} x \Delta w ^ {\mathrm {t} * \mathrm {T}} x\right) \left(\Delta w ^ {\mathrm {t} * \mathrm {T}} x x _ {l}\right) \right], \tag {40}
$$
$$
T _ {\infty l} = \mathbb {E} _ {x \neq x ^ {\prime}} \left[ \left(\Delta \bar {w} _ {\infty} ^ {\mathrm {s} * ^ {\mathrm {T}}} x \Delta w ^ {\mathrm {t} * ^ {\mathrm {T}}} x\right) \left(\Delta w ^ {\mathrm {t} * ^ {\mathrm {T}}} x ^ {\prime} x _ {l} ^ {\prime}\right) \right]. \tag {41}
$$
Note that for  $P = 1$  and  $P \to \infty$  the right-hand side of (39) only involves  $T_{1l}$  and  $T_{\infty l}$ , respectively (hence the labels). So far, we have assumed that the input elements are i.i.d. samples from  $\mathcal{P}_x$  with first two moments  $m_1 = 0$ , and  $m_2 = \sigma^2$ . We also recall that  $x_{\mu}$  and  $w_{\mu}^{\mathrm{s}}$  are uncorrelated, as each
Figure 5: Average steady-state greedy action.  $\bar{a}_{\infty}^{\mathrm{G}}$  vs cost of actions  $C$  for the linear TSA problem with  $x \sim \mathcal{N}(0,1)$ . The orange area represents the range of solutions (45) for  $P \in [1,10]$ . The red and blue lines show empirical evaluations for  $P = 1$  and  $P = 10$ , respectively. Parameters:  $D = 10$ ,  $\eta = 10^{-4} \times D$ ,  $a \in [-2,3]$ . Averages performed over 10 data streams of  $2^{4}$  batches, and over the last  $10^{3}$  steps in each stream.
batch is used only once in the TSA problem. We now further assume that input elements have third and fourth moments  $m_3 = 0$ , and  $m_4 < \infty$ . It is then immediate to verify that  $T_{1l} = T_{1l}(m_2, m_4)$  and  $T_{\infty l} = T_{\infty l}(m_2)$ , showing that finite batch-size effects depend on the kurtosis of the input distribution. Specifically, we find
$$
\begin{array}{l} T _ {1 l} = (m _ {4} - 3 m _ {2} ^ {2}) \Delta w _ {l} ^ {\mathrm {t *}} ^ {2} \Delta \bar {w} _ {\infty l} ^ {\mathrm {s}} + m _ {2} ^ {2} | \Delta w ^ {\mathrm {t *}} | ^ {2} \Delta \bar {w} _ {\infty l} ^ {\mathrm {s}} + 2 m _ {2} ^ {2} \Delta w ^ {\mathrm {t *}} ^ {\mathrm {T}} \Delta \bar {w} _ {\infty} ^ {\mathrm {s}} \Delta w _ {l} ^ {\mathrm {t *}}, \\ T _ {\infty l} = m _ {2} ^ {2} \Delta w ^ {\mathrm {t * T}} \Delta \bar {w} _ {\infty} ^ {* \mathrm {s}} \Delta w _ {l} ^ {\mathrm {t *}}. \tag {42} \\ \end{array}
$$
For  $\mathcal{P}_x = \mathcal{N}(0,1)$ , the first term in  $T_{1l}$  disappears, and the above expressions simplify to
$$
T _ {1 l} = | \Delta w ^ {\mathrm {t *}} | ^ {2} \Delta \bar {w} _ {\infty l} ^ {* \mathrm {s}} + 2 \Delta w ^ {\mathrm {t *}} ^ {\mathrm {T}} \Delta \bar {w} _ {\infty} ^ {* \mathrm {s}} \Delta w _ {l} ^ {\mathrm {t *}},
$$
$$
T _ {\infty l} = \Delta w ^ {t *} ^ {T} \Delta \bar {w} _ {\infty} ^ {* s} \Delta w _ {l} ^ {t *}. \tag {43}
$$
Substituting in Eq. (39) the expressions of  $T_{1l}$ ,  $T_{\infty l}$ , and  $\tilde{C}$  from (23), and looking for a solution of  $\bar{w}_{\infty}^{\mathrm{s}}$  as a linear combination of  $w^{\mathrm{t}}$  and  $w^{*}$ , we find
$$
\bar {w} _ {\infty} ^ {\mathrm {s}} = w ^ {\mathrm {t}} - \bar {d} _ {\infty} ^ {\mathrm {G}} \Delta w ^ {\mathrm {t *}}, \quad \bar {d} _ {\infty} ^ {\mathrm {G}} = \left(\left(\frac {P}{P + 2}\right) C + 1\right) ^ {- 1}. \tag {44}
$$
Note that  $\bar{d}_{\infty}^{\mathrm{G}}$  is the relative distance (6) of the student with parameters  $\bar{w}_{\infty}^{\mathrm{s}}$ . From this result, we can easily find the mean steady-state greedy action by averaging (38) over data streams, giving
$$
\bar {a} _ {\infty} ^ {\mathrm {G}} = \frac {f (P)}{f (P) C + 1}, \quad f (P) = P / (P + 2). \tag {45}
$$
Fig. 5 shows a comparison of the above result with empirical evaluations, which are in excellent agreement. Together, (44) and (45) demonstrate that greedy attacks become more efficient as the batch size decreases, reducing the distance between the student and the target while using smaller perturbations. We remark that this result is derived using the expression of  $\tilde{\gamma}$  from the  $P\to \infty$  analysis, as in Eq. (32). In our experiments, this value is near-optimal also for  $P$  finite for linear learners, while for non-linear architectures it provides a good first guess during the calibration phase.
C.1 Sample-specific perturbations
In this section, we address the case where the attacker has finer control over the data labels and can apply sample-specific perturbations, rather than batch-specific ones. The control variable  $a$  is  $P$ -dimensional, with  $P$  the size of the streaming batches, and each label  $y_{p}^{\mathrm{t}}$  is perturbed as
$$
y _ {p} ^ {\dagger} = y _ {p} ^ {\mathrm {t}} \left(1 - a _ {p}\right) + y _ {p} ^ {*} a _ {p}, \tag {46}
$$
while the perturbation cost is  $g_{\mu}^{\mathrm{per}} = \tilde{C} |a_{\mu}|^{2} / 2P$ . We can write the SGD update rule as
$$
w _ {\mu + 1} ^ {\mathrm {s}} = \frac {1}{P} \sum_ {p = 1} ^ {P} v _ {1 p} + v _ {2 p} a _ {\mu p}, \tag {47}
$$
where now
$$
v _ {1 p} = w _ {\mu} ^ {\mathrm {s}} - \eta \left(\Delta w _ {\mu} ^ {\mathrm {s t}} ^ {\mathrm {T}} x _ {\mu p}\right) x _ {\mu p} / D,
$$
$$
v _ {2 p} = - \eta \left(\Delta w _ {\mu} ^ {\mathrm {t *}} ^ {\mathrm {T}} x _ {\mu p}\right) x _ {\mu p} / D. \tag {48}
$$
Figure 6: Relative distance with sample-specific attacks. Student-teacher distance vs SGD step  $\mu$  for the linear TSA problem with  $x\sim \mathcal{N}(0,1)$ . The light-red and blue lines show a single realization of the dynamics for batch-specific control. The dark-red line shows the same quantity for sample-specific, multi-dimensional control. Parameters:  $C = 1$ ,  $D = 10$ ,  $\eta = 10^{-2}\times D$ ,  $a\in [-2,3]$ .
We use again the notation  $\Delta w^{ab} = w^a -w^b$ . Note that (47) reduces to (30) when the control variable is one-dimensional and batch-specific, i.e.  $a_{\mu p} = a_{\mu}\forall p$ . The minimization problem (7) for greedy actions now consists of  $P$  coupled equations, one for each control variable  $a_{\mu p}$ . For i.i.d. input elements with zero mean and variance  $\sigma^2$ , we find
$$
a _ {\mu p} ^ {\mathrm {G}} = \underset {a _ {\mu p}} {\operatorname {a r g m i n}} \left[ \frac {1}{2} \tilde {C} | a _ {\mu} | ^ {2} + \tilde {\gamma} \frac {\sigma^ {2}}{2 D} | w _ {\mu + 1} ^ {\mathrm {s}} (a _ {\mu}) - w ^ {*} | ^ {2} \right], \tag {49}
$$
and the first-order conditions lead to
$$
\tilde {C} a _ {\mu p} + \tilde {\gamma} \frac {\sigma^ {2}}{D} \left(\frac {1}{P} \sum_ {j = 1} ^ {P} v _ {1 j} ^ {\mathrm {T}} + a _ {\mu j} v _ {2 j} ^ {\mathrm {T}} - w ^ {* \mathrm {T}}\right) v _ {2 p} = 0. \tag {50}
$$
A straightforward and explicit solution can be found by considering only the first-order terms in  $\eta$ , effectively decoupling the above system of equations. With  $\tilde{\gamma} = D / \sigma^2$ , we obtain
$$
a _ {\mu p} ^ {\mathrm {G}} \simeq \frac {1}{\tilde {C} D} \Delta w _ {\mu} ^ {\mathrm {s * T}} x _ {\mu p} \Delta w ^ {\mathrm {t * T}} x _ {\mu p}, \tag {51}
$$
which coincides with the  $P = 1$  solution for batch-specific greedy attacks (38). We can now use this policy in the update rule (47) to investigate the steady state of the system. Once more, we average across data streams with i.i.d. data sampled from  $\mathcal{P}_x = \mathcal{N}(0,1)$  and impose that the average student weights  $\bar{w}_{\mu}^{\mathrm{s}}$  reach a steady state for  $\mu \to \infty$ . This leads to the following  $P$ -independent solution
$$
\bar {w} _ {\infty} ^ {\mathrm {s}} = w ^ {\mathrm {t}} - \bar {d} _ {\infty} ^ {\mathrm {G}} \Delta w ^ {\mathrm {t *}}, \quad \bar {d} _ {\infty} ^ {\mathrm {G}} = \left(C / 3 + 1\right) ^ {- 1}. \tag {52}
$$
Fig. 6 shows empirical evaluations confirming this result. Finally, averaging the policy function (51) across data streams and using the above result for  $\bar{w}_{\infty}^{\mathrm{s}}$  we find
$$
\bar {a} _ {\infty p} ^ {\mathrm {G}} \simeq \frac {1 / 3}{C / 3 + 1}. \tag {53}
$$
C.2 Mixing clean and perturbed data
The assumption that the attacker can perturb every data sample in each batch may not always be satisfied. For example, in a federated learning scenario, the central server coordinating the training may have access to a trusted source of clean data that is inaccessible to the attacker. Moreover, the attacker may face a limited computational budget and only be able to process a fraction of samples in the data stream at each timestep. Here, we consider the case where only a fraction  $\rho$  of the  $P$  samples in each batch is perturbed by the attacker. Moreover, we assume that the attacker ignores the amount of clean samples used for training. Precisely, the attacker considers the update rule
$$
\begin{array}{l} \omega_ {\mu + 1} ^ {\mathrm {s}} = \omega_ {\mu} ^ {\mathrm {s}} - \eta \nabla_ {\omega_ {\mu} ^ {\mathrm {s}}} \mathcal {L} _ {\mu} \left(B _ {\mu} ^ {\rho P \dagger}\right) \\ = \omega_ {\mu} ^ {\mathrm {s}} - \frac {\eta}{D \rho P} \left(\sum_ {p = 1} ^ {\rho P} \left(\Delta w _ {\mu} ^ {\mathrm {s t}} {} ^ {\mathrm {T}} x _ {\mu p}\right) x _ {\mu p} + a _ {\mu} \sum_ {p = 1} ^ {\rho P} \left(\Delta w _ {\mu} ^ {\mathrm {t *}} {} ^ {\mathrm {T}} x _ {\mu p}\right) x _ {\mu p}\right), \tag {54} \\ \end{array}
$$
with  $\rho P$  perturbed samples, while training follows
$$
\begin{array}{l} \omega_ {\mu + 1} ^ {\mathrm {s}} = \omega_ {\mu} ^ {\mathrm {s}} - \eta \rho \nabla_ {\omega_ {\mu} ^ {\mathrm {s}}} \mathcal {L} _ {\mu} \left(B _ {\mu} ^ {\rho P \dagger}\right) - \eta (1 - \rho) \nabla_ {\omega_ {\mu} ^ {\mathrm {s}}} \mathcal {L} _ {\mu} \left(B _ {\mu} ^ {(1 - \rho) P}\right) \\ = \omega_ {\mu} ^ {\mathrm {s}} - \frac {\eta}{D P} \left(\sum_ {p = 1} ^ {P} \left(\Delta w _ {\mu} ^ {\mathrm {s t}} ^ {\mathrm {T}} x _ {\mu p}\right) x _ {\mu p} + a _ {\mu} \sum_ {p = 1} ^ {\rho P} \left(\Delta w _ {\mu} ^ {\mathrm {t *}} ^ {\mathrm {T}} x _ {\mu p}\right) x _ {\mu p}\right), \tag {55} \\ \end{array}
$$
Figure 7: Mixing clean and poisoned samples. Average steady-state distance vs fraction  $\rho$  of poisoned samples, under greedy attacks. The dashed line shows the estimate of Eq. (58), and the red dots display the associated empirical evaluations. Blue crosses show empirical results obtained for a logistic regression training on MNIST digits. Parameters:  $C = 1$ ,  $P = 100$ ,  $D = 10$ ,  $\eta = 0.1$ ,  $a \in [-2,3]$ . Averages performed over 10 data streams of  $10^{4}$  batches, and over the last  $10^{3}$  steps in each stream.
thus involving  $\rho P$  perturbed samples and  $(1 - \rho)P$  clean samples. We recall that  $\Delta w^{ab} = w^a -w^b$ , and that the stream has input data drawn i.i.d. from  $\mathcal{P}_x$ . Following (54), the greedy policy (38) in this case reads
$$
a _ {\mu} ^ {\mathrm {G}} = \frac {1}{\tilde {C} D \rho P} \sum_ {p = 1} ^ {\rho P} \Delta w _ {\mu} ^ {\mathrm {s * T}} x _ {\mu p} \Delta w ^ {\mathrm {t * T}} x _ {\mu p} + O (\eta). \tag {56}
$$
We can now substitute the above expression into (55) to investigate the steady state of the system. As before, we can obtain a steady-state condition for  $\bar{w}_{\infty}^{\mathrm{s}}$  by averaging across data streams and imposing  $\bar{w}_{\mu}^{\mathrm{s}} = \bar{w}_{\mu +1}^{\mathrm{s}} = \bar{w}_{\infty}^{\mathrm{s}}$  for  $\mu \to \infty$ . We find
$$
\Delta \bar {w} _ {\infty l} ^ {\mathrm {s t}} = \frac {1}{\tilde {C} D \rho P ^ {2}} \left(\rho P T _ {1 l} + \rho P (\rho P - 1) T _ {\infty l}\right), \tag {57}
$$
with  $T_{1l}$  and  $T_{\infty l}$  as in (43) for  $\mathcal{P}_x = \mathcal{N}(0,1)$ . It is then immediately verified that
$$
\bar {w} _ {\infty} ^ {\mathrm {s}} = w ^ {\mathrm {t}} - \bar {d} _ {\infty} ^ {\mathrm {G}} \Delta w ^ {\mathrm {t *}}, \quad \bar {d} _ {\infty} ^ {\mathrm {G}} = \left(\left(\frac {P}{\rho P + 2}\right) C + 1\right) ^ {- 1}. \tag {58}
$$
Note that for large batches ( $\rho P \gg 2$ ) the above result reduces to  $\bar{d}_{\infty}^{\mathrm{G}} \approx (C / \rho + 1)^{-1}$ , and it has a simple and intuitive interpretation: when perturbations contaminate a fraction  $\rho$  of the batch samples only, the cost of actions effectively increases by a factor  $\rho^{-1}$ . Fig. 7 shows the average steady-state distance reached by the student model as a function of  $\rho$ , demonstrating that empirical evaluations using synthetic data perfectly match the theoretical estimate of Eq. (58). We also performed real data experiments using MNIST images (as in Sec. 4.2), which again exhibit a good qualitative agreement with our theoretical result.
D Supplementary figures
A. Greedy policy