Title: Attacks on Online Learners: a Teacher-Student Analysis
Venue: NeurIPS 2023
Accepted: yes
Keywords: Adversarial attacks

data poisoning

online learning

optimal control

teacher-student setup

solvable model
Abstract: Machine learning models are famously vulnerable to adversarial attacks: small ad-hoc perturbations of the data that can catastrophically alter the model predictions. While a large literature has studied the case of test-time attacks on pre-trained models, the important case of attacks in an online learning setting has received little attention so far. In this work, we use a control-theoretical perspective to study the scenario where an attacker may perturb data labels to manipulate the learning dynamics of an online learner. We perform a theoretical analysis of the problem in a teacher-student setup, considering different attack strategies, and obtaining analytical results for the steady state of simple linear learners. These results enable us to prove that a discontinuous transition in the learner's accuracy occurs when the attack strength exceeds a critical threshold. We then study empirically attacks on learners with complex architectures using real data, confirming the insights of our theoretical analysis. Our findings show that greedy attacks can be extremely efficient, especially when data stream in small batches.

Attacks on Online Learners: a Teacher-Student Analysis

Abstract

Machine learning models are famously vulnerable to adversarial attacks: small adhoc perturbations of the data that can catastrophically alter the model predictions. While a large literature has studied the case of test-time attacks on pre-trained models, the important case of attacks in an online learning setting has received little attention so far. In this work, we use a control-theoretical perspective to study the scenario where an attacker may perturb data labels to manipulate the learning dynamics of an online learner. We perform a theoretical analysis of the problem in a teacher-student setup, considering different attack strategies, and obtaining analytical results for the steady state of simple linear learners. These results enable us to prove that a discontinuous transition in the learner's accuracy occurs when the attack strength exceeds a critical threshold. We then study empirically attacks on learners with complex architectures using real data, confirming the insights of our theoretical analysis. Our findings show that greedy attacks can be extremely efficient, especially when data stream in small batches.

1 Introduction

Adversarial attacks pose a remarkable threat to modern machine learning (ML), as models based on deep networks have proven highly vulnerable to such attacks. Understanding adversarial attacks is therefore of paramount importance, and much work has been done in this direction; see [1, 2] for reviews. The standard setup of adversarial attacks aims to change the predictions of a trained model by applying a minimal perturbation to its inputs. In the data poisoning scenario, instead, inputs and/or outputs of the training data are corrupted by an attacker whose goal is to force the learner to get as close as possible to a 'nefarious' target model, for example to include a backdoor [3, 4]. Data poisoning has also received considerable attention that has focused on the offline setting, where models are trained on fixed datasets [5]. An increasing number of real-world applications instead require that machine learning models are continuously updated using streams of data, often relying on transfer-learning techniques. In many cases, the streaming data can be subject to adversarial manipulations. Examples include learning from user-generated data (GPT-based models) and collecting information from the environment, as in e-commerce applications. Federated learning constitutes yet another important example where machine learning models are updated over time, and where malicious nodes in the federation have private access to a subset of the data used for training.

In the online setting, the attacker intercepts the data stream and modifies it to move the learner towards the nefarious target model, see Fig. 1-A for a schematic where attacks change the labels. In this streaming scenario, the attacker needs to predict both the future data stream and model states, accounting for the effect of its own perturbations, so as to decide how to poison the current data batch. Finding the best possible attack policy for a given data batch and learner state can be formalized as a stochastic optimal control problem, with opposing costs given by the magnitude of perturbations

and the distance between learner and attacker's target [6]. The ensuing dynamics have surprising features. Take for example Fig. 1-B, where we show the accuracy of an attacked model (a logistic regression classifier classifying MNIST images) as a function of the attack strength (we define this quantity in Sec. 2.2). Even in such a simple model, we observe a strong nonlinear behavior where the accuracy drops dramatically when crossing a critical attack strength. This observation raises several questions about the vulnerability of learning algorithms to online data poisoning. What is the effect of the perturbations on the learning dynamics and long-term behavior of the model under attack? Is there a minimum perturbation strength required for the attacker to achieve a predetermined goal? And how do different attack strategies compare in terms of efficacy? We investigate these questions by analyzing the attacker's control problem from a theoretical standpoint in the teacher-student setup, a popular framework for studying machine learning models in a controlled way [7-10]. We obtain analytical results characterizing the steady state of the attacked model in a linear regression setting, and we show that a simple greedy attack strategy can perform near optimally. This observation is empirically replicated across more complex learners in the teacher-student setup, and the fundamental aspects of our theoretical results are also reflected in real-data experiments.

Main contributions

We present a theoretical analysis of online data poisoning in the teacher-student setup, providing analytical results for the linear regression case [RESULT 1-5]. In particular, we demonstrate that a phase transition takes place when the batch size approaches infinity, signaled by a discontinuity in the accuracy of the student against the attacker's strength [RESULT 2].

We provide a quantitative comparison between different attack strategies. Surprisingly, we find that properly calibrated greedy attacks can be as effective as attacks with full knowledge of the incoming data stream. Greedy attacks are also computationally efficient, thus constituting a remarkable threat to online learners.

We empirically study online data poisoning on real datasets (MNIST, CIFAR10), using architectures of varying complexities including LeNet, ResNet, and VGG. We observe qualitative features that are consistent with our theoretical predictions, providing validation for our findings.

1.1 Further related work

Data poisoning attacks have been the subject of a wide range of studies. Most of this research focused on the offline (or batch) setting, where the attacker interacts with the data only once before training starts [3, 11-14]. Mei et al. notably proposed an optimization approach for offline data poisoning [15]. Wang and Chaudhuri addressed a streaming-data setting, but in the idealized scenario where the attacker has access to the full (future) data stream (the so-called clairvoyant setting), which effectively reduces the online problem to an offline optimization [16]. So far, the case of online data poisoning has received only a few contributions with a genuine online perspective. In [17], Pang and coauthors investigated attacks on online learners with poisoned batches that aim for an immediate disruptive impact rather than manipulating the long-term learning dynamics. Zhang et al. proposed an optimal control formulation of online data poisoning by following a parallel line of research, that of online teaching [18, 19], focussing then on practical attack strategies that perturb the input [6]. Our optimal control perspective is similar to that of Zhang and co-authors, though our contribution differs from theirs in various aspects. First, we consider a supervised learning setting where attacks involve data labels, and not input data. We note that attacks on the labels have a reduced computational cost compared to high-dimensional adversarial perturbations of the inputs, which makes label poisoning more convenient in a streaming scenario. Second, we provide a theoretical analysis and explicit results for the dynamics of the attacked models, and we consider non-linear predictors in our experiments. We observe that our setup is reminiscent of continual learning, which has also recently been analyzed in a teacher-student setup [20]. However, in online data poisoning the labels are dynamically changed by the attacker to drive the learner towards a desired target. In continual learning there is no entity that actively modifies the labels, and the learner simply switches from one task to another. Finally, we note that standard (static, test-time) adversarial attacks have been studied in a teacher-student framework [21].

Reproducibility . The code and details for implementing our experiments are available here.

label:

y

t

input:

x

Teacher

target:

y

*

Attacker

Student

x

μ

,

y

t

μ

x

μ

y

†

μ

y

s

μ

θ

s

μ

+1

θ

s

μ

label:

y

t

input:

x

Teacher

target:

y

*

Attacker

Student

x

μ

,

y

t

μ

x

μ

y

†

μ

y

s

μ

θ

s

μ

+1

θ

s

μ

2 Problem setup

2.1 The teacher, the student, and the attacker

The problem setup is depicted in Fig. 1-A. There are three entities: the teacher, the student, and the attacker. The teacher is a model of the environment, which generates a stream of clean inputoutput data batches. The inputs x ∈ R D have elements sampled from a distribution P x , while clean labels are provided by the scalar teacher function: y t = ϕ t ( x ) . The µ th batch of the stream is B P µ = { ( x µ , y t µ ) p } P p =1 , with P the batch size. The attacker intercepts the clean batch B P µ and perturbs the labels so as to make the student learn the target function ϕ ∗ . The student ϕ s µ represents the learning model, and it updates its weights θ s µ on the stream of perturbed batches via SGD, using each (perturbed) batch only once. We will consider teacher, attacker's target, and student with the same architecture ϕ ( x ; θ ) ; they only differ by the value of their parameters, respectively θ t , θ ∗ , θ s µ . We will refer to this Teacher-Student-Attacker setup as the TSA problem .

If the student trained directly on the input-label pairs provided by the teacher, we would recover the standard teacher-student setup that has been analyzed extensively in the literature [7-10]. The key novelty of this paper is that we consider a setup where each new batch of data is instead poisoned by the attacker with the attempt of deviating the student towards a different target function.

2.2 The attack

The goal of the attacker is for the student to predict a target label y ∗ = ϕ ∗ ( x ) for each given input x . Target label and clean label can coincide for specific inputs, however they won't generally do so. The attacker performs an action a µ that produces perturbed labels as

Note that a µ = 0 implies no perturbation, while for a µ = 1 perturbed and target labels coincide. We allow a min < 0 and a max > 1 to model attackers that under/overshoot. The attacker's action a µ thus constitutes the control variable of the TSA problem. We primarily consider the case where the same value of a µ is used for all labels in each batch, but can be different for different batches. We also explore settings where this assumption is relaxed, including the case of sample-specific perturbations, and the scenario where only a fraction of the labels in each batch is perturbed.

The aim of the attacker is to sway the student towards its target, and so minimize the nefarious cost

using small perturbations. For this reason, it also tries to minimize the perturbation cost

with C a parameter expressing the cost of actions, and E ( ϕ ∗ ) = E x [ ( ϕ ∗ ( x ) -ϕ t ( x )) 2 ] the expected squared error of the attacker's target function; E x indicates the average over the input distribution. We use the pre-factor E ( ϕ ∗ ) in g per µ to set aside the effect of the teacher-target distance on the balance between the two costs. Note that C is inversely proportional to the intuitive notion of attack strength : low C implies a low cost for the action and therefore results in larger perturbations; the x -axis in Fig. 1-B shows the inverse of C . The attacker's goal is to find the sequence of actions a opt µ that minimize the total expected cost :

where E fut represents the average over all possible realizations of the future data stream, and γ ∈ (0 , 1) is the future discounting factor. In relation to γ , we make the crucial assumption that G is dominated by the steady-state running costs; we use γ = 0 . 995 in our experiments.

2.3 SGD dynamics under attack

The perturbed batch B P † µ = { ( x µ , y † µ ) p } P p =1 is used to update the model parameters θ s µ following the standard (stochastic) gradient descent algorithm using the MSE loss function:

where η is the learning rate. We characterize the system's dynamics in terms of the relative teacherstudent distance, defined as

Note that d = 0 (resp. d = 1 ) for a student ϕ s that perfectly predicts the clean (resp. target) labels, on average. We will split the training process into two phases: first, the student trains on clean data, while the attacker calibrates its policy. Then, after convergence of the student, attacks start and training continues on perturbed data. A typical realization of the dynamics is depicted in Fig. 1-C, which shows d µ for a logistic regression classifying MNIST digits during the clean-labels phase, and under attacks of increasing strength (decreasing values of the cost C ). The sharp transition observed in Fig. 1-B occurs for d µ →∞ ≈ 0 . 5 , when the student is equally distant from the teacher and target.

2.4 Attack strategies

Finding the optimal actions sequence that solves the control problem (4) involves performing the average E fut , and so it is only possible if the attacker knows the data generating process. Even then, however, optimal solutions can be found only for simple cases, and realistic settings require approximate solutions. We consider the following approaches:

Constant attacks . The action is observation-independent and remains constant for all SGD steps: a µ = a c ∀ µ . This simple baseline can be optimized by sampling data streams using past observations, simulating the dynamics via Eq. (5), and obtaining numerically the value of a c that minimizes the simulated total expected cost.

Greedy attacks . A greedy attacker maximizes the immediate reward. Since the attacker actions have no instantaneous effect on the learner, the greedy objective has to include the next training step. The greedy action is then given by

The average E x µ +1 can be estimated with input data sampled from past observations, and using (5) to obtain the next student state. Similarly to constant attacks, the greedy future weight ˜ γ can be calibrated by minimizing the simulated total expected cost using past observations.

Reinforcement learning attacks . The attacker can use a parametrized policy function a ( x, θ s ) and optimize it via reinforcement learning (RL). This is a model free approach, and it is suitable for grey box settings. Tuning deep policies is computationally costly, however, and it requires careful calibrations. We utilize TD3 [22], a powerful RL agent that improves the deep deterministic policy gradient algorithm [23]. We employ TD3 as implemented in Stable Baselines3 [24].

Clairvoyant attacks . A clairvoyant attacker has full knowledge of the future data stream, so it simply looks for the sequence { a µ } that minimizes G ( { a µ } , T ≫ 1) . Although the clairvoyant setting is not realistically plausible, it provides an upper bound for the attacker's performance in our experiments. We use Opty [25] to cast the clairvoyant problem into a standard nonlinear program, which is then solved by an interior-point solver (IPOPT) [26].

The above attack strategies are summarized as pseudocode in Appendix A.

3 Theoretical analysis

We first analyze online label attacks with synthetic data to establish our theoretical results. We will confirm our findings with real data experiments in Sec. 4.2. Solving the stochastic optimal control problem (4) is very challenging, even for simple architectures ϕ ( x, θ ) and assuming that P x is known. We discuss an analytical treatment in two scenarios for the linear TSA problem with unbounded action space, where

3.1 Large batch size: deterministic limit

In the limit of very large batches, batch averages of the nefarious cost g nef µ and loss L µ in Eqs. (2, 5) can be approximated with averages over the input distribution. For standardized and i.i.d. input elements, the resulting equations are

In this setting, the attacker faces a deterministic optimal control problem, which can be solved using a Lagrangian-based method. Following this procedure, we obtain explicit solutions for the steady-state student weights w s ∞ and optimal action a opt ∞ . The relative distance d opt ∞ between student and teacher then follows from Eq. (6). We find

The above result is intuitive: the student learns a function that approaches the attacker's target as the cost of actions C tends to zero, and d opt ∞ ( C = 0) = 1 . Vice versa, for increasing values of C , the attacker's optimal action decreases, and the student approaches the teacher function.

In the context of a classification task, we can characterize the system dynamics in terms of the accuracy of the student A µ ( C, P ) = E x [ S µ ( x )] , with S µ ( x ) = ( sign( ϕ s µ ( x )) + sign( ϕ t ( x )) ) / 2 . For label-flipping attacks, where ϕ ∗ = -ϕ t , a direct consequence of Eq. (10) is that the steady-state accuracy of the student exhibits a discontinuous transition in C = 1 . More precisely, we find

with H ( · ) the Heaviside step function. This simple result explains the behaviour observed in Fig. 1-B, where the collapse in accuracy occurs for attack strength C -1 ≈ 1 . We refer to Appendix B for a detailed derivation of results (10, 11).

3.2 Finite batch size: greedy attacks

When dealing with batches of finite size, the attacker's optimal control problem is stochastic and has no straightforward solution, even for the simple linear TSA problem. To make progress, we consider greedy attacks, which are computationally convenient and analytically tractable. While an explicit solution to (7) is generally hard to find, it is possible for the linear case of Eq. (8).

C.

D.

2

mean: 0.3

1

S-*

G=1

0

0

mean:0.27

-1

S-t

1

2

0

1

2

0

250

500

μ

We find the greedy action policy

Note that a G decreases with the cost of actions, and a G → 0 for ˜ C →∞ . With a policy function at hand, we can easily investigate the steady state of the system by averaging over multiple realizations of the data stream. For input elements sampled i.i.d. from P x = N (0 , 1) , we find the average steady-state weights ¯ w s ∞ as

where ¯ d G ∞ is the distance (6) of the average steady-state student; we refer to Appendix C for the derivation of the above result. The expression of ¯ d G ∞ includes a surprising batch-size effect: for a fixed value of the cost C , greedy attacks become more effective for batches of decreasing sizes, as the weights get closer to w ∗ while perturbations become smaller (see Eq. (45)). This effect is displayed in Fig. 2-A, which shows d µ vs SGD step for greedy attacks, and for P = 1 , 10 . Note that only the perturbed training phase is shown, so d 0 = 0 as the student converges to the teacher function during the preceding clean training phase. The black dashed lines correspond to the steady-state values given by (13), which are in very good agreement with the empirical averages (up to a finiteη effect) 2 . We also observe that d µ is characterized by fluctuations with amplitude that decreases as the batch size increases. A consequence of such fluctuations is that the average steady-state accuracy of the student undergoes a smooth transition. The transition becomes sharper as the batch size increases, and in the limit P →∞ it converges to the prediction (11); see Fig. 2-B, which shows numerical evaluations of A ∞ ( C, P ) averaged over multiple data streams. In our experiments, we draw the teacher weights elements as i.i.d. samples from N (0 , 1) and normalize them so that | w t | 2 = D , while w ∗ = -w t .

Remark. In order to derive the result of Eq. (13), we have set ˜ γ = D/η so that ¯ d G ∞ → d opt ∞ for P → ∞ (see Appendix B.1). This guarantees the steady-state optimality of the greedy action policy in the large batch limit. Note that ˜ γ coincides with the timescale of the linear TSA problem, and it lends itself to a nice interpretation: in the greedy setting, where the horizon ends after the second training step, the future is optimally weighted by a factor proportional to the decorrelation time of the weights. Optimality for P finite is not guaranteed, though we observe numerically that ˜ γ approximately matches our choice even for P = 1 , 10 . We use again ˜ γ = D/η to obtain the theoretical results of the next two sections.

3.2.1 Sample-specific perturbations

The batch-size effect observed for greedy attacks can be explained as a signature of the batch average appearing in the policy function (12). When dealing with a single data point at a time, the attacker has precise control over the data labels, designing sample-specific perturbations. For batches with multiple samples, instead, perturbations are designed to perform well on average and do not exploit the fluctuations within the data stream. As a result, attacks become less effective, increasing in magnitude and leading to a smaller average steady-state distance between the student and the teacher function. A straightforward solution to this problem consists of applying sample-specific perturbations, thus using a multi-dimensional control a ∈ R P , which, however, results in a higher computational cost. This can be achieved by using the policy function (12) independently for each sample, so that the p -th element of a is given by

In Appendix C.1, we show that this strategy coincides with the optimal greedy solution for multidimensional, sample-specific control, up to corrections of order η . We also show that the resulting average steady state reached by the student, for normal and i.i.d. input data, is

Remarkably, there is no batch-size dependence in the above solution, and the average steady-state distance coincides with (13) for P = 1 . This result demonstrates that precise, sample-specific greedy attacks remain effective independently of the batch size of the streaming data.

3.2.2 Mixing clean and perturbed data

In the previous section, we addressed a case of enhanced control, where batch perturbations are replaced by sample-specific ones. Here, instead, we consider a case of reduced control, where the attacker can apply the same perturbation to a fraction of the samples in each batch. This scenario can arise when the attacker faces computational limitations or environmental constraints. For example, the streaming frequency may be too high for the attacker to contaminate all samples at each time step, or, in a poisoned federated learning setting, the central server could have access to a source of clean data, beyond the reach of the attackers. Concretely, we assume that the attacker has access to a fraction ρ of the batch samples only. Therefore, training involves ρP perturbed and (1 -ρ ) P clean samples. Perturbations follow the policy function

In Appendix C.2, we find the following average steady-state solution for normal and i.i.d. input data:

see Fig. 7 for a comparison with empirical evaluations. Note that for large batches ¯ d G ∞ ≃ ( C/ρ +1) -1 , indicating that the cost of actions effectively increases by a factor ρ -1 . We remark that the above result is derived under the assumption that the attacker ignores the number of clean samples used during training, so it cannot compensate for the reduced effectiveness of the attacks.

4 Empirical results

4.1 Experiments on synthetic data

Having derived analytical results for the infinite batch and finite batch linear TSA problem, we now present empirical evidence as to how different strategies compare on different architectures. As a first question, we investigate the efficacy of greedy attacks for data streaming in batches of small size. By choosing an appropriate future weight ˜ γ , we know the greedy solution is optimal as the batch size tends to infinity. To investigate optimality in the finite batch setting, we compare greedy actions with clairvoyant actions on the same data stream, optimizing ˜ γ over a grid of possible values. By

C

: constant,

RL

: reinforcement learning,

G

: greedy,

: clairvoyant

A.  Linear regression

C RL* G CV

0.0

0.5

1.0

g

∞

/

g

C

∞

0.0

0.5

1.0

g

∞

/

g

C

∞

0.0

0.5

1.0

g

∞

/

g

C

∞

-

2

0

2

log

10

(

C

)

0.0

0.5

1.0

d

∞

-

2

0

2

log

10

(

C

)

0.0

0.5

1.0

d

∞

-

2

0

2

log

10

(

C

)

0.0

0.5

1.0

d

∞

P=1

P=10

P=1

P=10

P=1

P=10

definition, clairvoyant attacks have access to all the future information and therefore produce the best actions sequence. The scatterplot in Fig. 2-C shows greedy vs clairvoyant actions, respectively a G and a CV , on the same stream. Note that greedy actions are typically larger than their clairvoyant counterparts, except when a CV is very large. This is a distinctive feature of greedy strategies as they look for local optima. The clairvoyant vision allows the attacker to allocate its resources more wisely, for example by investing and paying extra perturbation costs for higher future rewards. This is not possible for the greedy attacker, as its vision is limited to the following training step. Substantially, though, the scattered points in Fig. 2-C lie along the diagonal, indicating that greedy attacks are nearly clairvoyant efficient, even for P = 1 . The match between a G and a CV corresponds to almost identical greedy and clairvoyant trajectories of the student-teacher and student-target weights overlap, defined respectively as O st µ = w s µ T w t /D and O s ∗ µ = w s µ T w ∗ /D (see Fig. 2-D).

Next, we investigate the steady state reached by the student under greedy attacks and the performance of the various attack strategies for three different architectures: the linear regression of Eq. (8), the sigmoidal perceptron ϕ ( x ; w ) = Erf( z/ √ 2) , with Erf( · ) the error function and z = w T x/ √ D , and the 2-layer neural network (NN)

with parameters w m ∈ R D and v ∈ R M . We continue to draw all teacher weights from the standard normal distribution, normalizing them to have unitary self-overlap. For the perceptron we set the target weights as w ∗ = -w t , while for the neural network w ∗ m = w t m and v ∗ = -v t . The top row in Fig. 3 shows ¯ d ∞ = lim µ →∞ ⟨ d µ ⟩ , i.e. the average steady-state distance reached by the student versus the cost of actions C for greedy attacks. The solid lines show the results obtained from simulations 3 . The yellow shaded area shows ¯ d G ∞ form Eq. (13) comprised between values of batch size P = 1 (upper contour) and P = 10 (lower contour). Note that the three cases show very similar behavior of ¯ d ∞ ( C, P ) and are in excellent agreement with the linear TSA prediction.

In order to compare the efficacy of all the attack strategies presented in Sec. 2.4, we consider the running cost , defined as g µ = g per µ + g nef µ . More precisely, we compute the steady-state average ¯ g ∞ over multiple data streams. The bottom row in Fig. 3 shows this quantity for the corresponding architecture of the top row. The evaluation of the clairvoyant attack is missing for the NN architecture, as this approach becomes unfeasible for complex models due to the high non-convexity of the associated minimization problem. Similarly, reinforcement learning attacks are impractical for large architectures. This is because the observation space of the RL agent, given by the input x and the parameters of the model under attack, becomes too large to handle for neural networks. Therefore, for the NN model we used a TD3 agent that only observes the read-out weights v of the student network; hence the asterisk in RL ∗ . We observe that the constant and clairvoyant strategies have respectively the highest and lowest running costs, as expected. Greedy attacks represent the second-best strategy, with an average running cost just above that of clairvoyant attacks. While we do not explicitly show it, it should also be emphasized that the computational costs of the different strategies vary widely; in particular, while greedy attacks are relatively cheap, RL attacks require longer data streams and larger computational resources, since they ignore the learning dynamics. We refer to Fig. 8, 9 in Appendix D for a comparison between greedy and RL action policies, and for a visual representation of the convergence in performance of the RL algorithm vs number of training episodes.

4.2 Experiments on real data

In this section, we explore the behaviour of online data poisoning in an uncontrolled scenario, where input data elements are not i.i.d. and are correlated with the teacher weights. Specifically, we consider two binary image classification tasks, using classes '1' and '7' from MNIST, and 'cats' and 'dogs' from CIFAR10, while attacks are performed using the greedy algorithm. We explore two different setups. First, we experiment on the MNIST task with simple architectures where all parameters are updated during the perturbed training phase. For this purpose, we consider a logistic regression (LogReg) and a simple convolutional neural network, the LeNet of [27]. Then, we address the case of transfer learning for more complex image-recognition architectures, where only the last layer is trained during the attacks, which constitutes a typical application of large pre-trained models. Specifically, we consider the visual geometry group (VGG11) [28] and residual networks (ResNet18) [29], with weights trained on the ImageNet-1k classification task. For all architectures, we fix the last linear layer to have 10 input and 1 output features, followed by the Erf activation function.

Preprocessing . For LogReg, we reduce the dimensionality of input data to 10 via PCA projection. For LeNet, images are resized from 28x28 to 32x32 pixels, while for VGG11 and ResNet18 images are resized from 32x32 to 224x224 pixels. Features are normalized to have zero mean and unit variance, and small random rotations of maximum 10 degrees are applied to each image, thus providing new samples at every step of training.

Learning . The training process follows the TSA paradigm: we train a teacher network on the given classification task, and we then use it to provide (soft) labels to a student network with the same architecture. As before, we consider two learning phases: first, the attacker is silent, and it calibrates ˜ γ . Then attacks start and the student receives perturbed data. For consistency with our teacher-student analysis, we consider the SGD optimizer with MSE loss; see Fig. 10 in Appendix D for a comparison between SGD and Adam-based learning dynamics. To keep the VGG11 and ResNet18 models in a regime where online learning is possible, we use the optimizer with positive weight decay.

The findings are presented in Fig. 4, where panels A and B represent the outcomes of the full training and transfer learning experiments, respectively. Each panel consists of two rows: the top row displays the dependence of the relative student-teacher distance on the cost of action C and batch size P . The orange area depicts ¯ d G ∞ from Eq. (13) for P ∈ [1 , 10] , which we included for a visual comparison. The bottom row shows the corresponding average steady-state accuracy of the student. In general, all learners exhibit comparable behavior to the synthetic teacher-student setup in terms of the relationship between the average steady-state distance ¯ d ∞ and C . Additionally, all experiments demonstrate a catastrophic transition in classification accuracy when the value of C surpasses a critical threshold. The impact of batch size P on the ¯ d ∞ is also evident across all experiments. The effect of the weight decay used in the transfer learning setting is also visible: the value of ¯ d ∞ remains larger than zero even when C is very large (and attacks are effectively absent). Similarly, for very low values of C , the relative distance remains below 1. We note that calibrating ˜ γ takes a long time for complex models updating all parameters, so we provide single-run evaluations for LeNet.

-

2

0

2

log

10

(

C

)

0.0

0.5

1.0

d

∞

-

2

0

2

log

10

(

C

)

0.0

0.5

1.0

accuracy

-

2

0

2

log

10

(

C

)

0.0

0.5

1.0

d

∞

-

2

0

2

log

10

(

C

)

0.0

0.5

1.0

d

∞

-

2

0

2

log

10

(

C

)

0.0

0.5

1.0

d

∞

-

2

0

2

log

10

(

C

)

0.0

0.5

1.0

accuracy

LeNet

P=1

P=10

LogReg

A. Full training - MNIST

-

2

0

2

log

10

(

C

)

0.0

0.5

1.0

accuracy

VGG11

-

2

0

2

log

10

(

C

)

0.0

0.5

1.0

accuracy

ResNet18

P=1

P=10

5 Conclusions

Understanding the robustness of learning algorithms is key to their safe deployment, and the online learning case, which we analyzed here, is of fundamental relevance for any interactive application. In this paper, we explored the case of online attacks that target data labels. Using the teacher-student framework, we derived analytical insights that shed light on the behavior of learning models exposed to such attacks. In particular, we proved that a discontinuous transition in the model's accuracy occurs when the strength of attacks exceeds a critical value, as observed consistently in all of our experiments. Our analysis also revealed that greedy attack strategies can be highly efficient in this context, especially when applying sample-specific perturbations, which are straightforward to implement when data stream in small batches.

A key assumption of our investigation is that the attacker tries to manipulate the long-run behavior of the learner. We note that, for complex architectures, this assumption may represent a computational challenge, as the attacker's action induces a slowdown in the dynamics. Moreover, depending on the context, attacks may be subject to time constraints and therefore involve transient dynamics. While we have not covered such a scenario in our analysis, the optimal control approach that we proposed could be recast in a finite-time horizon setting. We also note that our treatment of online data poisoning did not involve defense mechanisms implemented by the learning model. Designing optimal control attacks that can evade detection of defense strategies represents an important avenue for future research.

Further questions are suggested by our results. While the qualitative features of our theoretical predictions are replicated across all real-data experiments that we performed, we observed differences in robustness between different algorithms. For standard, test-time adversarial attacks, model complexity can aggravate vulnerability [30, 31], and whether this happens in the context of online attacks represents an important open question. Another topic of interest is the interaction of the poisoning dynamics with the structure of the data, and with the complexity of the task. Finally, we remark that our analysis involved sequences of i.i.d. data samples: taking into account temporal correlations and covariate shifts within the data stream constitutes yet another intriguing challenge.

Compute . We used a single NVIDIA Quadro RTX 4000 graphics card for all our experiments.

References

Huang, L., Joseph, A. D., Nelson, B., Rubinstein, B. I. & Tygar, J. D. Adversarial Machine Learning. Proceedings of the 4th ACM Workshop on Security and Artificial Intelligence, 43-58 (2011).

Biggio, B. & Roli, F. Wild patterns: Ten years after the rise of adversarial machine learning. Pattern Recognition, 317-331 (2018).

Chen, X., Liu, C., Li, B., Lu, K. & Song, D. Targeted Backdoor Attacks on Deep Learning Systems Using Data Poisoning. Preprint, ArXiv:1712.05526 (2017).

Gu, T., Liu, K., Dolan-Gavitt, B. & Garg, S. BadNets: Evaluating Backdooring Attacks on Deep Neural Networks. IEEE Access, 47230-47244 (2019).

Goldblum, M. et al. Dataset Security for Machine Learning: Data Poisoning, Backdoor Attacks, and Defenses. IEEE Transactions on Pattern Analysis and Machine Intelligence, 1563-1580 (2023).

Zhang, X., Zhu, X. & Lessard, L. Online Data Poisoning Attacks. Proceedings of the 2nd Conference on Learning for Dynamics and Control, 201-210 (2020).

Gardner, E. & Derrida, B. Three unfinished works on the optimal storage capacity of networks. Journal of Physics A: Mathematical and General, 1983 (1989).

Seung, H. S., Sompolinsky, H. & Tishby, N. Statistical mechanics of learning from examples. Physical review A (1992).

Engel, A. & Van den Broeck, C. Statistical mechanics of learning (2001).

Carleo, G. et al. Machine learning and the physical sciences. Reviews of Modern Physics (2019).

Biggio, B., Nelson, B. & Laskov, P. Poisoning Attacks against Support Vector Machines. Proceedings of the 29th International Coference on Machine Learning, 1467-1474 (2012).

Xiao, H. et al. Support vector machines under adversarial label contamination. Neurocomputing, 53-62 (2015).

Muñoz-González, L. et al. Towards Poisoning of Deep Learning Algorithms with Back-gradient Optimization. 10th ACM Workshop on Artificial Intelligence and Security, 27-38 (2017).

Ma, Y., Zhang, X., Sun, W. & Zhu, X. Policy Poisoning in Batch Reinforcement Learning and Control. Proceedings of the 33rd International Conference on Neural Information Processing Systems (2019).

Mei, S. & Zhu, X. Using Machine Teaching to Identify Optimal Training-Set Attacks on Machine Learners. Proceedings of the Twenty-Ninth AAAI Conference on Artificial Intelligence, 2871-2877 (2015).

Wang, Y. & Chaudhuri, K. Data Poisoning Attacks against Online Learning. Preprint, ArXiv:1808.08994 (2018).

Pang, T., Yang, X., Dong, Y., Su, H. & Zhu, J. Accumulative Poisoning Attacks on Real-time Data. Advances in Neural Information Processing Systems, 2899-2912 (2021).

Liu, W. et al. Iterative Machine Teaching. Proceedings of the 34th International Conference on Machine Learning, 2149-2158 (2017).

Lessard, L., Zhang, X. & Zhu, X. An Optimal Control Approach to Sequential Machine Teaching. Proceedings of the Twenty-Second International Conference on Artificial Intelligence and Statistics, 2495-2503 (2019).

Lee, S., Goldt, S. & Saxe, A. Continual Learning in the Teacher-Student Setup: Impact of Task Similarity. Proceedings of the 38th International Conference on Machine Learning, 6109-6119 (2021).

Yang, Z. et al. Understanding Robustness in Teacher-Student Setting: A New Perspective. Preprint, ArXiv:2102.13170 (2021).

Fujimoto, S., van Hoof, H. & Meger, D. Addressing Function Approximation Error in ActorCritic Methods. Preprint, ArXiv:1802.09477 (2018).

Lillicrap, T. P. et al. Continuous control with deep reinforcement learning. 4th International Conference on Learning Representations, ICLR (2016).

Raffin, A. et al. Stable-Baselines3: Reliable Reinforcement Learning Implementations. Journal of Machine Learning Research, 1-8 (2021).

Moore, J. K. & van Bogert, A. D. Opty: Software for trajectory optimization and parameter identification using direct collocation. Journal of Open Source Software (2018).

Wächter, A. & Biegler, L. T. On the Implementation of an Interior-Point Filter Line-Search Algorithm for Large-Scale Nonlinear Programming. Mathematical Programming, 25-57 (2006).

LeCun, Y., Bottou, L., Bengio, Y. & Haffner, P. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 2278-2324 (1998).

Simonyan, K. & Zisserman, A. Very Deep Convolutional Networks for Large-Scale Image Recognition. 3rd International Conference on Learning Representations, ICLR (eds Bengio, Y. &LeCun, Y.) (2015).

He, K., Zhang, X., Ren, S. & Sun, J. Deep Residual Learning for Image Recognition. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 770-778 (2016).

Tsipras, D., Santurkar, S., Engstrom, L., Turner, A. & Madry, A. Robustness may be at odds with accuracy. Preprint, ArXiv:1805.12152 (2018).

Carbone, G. et al. Robustness of bayesian neural networks to gradient-based attacks. Advances in Neural Information Processing Systems, 15602-15613 (2020).

Bressan, A. & Piccoli, B. Introduction to the mathematical theory of control (2007).

A Attack strategies: algorithms

Table 1 summarizes the attack strategies introduced in Sec. 2.4. For clarity, we present the algorithms for data streaming in batches of size P = 1 . We recall that the action value a c of constant attacks and the greedy future weight ˜ γ can be optimized by sampling future data streams from past observations, and using the SGD update rule (5) to simulate the future dynamics. Similarly, the attacker can sample past observations to calibrate the the RL policy a RL ( x, θ ) .

Algorithm 1 Online label attacks (batch size P = 1 )

Constant attacks

1: optimize a c

2: repeat

3: a µ ← a c

5: y s µ ← ϕ s ( x µ ; θ s µ )

4: y † µ ← y t µ +( y ∗ µ -y t µ ) a µ

6: θ s µ +1 ← θ s µ -η ∇ θ s µ L ( y s µ , y † µ )

7: until end of stream

Greedy attacks

1: optimize ˜ γ

2: repeat

3: observe clean batch ( x, y t ) µ , state θ s µ

4: a µ ← argmin ⟨ g per µ + ˜ γg nef µ +1 ⟩

6: y s µ ← ϕ s ( x µ ; θ s µ )

5: y † µ ← y t µ +( y ∗ µ -y t µ ) a µ

Reinforcement learning attacks

1: calibrate policy a RL ( x, θ )

2: repeat

3: observe clean batch ( x, y t ) µ , state θ s µ

4: a µ ← a RL ( x µ , θ s µ )

6: y s µ ← ϕ s ( x µ ; θ s µ )

8: until end of stream

Clairvoyant attacks

1: observe state θ s 1 , stream S = { ( x, y t ) µ } T µ =1

3: repeat

2: find { a µ } T µ =1 = argmin G ( S, θ s 1 )

4: y † µ ← y t µ +( y ∗ µ -y t µ ) a µ

7: θ s µ +1 ← θ s µ -η ∇ θ s µ L ( y s µ , y † µ )

5: y s µ ← ϕ s ( x µ ; θ s µ )

8: until end of stream

6: θ s µ +1 ← θ s µ -η ∇ θ s µ L ( y s µ , y † µ )

7: until end of stream

B Linear TSA problem: optimal solution

In the limit of very large batches, batch averages of the nefarious cost g nef µ and loss L µ in Eqs. (2, 5) can be approximated with averages over the input distribution. For a general architecture ϕ , and indicating with t, s, and ∗ the teacher, student, and attacker's target, respectively, we can write

where I ( a, b ) = E x [ ϕ a ( x ) ϕ b ( x ) ] for two models, a and b , with the same architecture but different parameters θ a , θ b , and I ′ = ∇ θ s I . In this case, the optimal control problem (4) is deterministic, and we can write it in continuous time ( η → 0 ) as

where g per µ = ˜ Ca 2 µ / 2 . Note that µ now is a continuous time index, and the above minimization problem considers all functions a µ ∈ [ a min , a max ] for µ ∈ [0 , ∞ ) . In the following, we will assume that the action space is unbounded, i.e. a µ ∈ ( -∞ , ∞ ) , and that the elements of x are i.i.d. samples with zero mean and variance σ 2 . In the linear TSA problem, where ϕ ( x ; w ) = w T x/ √ D , the above Eqs. (19, 20) simplify to

where ∆ w ab = w a -w b , and | · | 2 the squared 2-norm. We also find

We shall recall that in the TSA problem each batch is used only once during training. This assumption guarantees that inputs x µ and weights w s µ are uncorrelated, a necessary condition to derive the above expressions. We solve this problem with a Lagrangian-based approach: the Lagrangian L is obtained by imposing the constraint ˙ w s µ = -∇ w s L µ , at all times, with the co-state variables λ µ ∈ R D :

The Pontryagin's necessary conditions for optimality ∂ a µ L = 0 , [ ∂ w s µ -∂ µ ∂ ˙ w s µ ] L = 0 , and ∂ λ µ L = 0 give the following set of equations coupling a µ , w s µ , and λ µ :

The above system of ODEs can be solved by specifying the initial condition w s 0 and using the transversality condition lim µ →∞ λ µ = 0 . At steady state, we obtain the following equation for w s ∞ :

which admits only one (optimal) solution. We find it as a linear combination of w t and w ∗ :

From the definition of d µ (6), it follows that

As a final remark, we observe that the same procedure can in principle be applied to TSA problems with non-linear architectures, as long as an explicit expression for I ( a, b ) is available. However, one may find multiple steady-state solutions, and obtaining guarantees for optimality is often hard [32].

B.1 Optimal greedy solution

For the linear TSA problem, and in the limit of large batches, the discrete-time equation (7) for greedy actions simplifies to

The expression of w s µ +1 ( a µ ) is given by the update rule, which can be conveniently written as

where v 1 = w s µ -ησ 2 ∆ w st µ /D , and v 2 = -ησ 2 ∆ w t ∗ /D . The solution of (29) gives the greedy policy function

The right-hand side of the above equation depends on the future weighting factor ˜ γ . Requiring that the (greedy) update rule w s µ +1 = v 1 + v 2 a G µ gives the optimal steady-state solution of Eq. (27), and using the explicit expression of ˜ C (23), one finds the optimal weight

B.2 Accuracy vs cost of action

In Sec. 3.1, we addressed the case of attacks in a classification context, with class labels given by the sign of ϕ t ( x ) , and where the attacker aims to swap the labels, so ϕ ∗ = -ϕ t . A pointwise estimate of the accuracy is given by

and the system dynamics can be characterized by the accuracy of the student, defined as A µ ( C, P ) = E x [ S µ ( x )] , with E x the average over the input distribution. For the linear TSA problem, we have w ∗ = -w t , and, consistently with the definition of d µ (6),

It follows that S µ ( x ) = 1 for d µ ≤ 0 . 5 , and S µ ( x ) = 0 otherwise. In the deterministic limit of large batches, the optimal steady-state solution (27) gives

We then have S ∞ = 0 for C < 1 , and S ∞ = 0 otherwise, regardless of the value of x , giving

with H ( · ) the Heaviside step function.

C Stochastic control with greedy attacks

When the batch size P is finite, the attacker's optimal control problem is stochastic and has no straightforward solution, even for the simplest case of the linear TSA problem. It is however possible to find the average steady-state solution for greedy attacks, as we show in this Appendix. We derived the greedy policy in B.1 for P →∞ , and for i.i.d. input elements with zero mean and variance σ 2 . For a general P , the same expression (31) holds, with v 1 and v 2 now given by

where ∆ w ab = w a -w b . The first-order expansion of a G µ in powers of η reads

where we have used ˜ γ = D/σ 2 η , which guarantees that in the limit P →∞ we recover the optimal solution (27). We can use the above action policy in the SGD update rule (30) and take the average over data streams to get the steady-state equation

where we have imposed ¯ w s µ = ¯ w s µ +1 = ¯ w s ∞ for µ →∞ , and where

̸

Note that for P = 1 and P →∞ the right-hand side of (39) only involves T 1 l and T ∞ l , respectively (hence the labels). So far, we have assumed that the input elements are i.i.d. samples from P x with first two moments m 1 = 0 , and m 2 = σ 2 . We also recall that x µ and w s µ are uncorrelated, as each

-

2

0

2

log

10

(

C

)

0.0

0.5

1.0

a

G

∞

P=10

P=1

Figure 5: Average steady-state greedy action . ¯ a G ∞ vs cost of actions C for the linear TSA problem with x ∼ N (0 , 1) . The orange area represents the range of solutions (45) for P ∈ [1 , 10] . The red and blue lines show empirical evaluations for P = 1 and P = 10 , respectively. Parameters: D = 10 , η = 10 -4 × D , a ∈ [ -2 , 3] . Averages performed over 10 data streams of 2 4 batches, and over the last 10 3 steps in each stream.

batch is used only once in the TSA problem. We now further assume that input elements have third and fourth moments m 3 = 0 , and m 4 < ∞ . It is then immediate to verify that T 1 l = T 1 l ( m 2 , m 4 ) and T ∞ l = T ∞ l ( m 2 ) , showing that finite batch-size effects depend on the kurtosis of the input distribution. Specifically, we find

For P x = N (0 , 1) , the first term in T 1 l disappears, and the above expressions simplify to

Substituting in Eq. (39) the expressions of T 1 l , T ∞ l , and ˜ C from (23), and looking for a solution of ¯ w s ∞ as a linear combination of w t and w ∗ , we find

Note that ¯ d G ∞ is the relative distance (6) of the student with parameters ¯ w s ∞ . From this result, we can easily find the mean steady-state greedy action by averaging (38) over data streams, giving

Fig. 5 shows a comparison of the above result with empirical evaluations, which are in excellent agreement. Together, (44) and (45) demonstrate that greedy attacks become more efficient as the batch size decreases, reducing the distance between the student and the target while using smaller perturbations. We remark that this result is derived using the expression of ˜ γ from the P → ∞ analysis, as in Eq. (32). In our experiments, this value is near-optimal also for P finite for linear learners, while for non-linear architectures it provides a good first guess during the calibration phase.

C.1 Sample-specific perturbations

In this section, we address the case where the attacker has finer control over the data labels and can apply sample-specific perturbations, rather than batch-specific ones. The control variable a is P -dimensional, with P the size of the streaming batches, and each label y t p is perturbed as

while the perturbation cost is g per µ = ˜ C | a µ | 2 / 2 P . We can write the SGD update rule as

where now

0

2000

4000

µ

0.0

0.5

1.0

d

µ

P=1

P=10 *

P=10

We use again the notation ∆ w ab = w a -w b . Note that (47) reduces to (30) when the control variable is one-dimensional and batch-specific, i.e. a µp = a µ ∀ p . The minimization problem (7) for greedy actions now consists of P coupled equations, one for each control variable a µp . For i.i.d. input elements with zero mean and variance σ 2 , we find

and the first-order conditions lead to

A straightforward and explicit solution can be found by considering only the first-order terms in η , effectively decoupling the above system of equations. With ˜ γ = D/σ 2 , we obtain

which coincides with the P = 1 solution for batch-specific greedy attacks (38). We can now use this policy in the update rule (47) to investigate the steady state of the system. Once more, we average across data streams with i.i.d. data sampled from P x = N (0 , 1) and impose that the average student weights ¯ w s µ reach a steady state for µ →∞ . This leads to the following P -independent solution

Fig. 6 shows empirical evaluations confirming this result. Finally, averaging the policy function (51) across data streams and using the above result for ¯ w s ∞ we find

C.2 Mixing clean and perturbed data

The assumption that the attacker can perturb every data sample in each batch may not always be satisfied. For example, in a federated learning scenario, the central server coordinating the training may have access to a trusted source of clean data that is inaccessible to the attacker. Moreover, the attacker may face a limited computational budget and only be able to process a fraction of samples in the data stream at each timestep. Here, we consider the case where only a fraction ρ of the P samples in each batch is perturbed by the attacker. Moreover, we assume that the attacker ignores the amount of clean samples used for training. Precisely, the attacker considers the update rule

with ρP perturbed samples, while training follows

0.2

0.4

0.6

0.8

1.0

ρ

0.0

0.2

0.4

0.6

d

∞

Theory

LinReg syn.

LogReg MNIST

thus involving ρP perturbed samples and (1 -ρ ) P clean samples. We recall that ∆ w ab = w a -w b , and that the stream has input data drawn i.i.d. from P x . Following (54), the greedy policy (38) in this case reads

We can now substitute the above expression into (55) to investigate the steady state of the system. As before, we can obtain a steady-state condition for ¯ w s ∞ by averaging across data streams and imposing ¯ w s µ = ¯ w s µ +1 = ¯ w s ∞ for µ →∞ . We find

with T 1 l and T ∞ l as in (43) for P x = N (0 , 1) . It is then immediately verified that

Note that for large batches ( ρP ≫ 2 ) the above result reduces to ¯ d G ∞ ≈ ( C/ρ +1) -1 , and it has a simple and intuitive interpretation: when perturbations contaminate a fraction ρ of the batch samples only, the cost of actions effectively increases by a factor ρ -1 . Fig. 7 shows the average steady-state distance reached by the student model as a function of ρ , demonstrating that empirical evaluations using synthetic data perfectly match the theoretical estimate of Eq. (58). We also performed real data experiments using MNIST images (as in Sec. 4.2), which again exhibit a good qualitative agreement with our theoretical result.

D Supplementary figures

A.  Greedy policy

-

3

0

ρ

0.5

1.0

a

0.0

0.5

1.0

a

0

0

0

0.5

1

0.5

1

3

-3

x

ρ

-

3

0

ρ

0.5

0

0

0.5

1

3

-3

x

ρ

a

0

0.5

1

3

x

0.0

x

3

0.0

1.0

0.5

0.0

1.0

a

0

2

4

6

8

n.

episodes

0.5

1.0

1.5

2.0

¯

g

RL

∞

greedy ¯

g

∞

300

340

380

µ

-

0.5

0.5

1.5

a

µ

greedy

RL (best)

best model

0

2

4

6

8

n.

episodes

0.3

0.5

0.7

¯

g

RL

∞

greedy ¯

g

∞

300

340

380

µ

-

0.5

0.5

1.5

a

µ

greedy

RL (best)

best model

300

340

380

µ

-

0.5

0.5

1.5

a

µ

greedy

RL (best)

C. 2-layer NN

0

2

4

6

8

n.

episodes

1.0

1.5

¯

g

RL

∗

∞

greedy ¯

g

∞

best model

0

2000

4000

µ

0.0

0.5

1.0

d

µ

P=10

P=1

A.

-

2

0

2

log

10

(

C

)

0.0

0.5

1.0

d

∞

B.

P=10

P=1

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper sheds light on the vulnerabilities of online learners to adversarial attacks and provides insights into the manipulation of learning dynamics. The findings highlight the need for robust defenses against such attacks and contribute to the growing body of research on data poisoning attacks in machine learning.\n\nSTRENGTHS:\n1. The paper provides a theoretical analysis of online data poisoning in the teacher-student setup, which is a popular framework for studying machine learning models in a controlled way. \n2. The paper compares different attack strategies and finds that properly calibrated greedy attacks can be as effective as attacks with full knowledge of the incoming data stream. \n3. The authors empirically study online data poisoning on real datasets, such as MNIST and CIFAR10, using various architectures like LeNet, ResNet, and VGG.\n\nWEAKNESSES:\n1. This paper primarily focuses on linear regression models and does not extensively explore the impact of adversarial attacks on other types of machine learning models.\n2. This paper does not provide specific examples of real-world scenarios where adversarial attacks on online learners can have significant consequences. \n3. This paper lacks extensively discuss potential defense mechanisms or countermeasures to mitigate the impact of these attacks. \n4. This paper does not thoroughly compare its findings with existing literature on attacks on online learners. \n5. The practicality studied threat model should be discussed with more details.\n\nQUESTIONS:\n1. How do different attack strategies compare in terms of efficacy?\n2. Are attacks subject to time constraints and do they involve transient dynamics? \n3. Does the analysis take into account temporal correlations within the data stream? \n4. The model's robustness improves or decreases of over-parametrization?\n\nLIMITATIONS:\nMissing limitation and negative societal impact discussion in this paper.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nData poisoning attacks have been extensively studied in the offline setting, while poisoning attacks in the online setting have received little attention. In the online setting, the attacker is forced to craft attacks exploiting the data stream, taking into account the state of the model and the possible future data stream to divert the model. This work aims to study theoretically and empirically the learning dynamics of an online learner subject to a poisoning attack in the white-box setting, i.e., the attacker knows the architecture and the current parameters of the target model. The problem of finding the best attack policy is formalized as a stochastic optimal control problem and is theoretically studied in the teacher-student setup. From that, the authors were able to obtain results about the steady state of an attacked linear regression model, i.e., the model weights, the optimal action and the distance of the target model depending on the cost of the attack, supposing both infinite and finite size batches. Furthermore, the work shows analytically that a greedy strategy can perform quite optimally. The experimental evaluation confirms the theoretical results on synthetic data and explores the dynamics of online data poisoning attacks on real data and non-linear predictors, showing that the theoretical findings almost hold in more complex scenarios.\n\nSTRENGTHS:\nThis paper makes a valuable contribution to the literature on data poisoning attacks, where there has been limited work in online settings. In particular:\n\n- The idea of analyzing the attacker’s control problem from the perspective of the teacher-student framework represents the strong main novelty of the paper, since the formulation of the attack problem as an optimal control problem has been discussed before. The analysis allows the authors to derive interesting results about the dynamics of the learning process of the model and the steady state of a linear regressor that have not previously been discussed in the literature. It is interesting to see that above a certain attack strength the accuracy of the attacked model may drop dramatically and perturbations above the threshold may not be sufficient to realize the attack.\n\n- The set of considered attack strategies sufficiently covers the possible options available to the attacker. The idea of showing that a greedy attack can be as effective as an attack with full knowledge is valuable. \n\n- The theoretical results are confirmed empirically and punctually. Moreover, the experimental results on non-linear architectures and real data suggest that the theoretical results may also hold for more complex scenarios, that’s an interesting finding.\n\n- The derivation of the theoretical results is explained in the appendix and the well-organized code to reproduce the experimental results is available (even though more instructions to reproduce the results may be added, like the Python version used by the authors to run the experiments).\n\nWEAKNESSES:\n- The theoretical analysis is limited to consider linear regression as the student model, the standard normal distribution as input distribution and an infinite time horizon, so the theoretical results are a bit limited. However, it is recognized by the authors that more general results are missing because deriving them in a more general scenario is hard. Furthermore, the authors tried to derive some intuitions about the results in a more complex scenario through the experimental evaluation. Covering the theoretical analysis in a more complex setting and with a finite time horizon is an interesting future work to explore.\n\n- It is not clearly motivated why the attacker can only modify the labels instead of the entire inputs and why the attacker should be able to apply the same perturbation to the entire batch (which turns out to be a very strong assumption). Data poisoning attackers should assume little control over the data to be considered practical [P1, P2].\n\n- The presentation of the theoretical results is a bit confusing since it is mixed with the presentation of part of the theoretical results.\n \n- The presentation of the attack strategies and their implementation lack details that cannot be found in the appendix. See the “Questions” section for more details.\n\n- From the paper, it is not evident what the take-home message is and what suggestions are provided to improve robustness in stream pipelines or make attackers more stealthy. The paper presents numerous useful analyses and extensive experimental settings but lacks a \"Take-home message/Discussion\" section that summarizes the overall findings and suggestions. While the authors have reported some of these in the conclusion section, I believe they can be further expanded with higher priority/attention.\n\n[P1] Back to the Drawing Board: A Critical Evaluation of Poisoning Attacks on Federated Learning. In IEEE Symposium on Security and Privacy (2022). \n\n[P2] Wild Patterns Reloaded: A Survey of Machine Learning Security against Training Data Poisoning. ACM Computing Surveys (2022).\n\nQUESTIONS:\nMajor questions:\n\n- Why are the clairvoyant and RL attacks infeasible to deploy for complex models? Is it due to the time required for the calibration? This is not very clear from the text.\n\n- In Eq. 1, please clarify why it should be important to model an attacker that “under/overshoots”. Why can the attacker obtain a perturbed label that resides outside the interval between the clean and target labels?\n\n- I think that more details about the RL attack are needed. How is the problem of finding the optimal sequence of actions mapped to the RL problem? Which are the reasons for choosing to utilize TD3 (and a deep learning policy in general) instead of other RL approaches?\n\n- Which is the utility of gamma^tilde in the objective function minimized by the greedy attack? Is it used to address the fact that the attacker should not try the most effective attack at the subsequent time step, but she has to take into account all the future streams of instances? How does the calibration of gamma^tilde work? I invite the authors to add more details about this.\n\n- I suggest that the authors add a \"Take-home-messages/Discussion” section that summarizes the theoretical and empirical results and highlights how the results can be useful for designing a poisoning attack or defending against it. \n\n- The authors should clarify their threat model in the paper and where it may apply.\n\n\nMinor questions:\n\n - Are the results of Figures 1-B and 1-C obtained from the experimental evaluation? If yes, the setting used for obtaining these results should be indicated in the text. I may have missed it, so I kindly ask the authors to point me out the lines containing this information if it is included in the paper.\n\n- Should g^nef be at the place of one of the g^pre in eq. 16?\n\nEventually, I kindly ask the authors to fix the links to the equations that appear in the text, since they don’t work at the moment.\n\nLIMITATIONS:\nThe authors have briefly discussed some limitations of their work in the conclusion section and they have clearly stated that extending their analysis to non-linear architectures and different probability distributions of the inputs is challenging. However, I think that a discussion about the limitations of the threat model considered is missing and I invite the authors to cover this point, by discussing, for example, why the attacker should only modify the labels of the instances. Moreover, I ask the authors to include a “take-on-messages/discussion” section to summarize and clarify the obtained results and their implications.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper analyzes the robustness of online learners when the labels of the received data are manipulated. In particular, it analyzes a student-teacher online learning problem where the attacker poisons the labels provided by the teacher before feeding the student the labeled batch. The setup is analyzed both theoretically and experimentally where with the increase of the attack budget, the performance of the learner degrades significantly.\n\nSTRENGTHS:\nThe main strengths of this work are:\n\n(1) The paper is well-written and the figures are polished.\n\n(2) The setup is clearly explained and the theoretical analysis on synthetic data support the results.\n\nWEAKNESSES:\nThere are few weaknesses that I hope to be addressed before getting this paper accepted:\n\n(1) The paper is missing a strong practical motivational example from real world scenarios when attackers have access to manipulate the *labels* before feeding the labeled batches to the learner. \n\n(2) While the proposed problem setup studies an online learning scheme, the attacker is allowed for an unbounded computational budget to manipulate the labels. However, in online learning, the environment can reveal new batches *irrespective* of how efficient the attacker is in manipulating the previously revealed batch. Note that this assumption (given unlimited computation to the attacker) is not realistic and it questions that effectiveness of the proposed attack.\n\n(3) Experimental evaluation are conducted on very small scale problems and datasets. I would expect evaluating the results on online learning datasets such as CLOC [A] and Firehose [B].\n\n(4) The analyzed attackers should be evaluated when a defense mechanism is presented. For instance, would a learner upon convergence still update all model parameters on each received data? What happens if the learner stored labeled examples from the stream before the poisoning phase starts and update on mixed batches?\n\n[A] Online Continual Learning with Natural Distribution Shifts: An Empirical Study with Visual Data, ICCV 2021.\n\n[B] Drinking from a Firehose: Continual Learning with Web-scale Natural Language, 2020.\n\nQUESTIONS:\nIn addition to the points raised in the weaknesses part, I have the following question:\\\n\n- To guarantee that the attack is imperceptible (in case of perturbing the input), the attacker has to produce the perturbations under budget constraints (e.g. $\\|\\delta\\|_\\infty \\leq \\frac{8}{255}$). How is this constraint translated to the proposed setup?\n\n- The perturbed label in Equation (1) could produce a real number for integer valued labels. Couldn't this be a simple check for the learner to reject such samples from training on them?\n\nLIMITATIONS:\nThe authors have not addressed the limitations of this work such as the practicality of the proposed setup, the limited experimental setup (problem and dataset scale) along with how expensive is it to conduct such attacks.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-10"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors conduct theoretical analysis and empirical evaluation of poisoning attacks in the online learning setting, where the attacker can intervene in the labels of sequentially provided data. As a result of the theoretical analysis, the authors show that the strength of the attack becomes discontinuously stronger as the batch size asymptotically approaches infinity. It is also shown that even a greedy attack can be as strong as that in the clairvoyance setting under an appropriate condition.\n\nSTRENGTHS:\nIt introduces a new problem setup for poisoning, and although the analysis is performed in a very simple setting, it leads to interesting results.\nIt is interesting that a properly designed greedy attack is as effective as an attack in the clairvoyant setting.\nTheoretical analysis utilizing optimal control theory is not very common, and it is useful to introduce such methods.\n\nWEAKNESSES:\nThere is little discussion of threat models and attacker models when viewed as a security issue. There is no disagreement that poisoning in an online setting is a serious threat, but the paper does not discuss where the significance of considering this type of dirty label attack lies and whether it could be a significant risk. In particular, I wonder if there is any point in dealing with a setting where the batch size is infinite in an online setting.\n\nQUESTIONS:\nIn equation 4, is a parameter to adjust the balance between g^per and g^nef necessary?\nIn equation 6, when d_mu(C,P)=1, it cannot be said that the student's prediction and the target's prediction coincide since they merely coincide in their expectations.\n\nLIMITATIONS:\nThe subject is an attack on a learning system and it can affect society in a malicious way, but it is obvious that the research interest is in the theoretical aspect of the behavior of the system under attack.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-19"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper provide an analysis of steady state of linear learners trained via SGD under online setting, where the found a phase transition in terms of attack strength. Some experiments are also conducted to demonstrate the insight from the theory.\n\nSTRENGTHS:\nThe formulation of attacking learning model under online setting is clean and intuitive. The theory is also illuminating. The insight that model performance will sharply decrease if attach strength is higher than a threshold is an interesting observation. In addition, the theory is supported by experiments.\n\nWEAKNESSES:\nOverall my feeling is both empirically and theoretically, the paper is not strong. Theory side, the main concern I have on the work is that the problem might be oversimplified, thus it is not sure how much the insight can carry over to more realistic settings. On empirical side, there are quite a few simplifications in the setting analyzed, e.g., linear model, SGD only training. While I agree theory should start simple, it would be more convincing to see experiments on more realistic settings, i.e., to test if the insights from the simple model can hold  for more practical settings, even if they do not hold, it would be still interesting to see the results.\n\nQUESTIONS:\n1. If the training algorithm is not SGD, instead it is, say Adam, how would this change the analysis result conceptually? In addition, in experiment would it change anything empirically?\n\n2. In Fig. 3, reinforcement learning and  clairvoyant methods are not compared for neural nets due to their complications. I can understand  clairvoyant might be infeasible due to high nonconvexity of the problem, but why RL methods are not feasible? It seems like the online adversarial attack could be a reinforcement learning problem by nature.\n\nLIMITATIONS:\nLimitation are sufficiently discussed.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-21"}