Accumulative Poisoning Attacks on Real-time Data
Year: 2021

Abstract:
Collecting training data from untrusted sources exposes machine learning services to poisoning adversaries, who maliciously manipulate training data to degrade the model accuracy. When trained on offline datasets, poisoning adversaries have to inject the poisoned data in advance before training, and the order of feeding these poisoned batches into the model is stochastic. In contrast, practical systems are more usually trained/fine-tuned on sequentially captured real-time data, in which case poisoning adversaries could dynamically poison each data batch according to the current model state. In this paper, we focus on the real-time settings and propose a new attacking strategy, which affiliates an accumulative phase with poisoning attacks to secretly (i.e., without affecting accuracy) magnify the destructive effect of a (poisoned) trigger batch. By mimicking online learning and federated learning on MNIST and CIFAR-10, we show that model accuracy significantly drops by a single update step on the trigger batch after the accumulative phase. Our work validates that a well-designed but straightforward attacking strategy can dramatically amplify the poisoning effects, with no need to explore complex techniques.

heading: 1 Introduction
text: In practice, machine learning services usually collect their training data from the outside world, and automate the training processes. However, untrusted data sources leave the services vulnerable to poisoning attacks [5, 28], where adversaries can inject malicious training data to degrade model accuracy. To this end, early studies mainly focus on poisoning offline datasets [36, 45, 46, 65], where the poisoned training batches are fed into models in an unordered manner, due to the usage of stochastic algorithms (e.g., SGD). In this setting, poisoning operations are executed before training, and adversaries are not allowed to intervene anymore after training begins.
On the other hand, recent work studies a more practical scene of poisoning real-time data streaming [63, 66], where the model is updated on user feedback or newly captured images. In this case, the adversaries can interact with the training process, and dynamically poison the data batches according to the model states. What’s more, collaborative paradigms like federated learning [30] share the model with distributed clients, which facilitates white-box accessibility to model parameters. To alleviate the threat of poisoning attacks, several defenses have been proposed, which aim to detect and filter out poisoned data via influence functions or feature statistics [6, 11, 17, 53, 57]. However, to timely update the model on real-time data streaming, on-device applications like facial recognition [62] and automatic driving [68], large-scale services like recommendation systems [24] may use some heuristic detection strategies (e.g., monitoring the accuracy or recall) to save computation [32].
In this paper, we show that in real-time data streaming, the negative effect of a poisoned or even clean data batch can be amplified by an accumulative phase, where a single update step can break down the model from 82.09% accuracy to 27.66%, as shown in our simulation experiments on CIFAR-10 [31]
∗Equal contribution. †Corresponding author.
35th Conference on Neural Information Processing Systems (NeurIPS 2021).
Poisoning Attacks Accumulative Poisoning Attacks
(demo in the right panel of Fig. 1). Specifically, previous online poisoning attacks [63] apply a greedy strategy to lower down model accuracy at each update step, which limits the step-wise destructive effect as shown in the left panel of Fig. 1, and a monitor can promptly intervene to stop the malicious behavior before the model is irreparably broken down. In contrast, our accumulative phase exploits the sequentially ordered property of real-time data streaming, and induces the model state to be sensitive to a specific trigger batch by a succession of model updates. By design, the accumulative phase will not affect model accuracy to bypass the heuristic detection monitoring, and later the model will be suddenly broken down by feeding the trigger batch. The operations used in the accumulative phase can be efficiently computed by applying the reverse-mode automatic differentiation [21, 50]. This accumulative attacking strategy gives rise to a new threat for real-time systems, since the destruction happens only after a single update step before a monitor can perceive and intervene. Intuitively, the mechanism of the accumulative phase seems to be analogous to backdoor attacks [38, 54], while in Sec. 3.4 we discuss the critical differences between them.
Empirically, we conduct experiments on MNIST and CIFAR-10 by simulating different training processes encountered in two typical real-time streaming settings, involving online learning [8] and federated learning [30]. We demonstrate the effectiveness of accumulative poisoning attacks, and provide extensive ablation studies on different implementation details and tricks. We show that accumulative poisoning attacks can more easily bypass defenses like anomaly detection and gradient clipping than vanilla poisoning attacks. While previous efforts primarily focus on protecting the privacy of personal/client data [42, 44, 55], much less attention is paid to defend the integrity of the shared online or federated models. Our results advocate the necessity of embedding more robust defense mechanisms against poisoning attacks when learning from real-time data streaming.

heading: 2 Backgrounds
text: In this section, we will introduce three attacking strategies, and two typical paradigms of learning from real-time data streaming. For a classifier f(x; θ) with model parameters θ, the training objective is denoted as L(x, y; θ), where (x, y) is the input-label pair. For notation compactness, we denote the empirical training objective on a data set or batch S = {xi, yi}Ni=1 as L(S; θ).

heading: 2.1 Attacking strategies
text: Below we briefly introduce the concepts of poisoning attacks [5], backdoor attacks [9], and adversarial attacks [19]. Although they may have different attacking goals, the applied techniques are similar, e.g., solving certain optimization problems by gradient-based methods.
Poisoning attacks. There is extensive prior work on poisoning attacks, especially in the offline settings against SVM [5], logistic regression [45], collaborative filtering [36], feature selection [65], and neural networks [13, 28, 29, 46, 56, 58]. In the threat model of poisoning attacks, the attacking goal is to degrade the model performance (e.g., test accuracy), while adversaries only have access to training data. Let Strain be the clean training set and Sval be a separate validation set, a poisoner will modify Strain into a poisoned P(Strain), and the malicious objective is formulated as
max P L (Sval; θ∗) , and θ∗ ∈ argmin θ L (P(Strain); θ) , (1)
where Strain and Sval are sampled from the same underlying distribution. The minimization problem is usually solved by stochastic gradient descent (SGD), where feeding data is randomized.
Backdoor attacks. As a variant of poisoning attacks, a backdoor attack aims to mislead the model on some specific target inputs [1, 16, 25, 54, 69], or inject trigger patterns [9, 22, 38, 52, 59, 61], without affecting model performance on clean test data. Backdoor attacks have a similar formulation as poisoning attacks, except that Sval in Eq. (1) is sampled from a target distribution or with trigger patterns. Compared to the threat model of poisoning attacks, backdoor attacks assume additional accessibility in inference, where the test inputs could be specified or embedded with trigger patches.
Adversarial attacks. In recent years, adversarial vulnerability has been widely studied [12, 14, 19, 41, 47, 48, 60], where human imperceptible perturbations can be crafted to mislead image classifiers. Adversarial attacks usually only assume accessibility to test data. Under `p-norm threat model, adversarial examples are crafted as
x∗ ∈ argmax x′ L(x′, y; θ), such that ‖x′ − x‖p ≤ , (2)
where is the allowed perturbation size. In the adversarial literature, the constrained optimization problem in Eq. (2) is usually solved by projected gradient descent (PGD) [41].

heading: 2.2 Learning on real-time data streaming
text: This paper considers two typical real-time learning paradigms, i.e., online learning and federated learning, as briefly described below.
Online learning. Many practical services like recommendation systems rely on online learning [3, 8, 24] to update or refine their models, by exploiting the collected feedback from users or data in the wild. After capturing a new training batch St at round t, the model parameters are updated as
θt+1 = θt − β∇θL(St; θt), (3)
where β is the learning rate of gradient descent. The optimizer can be more advanced (e.g., with momentum), while we use the basic form of gradient descent in our formulas to keep compactness.
Federated learning. Recently, federated learning [30, 43, 55] become a popular research area, where distributed devices (clients) collaboratively learn a shared prediction model, and the training data is kept locally on each client for privacy. At round t, the server distributes the current model θt to a subset It of the total N clients, and obtain the model update as
θt+1 = θt − β ∑ n∈It Gnt , (4)
where β is the learning rate and G1t , · · · , GNt are the updates potentially returned by the N clients. Compared to online learning that captures semantic data (e.g., images), the model updates received in federated learning are non-semantic for a human monitor, and harder to execute anomaly detection.

heading: 3 Accumulative poisoning attacks
text: Conceptually, a vanilla online poisoning attack [63] greedily feeds the model with poisoned data, and a monitor could stop the training process after observing a gradual decline of model accuracy. In contrast, we propose accumulative poisoning attacks, where the model states are secretly (i.e., keeping accuracy in a reasonable range) activated towards a trigger batch by the accumulative phase, and the model is suddenly broken down by feeding in the trigger batch, before the monitor gets
conscious of the threat. During the accumulative phase, we need to calculate higher-order derivatives, while by using the reverse-mode automatic differentiation in modern libraries like PyTorch [50], the extra computational burden is usually constrained up to 2 ∼ 5 times more compared to the forward propagation [20, 21], which is still efficient. This section formally demonstrates the connections between a vanilla online poisoner and the accumulative phase and provides empirical algorithms for evading the models trained by online learning and federated learning, respectively.

heading: 3.1 Poisoning attacks in real-time data streaming
text: Recent work takes up research on poisoning attacks in real-time data streaming against online SVM [7], autoregressive models [2, 10], bandit algorithms [27, 37, 40], and classification [35, 63, 66]. In this paper, we focus on the classification tasks under real-time data streaming, e.g., online learning and federated learning, where the minimization process in Eq. (1) is usually substituted by a series of model update steps [11]. Assuming that the poisoned data/gradients are fed into the model at round T , then according to Eq. (3) and Eq. (4), the real-time poisoning problems can be formulated as
max P L (Sval; θT+1) , where θT+1 = { θT − β∇θL (P(ST ); θT ) , online learning; θT − β ∑ n∈IT P(G n T ), federated learning, (5)
where IT is the subset of clients selected at round T . The poisoning operation P acts on the data points in online learning, while acting on the gradients in federated learning. To quantify the ability of poisoning attackers, we define the poisoning ratio asR(P, S) = |P(S)\S||S| , where S could be a set of data or model updates, and |S| represents the number of elements in S. Note that in Eq. (5), the poisoning operation P optimizes a shortsighted goal, i.e., greedily decreasing model accuracy at each update step, which limits the destructive effect induced by every poisoned batch, and only causes a gradual descent of accuracy. Moreover, this kind of poisoning behavior is relatively easy to perceive by a heuristic monitor [51], and set aside time for intervention before the model being irreparably broken. In contrast, we propose an accumulative poisoning strategy, which can be regarded as indirectly optimizing θT in Eq. (5) via a succession of updates, as detailed below.

heading: 3.2 Accumulative poisoning attacks in online learning
text: By expanding the online learning objective of Eq. (5) in first-order terms (up to an O(β2) error), we can rewrite the maximization problem as
max P L (Sval; θT )− β∇θL (Sval; θT )>∇θL (P(ST ); θT )
⇒min P ∇θL (Sval; θT )>∇θL (P(ST ); θT ) .
(6)
Notice that the vanilla poisoning attack only maliciously modifies ST , while keeping the pretrained parameters θT uncontrolled. Motivated by this observation, a natural way to amplify the destructive effects (i.e., obtain a lower value for the minimization problem in Eq. (6)) is to jointly poison θT and ST . Although we cannot directly manipulate θT , we exploit the fact that the data points are captured sequentially. We inject an accumulative phase A to make A(θT )1 be more sensitive to the clean batch ST or poisoned batch P(ST ), where we call ST or P(ST ) as the trigger batch in our methods. Based on Eq. (6), accumulative poisoning attacks can be formulated as
min P,A ∇θL (Sval;A(θT ))>∇θL (P(ST );A(θT )) , (7)
where L (Sval;A(θT )) ≤ L (Sval; θT ) + γ, and γ is a hyperparameter controlling the tolerance on performance degradation in the accumulative phase, in order to bypass monitoring on model accuracy.
Implementation of A. Now, we describe how to implement the accumulative phase A. Assuming that the online process begins from a burn-in phase, resulting in θ0, and let S0, · · · , ST−1 be the clean online data batches at rounds 0, · · · , T − 1 after the burn-in phase. The accumulative phase iteratively trains the model on the perturbed data batch At(St), update the parameters as
θt+1 = θt − β∇θL(At(St); θt). (8) 1The notation A(θT ) refers to the model parameters at round T obtained after the accumulative phase.
Algorithm 1 Accumulative poisoning attacks in online learning
Input: Burn-in parameters θ0; training batches St = {xti, yti}Ni=1, t ∈ [0, T ]; validation batch Sval. Initialize P(ST ) = ST ; for t = 0 to T−1 do
Initialize At(St) = St; Bootstrap Sval, and/or normalize∇θL(St; θt),∇θL(ST ; θt),∇θL(Sval, θt); # optional for c = 1 to C do
Compute Gt = ∇θ ( ∇θL(Sval, θt)>∇θL(ST ; θt) ) ;
Compute Ht = ∇θL(St; θt)> [ ∇θL(S-t; θt) + λ ·Gt ] , where - stops gradients;
Update At(xti) = proj ( At(xti) + α · sign(∇xtiHt) ) for i ∈ [1, N ]; # update At(St)
Update P(xTi ) = proj ( P(xTi ) + α · sign(∇xTi Ht) ) for i ∈ [1, N ]; # update P(ST ) end for Update θt+1 = θt − β∇θL(At(St); θt); # feed in At(St) end for Update θT+1 = θT − β∇θL(P(ST ); θT ); # feed in P(ST ) Return: The poisoned parameters θT+1.
According to the malicious objective in Eq. (7) and the updating rule in Eq. (8), we can craft the perturbed data batch A(St) at round t by solving (under first-order expansion)
max P,At
∇θL(At(St); θt)> [ ∇θL(St; θt) + λ · ∇θ ( ∇θL(Sval,A(θT ))>∇θL(P(ST );A(θT )) )] ⇒max P,At ∇θL(At(St); θt)> [ ∇θL(St; θt)︸ ︷︷ ︸ keeping accuracy +λ · ∇θ ( ∇θL(Sval, θt)>∇θL(P(ST ); θt) )︸ ︷︷ ︸ accumulating poisoning effects for the trigger batch ] , (9)
where t ∈ [0, T − 1] (we abuse the notation [a, b] to denote the set of integers from a to b). Specifically, in the first line of Eq. (9), ∇θL(St; θt) is the gradient on the clean batch St, and ∇θ ( ∇θL(Sval,A(θT ))>∇θL(P(ST );A(θT )) ) is the gradient of the minimization problem in Eq. (7). Solving the maximization problem in Eq. (9) is to make the accumulative gradient ∇θL(At(St); θt) to align with ∇θL(St; θt) and ∇θ ( ∇θL(Sval,A(θT ))>∇θL(P(ST );A(θT )) ) simultaneously, with a trade-off hyperparameter λ. In the second line, since we cannot calculate A(θT ) in advance, we greedily approximate A(θT ) by θt in each accumulative step. In Algorithm 1, we provide an instantiation of accumulative poisoning attacks in online learning. At the beginning of each round of accumulation, it is optional to bootstrap Sval to avoid overfitting, and normalize the gradients to concentrate on angular distances. When computing Ht, we apply stopping gradients (e.g., the detach operation in PyTorch [50]) to control the back-propagation flows.
Capacity of poisoners. To ensure that the perturbations are imperceptible for human observers, we follow the settings in the adversarial literature [19, 60], and constrain the malicious perturbations on data into `p-norm bounds. The update rules of P and At are based on projected gradient descent (PGD) [41] under `∞-norm threat model, where iteration steps C and step size α, and maximal perturbation are hyperparameters. Other techniques like GANs [18] can also be applied to generate semantic perturbations, while we do not further explore them.
Poisoning ratios. The ratios of poisoned data have different meanings in online/real-time and offline settings. Namely, in real-time settings, we only poison data during the accumulative phase. If we ask the ratio of poisoned data points that are fed into the model, the formula should be
Per-batch poisoning ratio× Accumulative epochs Burin-in epochs + Accumulative epochs .
So for example in Fig. 1, even if we use 100% per-batch poisoning ratio during the accumulative phase for 2 epochs, the ratio of poisoned data points fed into the model is only 100%×2/(40+2) ≈ 4.76%, where 40 is the number of burn-in epochs. In contrast, if we poison 10% data in an offline dataset, then the expected ratio of poisoned data points fed into the model is also 10%.
Algorithm 2 Accumulative poisoning attacks in federated learning
Input: Burn-in parameters θ0; training updates {Gnt }Nn=1, t ∈ [0, T ], where we assume that GnT is computed by the local data batch SnT as G n T = ∇θL(SnT ; θT ); validation batch Sval. Input: Sampled index sets It, where t ∈ [0, T ]. # access to random seeds Initialize P(GnT ) = projη(−∇θL(SnT ; θ0)) for n ∈ IT ; # reverse trigger for t = 0 to T−1 do
Sample random vectors {Mnt }Nn=1 such that ∑N i=1M n t = 0; Initialize At(Gnt ) =Mnt for n ∈ It; Bootstrap Sval, and/or normalize∇θL(Sval, θt); # optional Update P(GnT ) = projη(P(GnT )− α · ∇θL(SnT ; θt)) for n ∈ IT ; Compute Ht = ∑ n∈It G n t + λ · ∇θ ( ∇θL (Sval; θt)> ∑ n∈It P(G n T ) )
; Update At(Gnt ) = projη(At(Gnt ) +Ht) for n ∈ It; Update θt+1 = θt − β ∑ n∈It At(G n t ); # feed in At(Gnt ), n ∈ It
end for Update θT+1 = θT − β ∑ n∈IT P(G n T ); # feed in P(GnT ), n ∈ IT Return: The poisoned parameters θT+1.

heading: 3.3 Accumulative poisoning attacks in federated learning
text: Similar as the derivations in Eq. (6) and Eq. (7), under first-order expansion, accumulative poisoning attacks in federated learning can be formulated by the minimization problem
min P,A ∇θL (Sval;A(θT ))> ∑ n∈IT P(GnT ), (10)
such that L (Sval;A(θT )) ≤ L (Sval; θT ) + γ. Assuming that the federated learning process begins from a burn-in process, resulting in a shared model of parameters θ0, and then being induced into an accumulative phase from round 0 to T−1. At round t, the accumulative phase updates the model as
θt+1 = θt − β ∑ n∈It At(Gnt ). (11)
According to the formulations in Eq. (10) and Eq. (11), the perturbation At can be obtained by
max P,At (∑ n∈It At(Gnt ) )> [∑ n∈It Gnt + λ · ∇θ ( ∇θL (Sval; θt)> ∑ n∈IT P(GnT ) )] , (12)
where λ is a trade-off hyperparameter similar as in Eq. (9), and A(θT ) is substituted by θt at each round t. In Algorithm 2, we provide an instantiation of accumulative poisoning attacks in federated learning. The random vectors Mnt are used to avoid the model updates from different clients being the same (otherwise, a monitor will perceive the abnormal behaviors). We assume white-box access to random seeds, while black-box cases can also be handled [4]. Different from the case of online learning, we do not need to run C-steps PGD to update the poisoned trigger or accumulative batches.
Capacity of poisoners. Unlike online learning in which the captured data is usually semantic, the model updates received during federated learning are just numerical matrices or tensors, thus a human observer cannot easily perceive the malicious behaviors. If we allow P and At to be arbitrarily powerful, then it is trivial to break down the trained models. To make the settings more practical, we clip the poisoned gradients under `p-norm, namely, we constrain that ∀n ∈ IT and t ∈ [0, T−1], there are ‖P(GnT )‖p ≤ η and ‖At(Gnt )‖p ≤ η, where η has a similar role as the perturbation size in adversarial attacks. Empirical results under gradient clipping can be found in Sec. 4.2.
Reverse trigger. When initializing the poisoned trigger, we apply a simple trick of reversing the clean trigger batch. Specifically, the gradient computed on a clean trigger batch ST at round t is ∇θL(ST ; θt). A simple way to increase the validate loss value is to reverse the model update to be −∇θL(ST ; θt), which convert the accumulative objective in Eq. (10) as
min At ∇θL(ST ;At(θt))>∇θL(Sval;At(θt)) =⇒ max At ∇θL(ST ;At(θt))>∇θL(Sval;At(θt)), (13)
where the later objective is easier to optimize since for a burn-in model, the directions of∇θL(ST ; θt) and ∇θL(Sval; θt) are approximately aligned, due to the generalization guarantee. This trick can maintain the gradient norm unchanged, and does not exploit the validation batch for training.
Recovered offset. Let Gt = {Gnt }n∈It be the gradient set received at round t, then if we can only modify a part of these gradients, i.e., R(At, Gt) < 1, we can apply a simple trick to recover the original optimal solution. Technically, assuming that we can only modify the clients in I ′t ⊂ It, and the optimal solution of At in Eq. (12) is A∗t , then we can modify the clients according to∑
n∈I′t At(Gnt ) = ∑ n∈It A∗t (Gnt )− ∑ n∈It\I′t Gnt , whereR(At, Gt) = |I ′t| |It| . (14)
The trick shown in Eq. (14) can help us to eliminate the influence of unchanged model updates, and stabilize the update directions to follow the malicious objective during the accumulative phase.

heading: 3.4 Differences between the accumulative phase and backdoor attacks
text: Although both the accumulative phase and backdoor attacks [22, 38] can be viewed as making the model be sensitive to certain trigger batches, there are critical differences between them:
(i) Data accessibility and trigger occasions. Backdoor attacks require accessibility on both training and test data, while our methods only need to manipulate training data. Besides, backdoor attacks trigger the malicious behaviors (e.g., fooling the model on specific inputs) during inference, while in our methods the malicious behaviors (e.g., breaking down the model accuracy) are triggered during training. (ii) Malicious objectives. We generally denote a trigger batch as Stri. For backdoor attacks, the malicious objective can be formulated as maxB L(Stri;B(θ)), where B is the backdoor operations. In contrast, our accumulative phase optimizes minA∇θL (Sval;A(θ))>∇θL (Stri;A(θ)).

heading: 4 Experiments
text: We mimic the real-time data training using the MNIST and CIFAR-10 datasets [31, 33]. The learning processes are similar to regular offline pipelines, while the main difference is that the poisoning attackers are allowed to intervene during training and have access to the model states to tune their attacking strategies dynamically. Following [49], we apply ResNet18 [23] as the model architecture, and employ the SGD optimizer with momentum of 0.9 and weight decay of 1× 10−4. The initial learning rate is 0.1, and the mini-batch size is 100. The pixel values of images are scaled to be in the interval of 0 to 1.2
Burn-in phase. For all the experiments in online learning and federated learning, we pre-train the model for 10 epochs on the clean training data of MNIST, and 40 epochs on the clean training data of CIFAR-10. The learning rate is kept as 0.1, and the batch normalization (BN) layers [26] are in train mode to record feature statistics.
Poisoning target. Usually, when training on real-time data streaming, there will be a background process to monitor the model accuracy or recall, such that the training progress would be stopped if the monitoring metrics have significant retracement. Thus, we exploit single-step drop as a threatening poisoning target, namely, the accuracy drop after the model is trained by a single step with poisoned behaviors (e.g., trained on a batch of poisoned data). Single-step drop can measure the destructive effect caused by poisoning strategies, before monitors can react to the observed retracement.
2Code is available at https://github.com/ShawnXYang/AccumulativeAttack.
Table 4: Results when using longer burn-in phase on CIFAR-10 (i.e., running the burn-in phase for 100 epochs, compared to 40 epochs in Table 3).
Method Loss No `∞-norm clip boundscaling clip 10 1 0.1
1 89.34 89.34 89.91 89.99 Poisoned 10 45.29 84.45 89.91 89.99 trigger 20 16.62 84.45 89.91 89.99
50 10.24 84.45 89.91 89.99
0.01 80.35 80.35 80.35 83.32 Accu. 0.02 25.45 25.45 25.45 76.06 phase 0.05 12.03 12.03 15.53 70.00
0.08 11.07 11.07 14.23 64.74
Table 5: Classification accuracy (%) on MNIST after the model updating by the trigger batch. The accumulative phase runs for 200 steps ( 13 epochs), with perturbation constraint = 16/255 and step size α = 2/255.
Method Loss No `∞-norm clip boundscaling clip 10 1 0.1
1 98.27 98.27 98.27 98.28 Poisoned 10 95.49 95.49 98.12 98.28
trigger 20 84.09 89.24 98.12 98.28 50 31.93 89.24 98.12 98.28
Accu. 0.08 22.49 22.49 32.87 51.28phase

heading: 4.1 Performance in the simulated experiments of online learning
text: At each step of the accumulative phase in online learning, we obtain a training batch from the ordered data streaming, and craft accumulative poisoning examples. The perturbations are generated by PGD [41], and restricted to some feasible sets under `∞-norm. To keep the accumulative phase being ‘secret’, we will early-stop the accumulative procedure if the model accuracy becomes lower than a specified range (e.g., 80% for CIFAR-10). We evaluate the effects of the accumulative phase in online learning using a clean trigger batch and poisoned trigger batches with different perturbation budgets . We set the number of PGD iterations as C = 100, and the step size is α = 2 · /C. Table 1 reports the test accuracy before and after the model is updated on the trigger batch. The single-step drop caused by the vanilla poisoned trigger is not significant, while after combining with our accumulative phase, the poisoner can make much more destructive effects. As seen, we obtain prominent improvements by introducing two simple techniques for the accumulative phase, including weight momentum which lightly increases the momentum factor as 1.1 (from 0.9 by default) of the SGD optimizer in accumulative phase, and optimizing P that constantly updates adversarial poisoned trigger in the accumulative phase. Besides, we re-sample different Sval to demonstrate a consistent performance in the setting named re-sampling Sval. We also study the effectiveness of setting different data poisoning ratios, as summarized in Table 2. To mimic more practical settings, we also do simulated experiments on the models using group normalization (GN) [64], as detailed in Appendix A.1. In Fig. 4 we visualize the training data points used during the burn-in phase (clean) and used during the accumulative phase (perturbed under = 16/255 constraint). As observed, the perturbations are hardly perceptible, while we provide more instances in Appendix A.2.
Anomaly detection. We evaluate the performance of our methods under anomaly detection. Kernel density (KD) [15] applies a Gaussian kernel K(z1, z2) = exp(−‖z1 − z2‖22/σ2) to compute the similarity between two features z1 and z2. For KD, we restore 1, 000 correctly classified training features in each class and use σ = 10−2. Local intrinsic dimensionality (LID) [39] applies K nearest neighbors to approximate the dimension of local data distribution. We restore a total of 10, 000 correctly classified training data points, and set K = 600. We also consider two model-based
detection methods, involving Gaussian mixture model (GMM) [67] and Gaussian discriminative analysis (GDA) [34]. In Fig. 2, the metric value for KD is the kernel density, for LID is the negative local dimension, for GDA and GMM is the likelihood. As observed, the accumulative poisoning samples can better bypass the anomaly detection methods, compared to the samples crafted by the vanilla poisoning attacks.

heading: 4.2 Performance in the simulated experiments of federated learning
text: In the experiments of federated learning, we simulate the model updates received from clients by computing the gradients on different mini-batches, following the setups in Konečnỳ et al. [30], and we synchronize the statistics recorded by local BN layers. We first evaluate the effect of the accumulative phase along, by using a clean trigger batch. We apply the recovered offset trick (described in Eq. (14)) to eliminate potential influences induced by limited poisoning ratio, and perform ablation studies on the values of λ in Eq. (12). As seen in Fig. 3 (b), we run the accumulative phase for different numbers of steps under each value of λ, and report the test accuracy before and after the model is updated on the trigger batch. As intuitively indicated, a point towards the bottom-right corner implies more secret before trigger batch (i.e., less accuracy drop and not easy to be perceived by monitor), while more destructive after the trigger batch. We can find that a modest value of λ = 0.02 performs well.
In Table 3, Table 4, and Table 5, we show that the accumulative phase can mislead the model with smaller norms of gradients, which can better bypass the clipping operations under both `2 and `∞ norm cases. In contrast, previous strategies of directly poisoning the data batch to degrade model accuracy would require a large magnitude of gradient norm, and thus is easy to be defended by gradient clipping. On the other hand, executing gradient clipping is not for free, since it will lower down the convergence of model training, as shown in Fig. 3 (a). Finally, in Table 6, we show that exploiting poisoned trigger batch can further improve the computational efficiency of the accumulative phase, namely, using fewer accumulative steps to achieve a similar accuracy drop.

heading: 5 Conclusion
text: This paper proposes a new poisoning strategy against real-time data streaming by exploiting an extra accumulative phase. Technically, the accumulative phase secretly magnifies the model’s sensitivity to a trigger batch by sequentially ordered accumulative steps. Our empirical results show that accumulative poisoning attacks can cause destructive effects by a single update step, before a monitor can perceive and intervene. We also consider potential defense mechanisms like different anomaly detection methods and gradient clipping, where our methods can better bypass these defenses and break down the model performance. These results can inspire more real-time poisoning strategies, while also appeal to strong and efficient defenses that can be deployed in practical online systems.

heading: Acknowledgements
text: This work was supported by the National Key Research and Development Program of China (Nos. 2020AAA0104304, 2017YFA0700904), NSFC Projects (Nos. 61620106010, 62061136001, 61621136008, 62076147, U19B2034, U19A2081, U1811461), Beijing Academy of Artificial Intelligence (BAAI), Tsinghua-Huawei Joint Research Program, a grant from Tsinghua Institute for Guo Qiang, Tsinghua University-China Mobile Communications Group Co.,Ltd. Joint Institute, Tiangong Institute for Intelligent Computing, and the NVIDIA NVAIL Program with GPU/DGX Acceleration.

Review summaries:
This paper presents a novel poisoning attack strategy for online learning settings, where the parameters of the machine learning model are updated as new batches of data are collected. For this, the authors propose an accumulative poisoning attack strategy with two phases. In the first phase, the attacker poisons the model in a stealthy way, so that the performance of the model is not affected, so that it magnifies the destructive effect of the poisoning attack conducted in the second phase, where the attacker injects poisoning points in a batch of samples to decrease the performance of the model. The authors shows that this attack strategy can also be applied for compromising federated learning. 

Data poisoning attacks usually target the offline setting and do not fully utilize the sequential nature of data utilization in the online learning scenario. This work aims to optimize the damage of a trigger batch during the training phase (akin to an availability attack) by adding selective perturbations to data leading up to the trigger batch. At that point, the trigger batch can cause significantly more damage than a vanilla trigger batch inserted in the data stream. The authors derive optimization requirements for the perturbations in this 'accumulative phase' and the trigger batch, and empirically demonstrate how the proposed attack can outperform other methods, all while being more robust to heuristics like gradient-clipping.

This paper proposes a poisoning attack in settings where data is acquired iteratively. Two concrete settings are examined: online learning and federated learning. Under the assumed threat model, an attacker may observe the state of the learning algorithm and poison the data it consumes in each iteration. In order to evade detection, a novel attack strategy is proposed where the learned model is primed over several iterations in a so-called _accumulative phase_ without causing harm. After the accumulative phase, the attack can be activated by a special trigger data batch to cause a sudden drop in model accuracy. The attack is formulated as an optimization problem, which is ultimately relaxed and solved using projected gradient descent. Experiments are performed using CIFAR-10 to understand the power of the attack for different parameter settings and variations of the accumulative phase. The results show that the attack is effective and can more easily bypass anomaly detection and gradient clipping defenses.