Accumulative Poisoning Attacks on Real-time Data
Authors: Tianyu Pang
Xiao Yang
Yinpeng Dong
Hang Su
Jun Zhu
Venue: NeurIPS 2021 Poster (2021)
Keywords: Poisoning attacks
Real-time data

Abstract: Collecting training data from untrusted sources exposes machine learning services to poisoning adversaries, who maliciously manipulate training data to degrade the model accuracy. When trained on offline datasets, poisoning adversaries have to inject the poisoned data in advance before training, and the order of feeding these poisoned batches into the model is stochastic. In contrast, practical systems are more usually trained/fine-tuned on sequentially captured real-time data, in which case poisoning adversaries could dynamically poison each data batch according to the current model state. In this paper, we focus on the real-time settings and propose a new attacking strategy, which affiliates an accumulative phase with poisoning attacks to secretly (i.e., without affecting accuracy) magnify the destructive effect of a (poisoned) trigger batch. By mimicking online learning and federated learning on MNIST and CIFAR-10, we show that model accuracy significantly drops by a single update step on the trigger batch after the accumulative phase. Our work validates that a well-designed but straightforward attacking strategy can dramatically amplify the poisoning effects, with no need to explore complex techniques.

Accumulative Poisoning Attacks on Real-time Data
Tianyu Pang",, Xiao Yang"', Yinpeng Dong! Hang Su 1.3 'Department of Computer Science & Technology, Institute for AI, BNRist Center, Sainghu--Boscl Joint ML Center, THBI Lab, Tsinghua University RRealAI Tsinghua Hniversity-Chine Mobile Communications Group Co., Ltd. Joint Institute {pty17, yangxiao19 http::iiio9..yyy111.. dyp17) dypi7)8uaile. Owaile. t tsinghus, eanugbus.edu..com edu. ca, (suhanges, doezj tta edu. cn
Sul Jun Zhu 1,2,33
Abstract
Collecting training data from untrusted sources exposes machine learning services to poisoning adversaries, who maliciously manipulate training data to degrade the model accuracy. When trained on offline datasets, poisoning adversaries have to inject the poisoned data advance before training, and the order of feeding these poisoned batches into the model is stochastic. In contrast, practical systems are more usually wraindd/fin--tnne on sequentially captured real-time data, in which case poisoning adversaries could dynamically poison each data batch according to the current model state. this paper, we focus on the real-time settings and propose a new attacking strategy, which affiliates an accumulative phase with poisoning attacks to secretly (i.e., without affecting accuracy) magnify the destructive effect a (poisoned) trigger batch. By mimicking online learning and federated learning on MNIST and CIFAR-1O, we show that model accuracy significantly drops by a single update step the trigger batch after the accumulative phase. Our work validates that a well-designed but straightforward attacking strategy can dramatically amplify the poisoning effects, with need to explore complex techniques.
Introduction
In practice, machine learning services usually collect their training data from the outside world, and automate the training processes. However, untrusted data sources leave the services vulnerable to poisoning attacks [5, 28), where adversaries can inject malicious training data degrade model accuracy. To this end, early studies mainly focus on poisoning offline datasets [36, 45, 46, 65], where the poisoned training batches are fed into models in unordered manner, due to the usage of stochastic algorithms (e.g., SGD). In this setting, poisoning operations are executed before training, and adversaries are not allowed to intervene anymore after training begins.
the other hand, recent work studies a more practical scene of poisoning real-time data stream- ing [63, 66], where the model is updated on user feedback or newly captured images. In this case, the adversaries can interact with the training process, and dynamically poison the data batches according to the model states. What's more, collaborative paradigms like federated learning [30] share the model with distributed clients, which facilitates white-box accessibility to model parameters. To alleviate the threat of poisoning attacks, several defenses have been proposed, which aim to detect and filter out poisoned data via influence functions or feature statistics [6. 11, 17, 53, 57]. However, to timely update the model on real-time data streaming, on-device applications like facial recognition [62] and automatic driving [68], large-scale services like recommendation systems [24] may use some heuristic detection strategies (e.g., monitoring the accuracy recall) to save computation [32].
In this paper, we show that in real-time data streaming, the negative effect a poisoned or even clean data batch can be amplified by accumulative phase, where single update step break down the model from 82.09% accuracy to 27.66%, as shown in our simulation experiments on CIFAR- 10 [31]
Equal contribution. *Corresponding author.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

Poisoning Attacks
Accumulative Poisoning Attacks
Test before persooegg butch:
90
Accumulative phase
83.38%
securacy before (ckam) trigger batch: 82,09%
Test accuracy (%)
Test accuracy (%)
70
Ted afler paisoned batch:
60
60
72,0%%
50
-0
Burb-in phuse
Baru-ia phase
Test accuncyy after (G trigger batch: 27.66%
20
Training epochs
Training opochs
The plots are visualized from the results in Table 3, where gradients are clipped by 10 under 22-norm to simulate practical scenes. The model architecture is ResNet-18, and the batch size is 100 on CIFAR-I0. A burn-in phase first trains the model for 40 epochs (20, 000 update steps). (Left) Poisoning attacks. The burn-in model is fed with a poisoned training batch [63], after which the accuracy drops from 83.38% to 72.07%. (Right) Accumulative poisoning attacks. The burn-in model is 'secretly' poisoned by an accumulative phase for 2 epochs (1, 000 update steps), while keeping test accuracy in a heuristically reasonable range of variation. Then trigger batch is fed into the model after the accumulative phase, and the model accuracy is broken down from 82.09% to 27. 7.66% by single update step. Note that we only clean trigger batch, while the destructive effect can be more significant if we further exploit a poisoned trigger batch as in Table
Figure 1:
(demo in the right panel of Fig. 1). Specifically, previous online poisoning attacks [63] apply greedy strategy to lower down model accuracy at each update step, which limits the step-wise destructive effect shown in the left panel of Fig. 1, and a monitor can promptly intervene stop the malicious behavior before the model is irreparably broken down. In contrast, our accumulative phase exploits the sequentially ordered property of real-time data streaming, and induces the model state be sensitive to specific trigger batch by succession of model updates. By design, the accumulative phase will not affect model accuracy to bypass the heuristic detection monitoring, and later the model will be suddenly broken down by feeding the trigger batch. The operations used in the accumulative phase can be efficiently computed by applying the reverse-mode automatic differentiation [21, 50]. This accumulative attacking strategy gives rise to new threat for real-time systems, since the destruction happens only after a single update step before a monitor can perceive and intervene. Intuitively, the mechanism of the accumulative phase seems to be analogous to backdoor attacks [38, 54J, while in Sec. 3.4 we discuss the critical differences between them.
Empirically, we conduct experiments on MNIST and CIFAR-10 10 by simulating different training processes encountered in two typical real-time streaming settings, involving online learning [8] and federated learning [30]. We demonstrate the effectiveness of accumulative poisoming attacks, and provide extensive ablation studies on different implementation details and tricks. We show that accumulative poisoning attacks can more easily bypass defenses like anomaly detection and gradient clipping than vanilla poisoning attacks. While previous efforts primarily focus on protecting the privacy of personal/client data [42, 44, 55], much less attention is paid to defend the integrity of the shared online or federated models. Our results advocate the necessity of embedding more robust defense mechanisms against poisoning attacks when learning from real-time data streaming.
2 Backgrounds
In this section, we will introduce three attacking strategies, and two typical paradigms of learning from real-time data streaming. For classifier (x; 0) with model parameters O, the training objective is denoted L(x, y; 8), where (xr, y) is the lnput-label pair. For notation compactness, denote the empirical training objective on data set or batch S = ( 8c as L(S:)).
2.1 Attacking strategies
Below we briefly introduce the concepts of poisoning attacks [5], backdoor attacks [9], and adversarial attacks [19]. Although they may have different attacking goals, the applied techniques are similar, e.g., solving certain optimization problems by gradient-based methods.

---

Poisoning attacks. There is extensive prior work on poisoning attacks, especially in the offline settings against SVM [5]. logistic regression [45], collaborative filtering [36], feature selection [65]. and neural networks [13, 28, 29, 46, 56, 58]. In the threat model of poisoning attacks, the attacking goal is to degrade the model performance (e.g.. test accuracy), while adversaries only have access to training data. Let Stnin be the clean training set and Sval be separate validation set, a poisoner will modify Stain into a poisoned P(Smain, and the malicious objective is formulated as
max & (Syal 0*), and 0* arg min C (P(Sna)))
where Suain and Sval are sampled from the same underlying distribution. The minimization problem is usually solved by stochastic gradient descent (SGD), where feeding data randomized.
Backdoor attacks. As variant of poisoning attacks, backdoor attack aims to mislead the model on some specific target inputs [1, 16, 25, 54, 69], or inject trigger patterns [9, 22, 38, 52, 59, 61], without affecting model performance on clean test data. Backdoor attacks have similar formulation as poisoning attacks, except that Sval in Eq. (1) is sampled from a target distribution or with trigger patterns. Compared to the threat model of poisoning attacks, backdoor attacks assume additional accessibility in inference, where the test inputs could be specified or embedded with trigger patches.
Adversarial attacks. In recent years, adversarial vulnerability has been widely studied [12, 14, 19, 41, 47, 48, 60], where human imperceptible perturbations can be crafted mislead image classifiers. Adversarial attacks usually only assume accessibility to test data. Under lp ---norm threat model, adversarial examples are crafted as
x² E arg max L(x', y; 0), such that |ar'=zll
2111 S,
where is the allowed perturbation size. In the adversarial literature, the constrained optimization problem in Eq. (2) is usually solved by projected gradient descent (PGD) [41].
2.2 Learning on real-time data streaming
This paper considers two typical real-time learning paradigms, i.e., online learning and federated learning, as briefly described below.
Online learning. Many practical services like recommendation systems rely on online learning [3, 8, 24] to update or refine their models, by exploiting the collected feedback from users data in the wild. After capturing a new training batch S; at round t, the model parameters are updated as
oc+1 = 0g 3V o))
where is the learning rate of gradient descent. The optimizer can be more advanced (e.g., with momentum), while we use the basic form of gradient descent in our formulas to keep compactness.
Federated learning. Recently, federated learning [30, 55) become a popular research area, where distributed devices (clients) collaboratively learn a shared prediction model, and the training data is kept locally on each client for privacy. At round I, the server distributes the current model O, to subset le of the total N clients, and obtain the model update as
0.4 0: - Σ EG. nET,
where B is the learning rate and ..,..., G.N are the updates potentially returned by the N clients. Compared to online learning that captures semantic data (e.g., images), the model updates received in federated learning are non-semantic for human monitor, and harder execute anomaly detection.
3 Accumulative poisoning attacks
Conceptually, a vanilla online poisoning attack [63] greedily feeds the model with poisoned data, and a monitor could stop the training process after observing a gradual decline of model accuracy. In contrast, we propose accumulative poisoning attacks, where the model states are secretly (i.e., keeping accuracy in reasonable range) activated towards a trigger batch by the accumulative phase, and the model is suddenly broken down by feeding in the trigger batch, before the monitor gets
3

---

conscious of the threat. During the accumulative phase, we need to calculate higher-order derivatives, while by using the reverse-mode automatic differentiation in modern libraries like PyTorch [50], the extra computational burden is usually constrained up to 2 ~ 5 times more compared to the forward propagation [20, 21], which still efficient. This section formally demonstrates the connections between vanilla online poisoner and the accumulative phase and provides empirical algorithms for evading the models trained by online learning and federated learning, respectively.
3.1 Poisoning attacks in real-time data streaming
Recent work takes up research on poisoning attacks in real-time data streaming against online SVM [7], autoregressive models [2, 10]. bandit algorithms [27, 37, 40J, and classification [35, 63, 66]. In this paper, we focus on the classification tasks under real-time data streaming, e.g., online learning and federated learning, where the minimization process in Eq. (1) usually substituted by a series of model update steps [11]. Assuming that the poisoned data/ data/gradients are fed into the model at round T, then according to Eq. (3) and Eq. (4), the real-time poisoning problems can be formulated as
0r VoL max C (Svali 0T+1), where 0T+1 - (0r 3 EnEI, P(GT),
online learning; federated learning,
(5)
where T is the subset of clients selected at round T. The poisoning operation P acts on the data points in online learning, while acting on the gradients in federated learning. To quantify the ability of poisoning attackers, we define the poisoning ratio as R(P, 5) = [P(S).5] where S could be a set [8]
of data model updates, and |S] represents the number of elements in S.
Note that in Eq. (5), the poisoning operation P optimizes a shortsighted goal, i... greedily decreasing model accuracy at each update step, which limits the destructive effect induced by every poisoned batch, and only causes a gradual descent of accuracy. Moreover, this kind of poisoning behavior relatively easy to perceive by heuristic monitor [51], and set aside time for intervention before the model being irreparably broken. In contrast, we propose an accumulative poisoning strategy, which can be regarded as indirectly optimizing Or in Eq. (5) via succession of updates, detailed below.
3.2 Accumulative poisoning attacks in online learning
By expanding the online learning objective of Eq. (5) in first-order terms (up an O(32) error), we can rewrite the maximization problem as
(S 0r) (P)()))): (
max L (Swa:: 0r)
min VoL (T) VoL (P(ST):
Notice that the vanilla poisoning attack only maliciously modifies Sy, while keeping the pretrained parameters Or uncontrolled. Motivated by this observation, a natural way to amplify the destructive effects (i.e., obtain lower value for the minimization problem in Eq. (6) is to jointly poison OT and Sr. Although we cannot directly manipulate fr, we exploit the fact that the data points are captured sequentially. We inject an accumulative phase A to make A(Or)) ! be more sensitive to the clean batch ST or poisoned batch P(Sr), where we call Sr or P(ST) as the trigger batch in our methods. Based on Eq. (6), accumulative poisoning attacks can formulated as
min VoL (Sval: A(Br): VoL(P(S): P.A
where L (Svai A(Or)) S performance degradation in the accumulative phase, in order to bypass monitoring model accuracy.
L (SSai: 0r) + 77, and y is a hyperparameter controlling the tolerance on
Implementation of A. Now, we describe how to implement the accumulative phase A. Assuming that the online process begins from a burn-in phase, resulting in Og. and let So.... ST be the clean online data batches at rounds 0, ... T after the burn-in phase. The accumulative phase iteratively trains the model the perturbed data batch A, update the parameters as
0++1 0, (S), 0.).
(8)
'The notation A(Or) refers the model parameters round ''' obtained after the accumulative phase.

---

Algorithm Accumulative poisoning attacks in online learning
Input: Burn-in parameters Op: training batches S (1f, of - 1 [0, 7); validation batch S Initialize P(Sr)= = Sr;
for 1o -
Initialize A.() S;;
Bootstrap Svat. and/or normalize VoL(S,; VoL( VoL(S VoC(SS for 1 to C do
# optional
Compute G2 Ve (V
Compute H, V V.L(S;;0) (Vo
G.], where stops gradients:
Update A (x proj, (A((z)) + sign V ( for [1,
update As(Si)
Update P(x!' = proje
(P(x;) + sign V Hi) for i [1,
update P(ST)
end for Update 0c+1 = 0t VV (VoL(A
# #feed in Ac(St)
end for
Update 0r+1 = 0r VV ( (St): Br);
# feed in P(Sr)
Return: The poisoned parameters 0T+1.
According to the malicious objective in Eq. (7) and the updating rule in Eq. (8), we can craft the perturbed data batch A(S)) at round by solving (under first-order expansion)
max (o.((A 0.) P.
Vo AAA VoL(S. 0.) V A(0r))
max VoL(A(())):
VoL(S,;0.) +X V V.
keeping accuracy
accumulating poisoning effects for the trigger batch
where E 10, 1] (we abuse the notation [a, to denote the set of integers from to b). Specifically, in the first line of Eq. (9), (o,(S);)))) is the gradient on the clean batch S, and Ve (V A(0r)) "Tor))((()(((((S))) (())
is the gradient of the minimization problem in Eq. (7). Solving the maximization problem in Eq. (9) is to make the accumulative gradient si-
(o 0e) to align with VoL(S,; 02) and V (VoL(S A(Or)) VoL(P(ST): multaneously, with a trade-off hyperparameter X. the second line, since we cannot calculate A(0r) in advance, we greedily approximate A(OT) by 0, in each accumulative step.
In Algorithm 1, we provide an instantiation of accumulative poisoning attacks in online learning. At the beginning of each round of accumulation, it is optional to bootstrap Sta to avoid overfitting, and normalize the gradients to concentrate on angular distances. When computing H,, we apply stopping gradients (e.g., the detach operation in PyTorch [50]) to control the back-propagation flows.
Capacity of poisoners. To ensure that the perturbations follow the settings in the adversarial literature [19, 60], and constrain the malicious perturbations on data into f,-norm bounds. The update rules of P and A, are based projected gradient descent (PGD) [41] under -oom threat model, where iteration steps C and step size a, and maximal perturbation are hyperparameters. Other techniques like GANs [18] also be applied generate semantic perturbations, while we do not further explore them.
imperceptible for human observers,
Poisoning ratios. The ratios of poisoned data have different meanings in online/real-tin time and offline settings. Namely, real-time settings, we only poison data during the accumulative phase. If we ask the ratio of poisoned data points that are fed into the model, the formula should be
Per-batch poisoning ratio Accumulative epochs
Burin-in epochs + Accumulative epochs
So for example in Fig. 1, even we use 100% per-batch poisoning ratio during the accumulative phase for 2 epochs, the ratio of poisoned data points fed into the model is only 100% x 2/(40++2 x 4.76%, where 40 is the number of burn-in epochs. In contrast, if we poison 10% data an offline dataset, then the expected ratio of poisoned data points fed into the model is also 10%.
5

---

Algorithm 2 Accumulative poisoning attacks in federated learning
Input:
[ Burn-in parameters Oo: training updates {G) === I [0, where assume that CTp. is
computed by the local data batch Sy. as G.j- = (oL(S 0r): validation batch Sval- Input:
Sampled index sets 4. where E [0. 7].
access random seeds # reverse trigger
Initialize P P( = proj,, (- (ST fo)) for n2 E lr;
t T do
Sample random vectors {M, In-i such that E Mr = 0: Initialize A (G,) = Mf" for It:
Bootstrap Sall and/or normalize (aL(S 0,);
# optional
Update P(G = proj, (P(GT) a VoL(S); 0.)) for E IT: Compute H, = Enel, 1 G7 x Ve Vo (Sval: 01) EnEr, P(G7)):
Update A((GT) = proj (Ao (Ai(G + He) for n 4;;
Update 0c+1 = 00 - 3 EnEr, A (GT);
# feed in A.(G)), n E I
end for
Update 0r+1 = 01 3 Enerr P(GT): Return: The poisoned parameters OT+1.
# feed in P(G n E Ir
3.3 Accumulative poisoning attacks in federated learning
Similar as the derivations in Eq. (6) and Eq. (7), under first-order expansion, accumulative poisoning attacks in federated learning can be formulated by the minimization problem
min VoL (Swal P.A
Σ P(G
(10)
nEIr
such that C (S A(Or)) S. (S,ai 0r) +y. Assuming that the federated learning process begins from a burn-in process, resulting in a shared model of parameters fg. and then being induced into an accumulative phase from round 0 to T 1. At round t, the accumulative phase updates the model as
0+1 = 0g - B Σ Ae(G.)).
(11)
nET
According to the formulations in Eq. (10) and Eq. (1I), the perturbation Ar can be obtained by
Σ G + Vo VoL(Sui 0.) ΣP(CR) ]. LEE
Σ Am(TT) mE
(12)
P.
nETT
where À is a trade-off hyperparameter similar as in Eq. (9), and A(Or) substituted by 0, at each round t. In Algorithm 2, provide instantiation of accumulative poisoning attacks in federated learning. The random vectors M," are used to avoid the model updates from different clients being the same (otherwise, monitor will perceive the abnormal behaviors). We assume white-box access to random seeds, while black-box cases can also be handled [4]. Different from the case of online learning, do not need to run C-steps PGD to update the poisoned trigger or accumulative batches.
Capacity of poisoners. Unlike online learning which the captured data is usually semantic, the model updates received during federated learning are just numerical matrices or tensors, thus human observer cannot easily perceive the malicious behaviors. we allow P and A, to be arbitrarily powerful, then it is trivial to break down the trained models. To make the settings more practical, we clip the poisoned gradients under fn-norm, namely, we constrain that Yn E Ir and t e [o, T - 1], 17, where has a similar role the perturbation size
there are [IP Ilp < 17 and ||,A, (G, in adversarial attacks. Empirical results under gradient clipping can be found Sec. 4.2.
Reverse trigger. When initializing the poisoned trigger, we apply simple trick of reversing the clean trigger batch. Specifically, the gradient computed on a clean trigger batch S7 at round is VaL(ST: 0,). A simple way to increase the validate loss value is reverse the model update to be -VL((ST; 0z), which convert the accumulative objective Eq. (10) as
min VoL(Sr: A.(e))
Tv. (o A(.)) - max (. V.L(Sai:AA.())) (13)

---

Table I: Classification accuracy (%) of the simulated online learning models CIFAR-10. The de- fault settings: ratio R = 100%, and the poisoned trigger P is fixed during the process of accumulative phase. We perform ablation studies different tricks used in the accumulative phase.
Method
Acc. before trigger
Acc. after trigger
A
Clean trigger
83.38
84.07
+00699
accumulative phase
80.90++0.50
76.94++0.89 76.65++0.93
-3.95++0.61
re-sampling Skal
80.69++0.34
-4.03++0.66
weight momentum
78.39-0..44
10.17
88.3310.88
Poisoned trigger
8/255
83.38
82.11
-1.27
accumulative phase
81.37=0.12
78.06:10.68
-3.3110.57
+ re-sampling Sval
80.45
78. 18.18420.84
-3.27-0..2
+ weight momentum
81.47:0.500
77.11
-4.36.10.44
+ optimizing P
81.31
76.05=00.40
-5.2630.33
+ weight momentum
80.77=1.00
74.05=
6.72100.0
+ 6 = 16/255
83.38
80.85
-2.33
accumulative phase
81.43-0.17
77.8
-3.541
+ re-sampling Spa
81.61=0.11
77.87
-3.7410.69
+ weight momentum
80.57=0.12
74.82
-5.75+1.08
+ optimizing P weight momentum
80.02++0992
11.10-1.66
-.92240777
80.17
69.08
-15.09
+ c = 0.1
83.38
80.52
-2.66
accumulative phase
81.20=00114
74.29
6.91110.17
+ re-sampling S,al
81.43-+0.41
74.73-+0.82
6.770--0.9
weight momentum
0.54
69.90++1001
-9.56-+0.77
+ optimizing P
81.16++0.57
70.13+00.8
-1..04+00.6
weight momentum
81.34
69.35
11.99
KD
GDA
GMM
Metric value
Metrie value
Metric value
0m0
302
Perturbation
Perturbation
Figure 2: Metric values w.r.t. perturbation sizes under four anomaly detection methods, where lower metric values indicate outliers. The simulated online learning trains the model on CIFAR-10.
where the later objective is easier to optimize since for a burn-in model, the directions of V V9L(ST;;) 0.) and (S 0,) are approximately aligned, due to the generalization guarantee. This trick can maintain the gradient norm unchanged, and does not exploit the validation batch for training.
Recovered offset. Let G, = {GT Jne,, be the gradient set received at round t, then if we can only modify part of these gradients, i.e., R(A,, G.) < 1, original optimal solution. Technically, assuming that we can only modify the clients in I/ C I, and modify the clients according
apply simple trick to recover the
the optimal solution of At in Eq. (12) is Aj, then
|
A( = Σ AC))T) - Σ GT, where R(Ar G.)
(14)
[4
REI:
nEh.
NEAV!
The trick shown in Eq. (14) can help us to eliminate the influence of unchanged model updates, and stabilize the update directions follow the malicious objective during the accumulative phase.
3.4 Differences between the accumulative phase and backdoor attacks
Although both the accumulative phase and backdoor attacks [22,38] can be viewed making the model be sensitive to certain trigger batches, there are critical differences between them:
7

---

Table 2: Classification accuracy (%) CIFAR-10 by setting different data poisoning ratios in online learning. The results are of fixing the poisoned trigger and optimizing it in the accumulative phase.
Ratio (%)
Method
100
90
80
70
60
50
40
30
20
10
Accumulative phase Poisoned trigger P
Before
81.64
81.49
80.03
81.02
81.06
81.57
81.60
81.90
81.35
81.43
After
74.94
74.11
74.66
76.10
77.04
78,46
78,65
79.79
79,46
79.28
A
6.67
7.38
5.37
4.92
4.02
3.11
2.95
2.11
1.89
2.15
Accumulative phase Optimizing P
Before
77.98
79.34
80.30
81.82
78.54
79.39
81.31
79.73
81.90
81.37
After
65,95
67.64
68.21
71.83
66.14
71.14
73.86
73.25
76.41
75.14
12.03
11.70
12.09
9.99
12.40
8.25
7.45
6,48
5.49
6.23
(a)
90
Test accuracy (%) after trigger
90
==001
80
A 0.002
Test accuracy (%)
70
70
=
50
gradient clip
Clip value
Clip value = 0.1
More 'sceret' before trigger
Clip
and destructtvve after trigger
30
Clip value = 0.02
0.01
20
25
30
35
40
81
83
84
85
86
Training epochs
Test accuracy (%) before trigger
Figure 3: (a) The negative effect of lowing down the convergence rate of model training, when we
apply gradient clipping to defend poisoning attacks. (b) Ablation studies on the value of À in Eq. (12).
(i) Data accessibility and trigger occasions. Backdoor attacks require accessibility on both training and test data, while our methods only need manipulate training data. Besides, backdoor attacks trigger the malicious behaviors (e.g., fooling the model on specific inputs) during inference, while in methods the malicious behaviors (e.g., breaking down the model accuracy)
triggered during training. (ii) Malicious objectives. We generally denote a trigger batch as S.n- For backdoor attacks, the malicious objective be formulated as maxg [(Smi B())), where B is the backdoor operations. In contrast, our accumulative phase optimizes minA VgL (Sval: A(0)) Vg (Smi;
Experiments
We mimic the real-time data training using the MNIST and CIFAR-10 datasets [31, 33]. The learning processes are similar to regular offline pipelines, while the main difference is that the poisoning attackers are allowed to intervene during training and have access to the model states to tune their attacking strategies dynamically. Following [49], apply ResNet18 [23] the model architecture, and employ the SGD optimizer with momentum of 0.9 and weight decay x 10 10-4. The initial learning rate is 1, and the mini-batch size is 100. The pixel values of images are scaled to be in the interval of 0 to 1.2
Burn-in phase. For all the experiments in online learning and federated learning, we pre-train the model for epochs on the clean training data of MNIST, and 40 epochs on the clean training data of CIFAR-10. The learning rate is kept as 0. and the batch normalization (BN) layers [26] are in train mode to record feature statistics.
Poisoning target. Usually, when training on real-time data streaming, there will be background process to monitor the model accuracy recall, such that the training progress would be stopped if the monitoring metrics have significant retracement. Thus, we exploit single-step drop as threatening poisoning target, namely, the accuracy drop after the model is trained by single step with poisoned behaviors (e.g., trained on a batch of poisoned data). Single-step drop can measure the destructive effect caused by poisoning strategies, before monitors can react to the observed retracement.
"Code available https::/ggttee com http:///tt....ooo/aa/aaaa///a/aa

---

Table 3: Classification accuracy (%) CIFAR-10 after the model updating by the trigger batch. The accumulative phase runs for 1,000 steps. Our methods better bypass the gradient clipping operations.
Loss scaling
No clip
fz-norm clip bound
to -orm clip bound
Method
10
0.1
10
0.1
83.32
83.32
83.39
83.68
82.96
83.32
83.32
10
65.28
70.16
83.14
83.66
65.28
68.04
82.07
Poisoned trigger
20
41.12
72.07
83.39
83.68
37.10
48.26
82.95
50
10.18
72.07
83.14
83.66
10.18
42.49
82.95
0.01
33.84
33.84
74.00
82.72
33.84
43.62
75.12
Accumulative phase
0.02
21.73
27.66
69.54
80.98
21.73
38.37
74.78
Clean trigger
0.05
12.64
25.42
63.47
78.98
12.64
35.02
70.57
0.08
11.17
21.17
61.87
76.55
11.17
21.17
64.31
Table 4: Results when using longer burn-in in phase on CIFAR-10 (i.e., running the burn-in phase for 100 epochs, compared to 40 epochs in Table 3).
Classification accuracy (%) on MNIST after the model updating by the trigger batch. The accumulative phase runs for 200 steps C epochs), with perturbation constraint c 16/255 and step size a 2/255.
Table 5:
Loss
No
-orm clip bound
Method
scaling
clip
10
0.1
Loss scaling
No
foe clip bound
1
89.34
89.34
89.91
89.99
Method
clip
10
0.1
Poisoned
10
45.29
84.45
89.91
89.99
trigger
20
16.62
84.45
89.91
89.99
98.27
98.27
98.27
98.28
50
10.24
84.45
89.91
89.99
Poisoned
10
95.49
95.49
98.12
98.28
trigger
20
84.09
89.24
98.12
98.28
0.01
80.35
80.35
80.35
83.32
50
31.93
89.24
98.12
98.28
Accu.
0.02
25,45
25.45
25.45
76.06
phase
0.05
12.03
12.03
15.53
70.00
Accu. phase
0.08
22.49
22.49
32.87 51.28
0.08
11.07
11.07
14.23
64.74
4.1
Performance in the simulated experiments of online learning
At each step of the accumulative phase in online learning, we obtain training batch from the ordered data streaming, and craft accumulative poisoning examples. The perturbations are generated by PGD [41], and restricted some feasible sets under -nomm. To keep the accumulative phase being 'secret', we will early-stop the accumulative procedure if the model accuracy becomes lower than a specified range (e.g., 80% for CIFAR-10), We evaluate the effects of the accumulative phase in online learning using clean trigger batch and poisoned trigger batches with different perturbation budgets e. We set the number of PGD iterations as 100, and the step size a 2.e/C.
Table reports the test accuracy before and after the model is updated on the trigger batch. The single-step drop caused by the vanilla poisoned trigger is not significant, while after combining with our accumulative phase, the poisoner can make much more destructive effects. As seen, we obtain prominent improvements by introducing two simple techniques for the accumulative phase, including weight momentun which lightly increases the momentum factor as 1.1 (from 0.9 by default) of the SGD optimizer in accumulative phase, and optimizing P that constantly updates adversarial poisoned trigger in the accumulative phase. Besides, we re-sample different Seal to demonstrate consistent performance in the setting named re-sampling Suat We also study the effectiveness of setting different data poisoning ratios, as summarized in Table 2. To mimic more practical settings, we also do simulated experiments on the models using group normalization (GN) [64], as detailed in Appendix I. In Fig. 4 we visualize the training data points used during the burn-in phase (clean) and used during the accumulative phase (perturbed under e = 16/ 255 constraint). As observed, the perturbations are hardly perceptible, while we provide more instances in Appendix A.2.
Anomaly detection. We evaluate the performance our methods under anomaly detection. Kernel density (KD) [15] applies a Gaussian kernel K ((21, 22) exp( -pp[-|221 22 =211/0²) to compute the similarity between two features 21 and 22. For KD, we restore 1, 000 correctly classified training features in each class and use
1 Local intrinsic dimensionality (LID) [39] applies K
nearest neighbors to approximate the dimension of local data distribution. We restore a total of 10, 000 correctly classified training data points, and set K 600. We also consider two model-based

---

Table 6: The effects of clean and poisoned trigger batch used with accumulative phase on CIFAR- 10. For example, using a poisoned trigger and ac- cumulating 200 steps leads to more destructive effect than using clean trigger for 500 steps.
Figure 4: Visualization of the training data points used in the simulated process of online leaning.
Loss scaling
Trigger batch
Accumulative steps T
50
100
200
500
Clean
85.17
83.52
76.96
58.83
0.01
Poisoned
78,86
67.00
58.12
26.40
Clean
84.12
77.69
63.02 41.71
Burn-in phase (clean)
Accumulative phase (e 16/255)
0.02
Poisoned
68.68 61.92 34.36
15.59
detection methods, involving Gaussian mixture model (GMM) [67] and Gaussian discriminative analysis (GDA) [34]. In Fig. 2, the metric value for KD is the kernel density, for LID is the negative local dimension, for GDA and GMM is the likelihood. As observed, the accumulative poisoning samples can better bypass the anomaly detection methods, compared to the samples crafted by the vanilla poisoning attacks.
4.2 Performance the simulated experiments of federated learning
In the experiments of federated learning, we simulate the model updates received from clients by computing the gradients on different mini-batches, following the setups in Konečný et al. [30], and we synchronize the statisties recorded by local BN layers. We first evaluate the effect of the accumulative phase along, by using a clean trigger batch. We apply the recovered offset trick (described in Eq. (14)) to eliminate potential influences induced by limited poisoning ratio, and perform ablation studies on the values of À in Eq. (12). As seen in Fig. (b), we run the accumulative phase for different numbers of steps under each value of X, and report the test accuracy before and after the model is updated on the trigger batch. As intuitively indicated, point towards the bottom-right corner implies more secret before trigger batch (i.e., less accuracy drop and not easy to be perceived by monitor), while more destructive after the trigger batch. We find that modest value of 0.02 performs well.
In Table 3, Table 4, and Table 5, we show that the accumulative phase can mislead the model with smaller norms of gradients, which can better bypass the clipping operations under both & and for norm cases. In contrast, previous strategies directly poisoning the data batch to degrade model accuracy would require large magnitude of gradient norm, and thus is easy to be defended by gradient clipping. On the other hand, executing gradient clipping is not for free, since will lower down the convergence of model training, shown exploiting poisoned trigger batch can further improve the computational efficiency of the accumulative phase, namely, using fewer accumulative steps to achieve similar accuracy drop.
in Fig. 3 (a). Finally, in Table 6, we show that
5 Conclusion
This paper proposes new poisoning strategy against real-time data streaming by exploiting extra accumulative phase. Technically, the accumulative phase secretly magnifies the model's 's sensitivity to trigger batch by sequentially ordered accumulative steps. Our empirical results show that accumulative poisoning attacks can cause destructive effects by a single update step, hefore a monitor can perceive and intervene. We also consider potential defense mechanisms like different anomaly detection methods and gradient clipping, where our methods can better bypass these defenses and break down the model performance. These results can inspire more real-time poisoning strategies, while also appeal to strong and efficient defenses that can be deployed in practical online systems.
Acknowledgements
This work was supported by the National Key Research and Development Program of China (Nos. 2020AAA0104304, 2017YFA0700904), NSFC Projects (Nos. 61620106010, 62061 136001, 61621 136008, 22776147, U19B2034, U19A2081, (1811161), Beijing Academy of Artificial Intelli- gence (BAAD), Tsinghua-Huawer Joint Research Program, grant from Tsinghua Institute for Guo Qiang, Tsinghua University- China Mobile Communications Group Lo.,Ltd. Joint Institute, Tiangong Institute for Intelligent Computing, and the NVIDIA NVAIL Program with GPU/DGX Acceleration.
10

---

References
[1] Hojjat Aghakhani, Dongyu Meng, Yu-Xiang Wang, Christopher Kruegel, and Giovanni Vigna. Bullseye polytope: A scalable clean-label poisoning attack with improved transferabilityy In IEEE European Symposium on Security and Privacy, 2021.
[2] Scott Alfeld, Xiaojin Zhu, and Paul Barford. Data poisoning attacks against autoregressive models. In AAAI Conference on Artificial Intelligence (AAAT), 2016.
[3] Terry Anderson. The theory and practice of online learning. Athabasca University Press, 2008.
[4] Anish Athalye, Nicholas Carlini, and David Wagner. Obfuscated gradients give a false sense of security: Circumventing defenses to adversarial examples. In International Conference on Machine Learning (ICML), 2018.
[5] Battista Biggio, Blaine Nelson, and Pavel Laskov. Poisoning attacks against support vector machines. In International Conference on Machine Learning (ICML), 2012.
[6] Eitan Borgnia, Jonas Geiping, Valeria Cherepanova, Liam Fowl, Arjun Gupta, Amin Ghiasi, Furong Huang, Micah Goldblum, and Tom Goldstein. Dp-instahide: Provably defusing poi- soning and backdoor attacks with differentially private data augmentations. arXiv preprint arXiv:2103.02077 2021.
[7] Cody Burkard and Brent Lagesse. Analysis of causative attacks against svms learning from data streams. In Proceedings of the 3rd ACM International Workshop on Security And Privacy Analytics, pages 31-36, 2017.
[8] Gal Chechik, Varun Sharma, Uri Shalit, and Samy Bengio. Large scale online learning of image similarity through ranking. Journal of Machine Learning Research (JMLR), 11(36):1109 1105- 2010.
[9] Xinyun Chen, Chang Liu,, Bo Li, Kimberly Lu, and Dawn Song. Targeted backdoor attacks on deep learning systems using data poisoning, arXiv preprint 1rXivv1712.05526 2017.
[10] Yiding Chen and Xiaojin Zhu. Optimal attack against autoregressive models by manipulating the environment. In AAAI Conference on Artificial Intelligence (AAAI), 2020.
[11] Greg Collinge, E Lupu, and Luis Munoz Gonzalez. Defending against poisoning attacks in online learning settings. In ESANN, 2019.
[12] Francesco Croce and Matthias Hein. Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks. International Conference on Machine Learning (ICML), 2020.
[13] Ambra Demontis, Marco Melis, Maura Pintor, Matthew Jagielski, Battista Biggio, Alina Oprea, Cristina Nita-Rotaru, and Fabio Roli. Why do adversarial attacks transfer? explaining transferability of evasion and poisoning attacks. In USENIX Security Symposium, pages 321-
338, 2019.
[14] Yinpeng Dong, Qi-An Fu, Xiao Yang, Tianyu Pang, Hang Su, Zihao Xiao, and Jun Zhu. Benchmarking adversarial robustness on image classification. In JEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
[15] Reuben Feinman, Ryan R Curtin, Saurabh Shintre, and Andrew B Gardner. Detecting adversarial samples from artifacts. arXiv preprint 1rX3v:/703.000 2017.
[16] Jonas Geiping, Liam Fowl, Ronny Huang, Wojciech Czaja, Gavin Taylor, Michael Moeller, and Tom Goldstein. Witches' brew: Industrial scale data poisoning via gradient matching, In International Conference on Learning Representations (ICLR), 2021.
[17] Jonas Geiping, Liam Fowl, Gowthami Somepalli, Micah Goldblum, Michael Moeller, and Tom Goldstein. What doesn't tt kill you makes you robust (ery: Adversarial training against poisons and backdoors. In ICLR Workshop Security and Safety in Machine Learning Systems, 2021.

---

[18] Ian Goodfellow, Jean Pouget-Abare Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets, In Advances in neural information processing systems (NeurIPS), pages 2672-2680, 2014.
[19] Ian Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversar- ial examples. In International Conference on Learning Representations (ICLR), 2015.
[20] Andreas Griewank. Some bounds on the complexity of gradients, jacobians, and hessians. In Complexity in numerical optimization, pages 128-162. World Scientific, 1993.
[21] Andreas Griewank and Andrea Walther. Evaluating derivatives: principles and techniques of algorithmic differentiation, volume 105. Siam, 2008.
[22] Tianyu Gu, Brendan Dolan-Gavitt, and Siddharth Garg. Badnets: Identifying vulnerabilities in the machine learning model supply chain. arXiv preprint arXiv:1708.06733, 2017.
[23] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings in deep residual networks. In European Conference on Computer Vision (ECCV), pages 630-645. Springer, 2016.
[24] Xiangnan He, Hanwang Zhang, Min-Yen Kan, and Tat-Seng Chua. Fast matrix factorization for online recommendation with implicit feedback. In International ACM SIGIR conference on Research and Development Information Retrieval (SIGIR), pages 549-558, 2016.
[25] W Ronny Huang, Jonas Geiping, Liam Fowl, Gavin Taylor, and Tom Goldstein. Metapoison: Practical general-purpose clean-label data poisoning. In Advances in Neural Information Processing Systems (NeurIPS), 2020.
[26] Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International Conference on Machine Learning (ICML), pages 448 456, 2015.
[27] Kwang-Sung Jun, Lihong Li, Yuzhe Ma, and Xiaojin Zhu. Adversarial attacks on stochastic bandits. In Annual Conference on Neural Information Processing Systems (NeurIPS), 2018.
[28] Pang Wei Koh and Percy Liang. Understanding black-box predictions via influence functions. In International Conference on Machine Learning (ICML), pages 1885-1894. PMLR, 2017.
[29] Pang Wei Koh, Jacob Steinhardt, and Percy Liang, Stronger data poisoning attacks break data sanitization defenses. arXiv preprint 18Xiv:8811.00744 2018.
[30] Jakub Konečný, H Brendan McMahan, Felix X Yu, Peter Richtárik, Ananda Theertha Suresh, and Dave Bacon. Federated learning: Strategies for improving communication efficiency. arXiv preprint arXiv: 2016.
[31] Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. Technical report, Citeseer, 2009.
[32] Ram Shankar Siva Kumar, Magnus Nyström, John Lambert, Andrew Marshall, Mario Goertzel, Andi Comissoneru, Matt Swann, and Sharon Xia. Adversarial machine sarring-indduste perspectives. In 2020 IEEE Security and Privacy Workshops (SPW). pages 69-75. IEEE, 2020.
[33] Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):2278-23) 23244, 1998.
[34] Kimin Lee, Kibok Lee, Honglak Lee, and Jinwoo Shin. simple unified framework for detecting out-of-distribution samples and adversarial attacks. In Advances in Neural Information Processing Systems (NeurIPS), 2018
[35] Laurent Lessard, Xuezhou Zhang, and Xiaojin Zhu. An optimal control approach to sequen- tial machine teaching. In International Conference on Artificial Intelligence and Statistics (AISTATS). PMLR, 2019.
12

---

[36] Bo Li, Yining Wang, Aarti Singh, and Yevgeniy Vorobeychik. Data poisoning attacks on factorization-bart collaborative filtering. In Advances in Neural Information Processing Systems (NeurIPS), 2016.
[37] Fang Liu and Ness Shroff. Data poisoning attacks on stochastic bandits. In International Conference on Muchine Leurning (ICML). PMLR, 2019.
[38] Yingqi Liu, Shiqing Ma, Yousra Aafer, Wen-Chuan Lee, Juan Zhai, Weihang Wang, and Xiangyu Zhang. Trojaning attack on neural networks. In Network and Distributed System Security Symposium (NDSS), 2018.
[39] Xingjun Ma, Bo Li, Yisen Wang, Sarah M Erfani, Sudanthi Wijewickrema, Michael E Houle, Grant Schoenebeck, Dawn Song, and James Bailey. Characterizing adversarial subspaces using local intrinsic dimensionality. In International Conference on Learning Representations (ICLR), 2018.
[40] Yuzhe Ma, Kwang-Sung Jun, Lihong Li, and Xiaojin Zhu. Data poisoning attacks in contextual bandits. In International Conference on Decision and Game Theory for Security, pages 186-204, Springer, 2018
[41] Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. In International Conference on Learning Representations (ICLR), 2018.
[42] Kelly Martin and Patrick E Murphy. The role of data privacy in marketing. Journal of the Academy of Marketing Science, 1552)1135-155, 2017.
[43] Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. communication-effiiie learning of deep networks from decentralized data. In International Conference Artificial Intelligence and Statistics (AISTATS), pages 1273-1282 PMLR, 2017.
[44] Abid Mehmood, Iynkaran Natgunanathan, Yong Xiang, Guang Hua, and Song Guo. Protection of big data privacy. IEEE access, 4:1821-1834. 2016.
[45] Shike Mei and Xiaojin Zhu, Using machine teaching to identify optimal training-set attacks on machine learners. In AAAI Conference on Artificial Intelligence (AAAI), 2015.
[46] Luis Muñoz González, GonzzzzG Battista Biggio. Ambra Demontis, Andrea Paudice, Vasin Wongras- samee, Emil C Lupu, and Fabio Roli. Towards poisoning of deep learning algorithms with back-gradient optimization. Proceedings of the 10th ACM Workshop on Artificial Intelligence and Security, pages 27-38, 2017.
[47] Tianyu Pang, Chao Du, Yinpeng Dong, and Jun Zhu. Towards robust detection of adversarial examples. Advances in Neural Information Processing Systems (NeurIPS), pages 4579-4588 2018.
[48] Tianyu Pang, Kun Xu, Chao Du, Ning Chen, and Jun Zhu. Improving adversarial robustness via promoting ensemble diversity. In International Conference on Machine Learning (ICML), 2019.
[49] Tianyu Pang, Xiao Yang, Yinpeng Dong, Hang Su, and Jun Zhu. Bag of tricks for adversarial training. In International Conference on Learning Representations (ICLR), 2021.
[50] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. In Advances in Neural Information Processing Systems (NeuriPS), pages 8024-8035, 2019.
[51] Andrea Paudice, Luis Muñoz-Gonzáler Andras Gyorgy, and Emil Lupu. Detection of adversarial training examples in poisoning attacks through anomaly detection. arXiv preprint 18X2v:/802.03044 2018.
[52] Aniruddha Saha, Akshayvarun Subramanya, and Hamed Pirsiavash. Hidden trigger backdoor attacks. In AAAI Conference on Artificial Intelligence (AAAI), 2020.
13

---

[53] Sanjay Seetharaman, Shubham Malaviya, Rosni KV, Manish Shukla, and Sachin Lodha. Influence based defense against data poisoning attacks in online learning. arXiv preprint arXiv:2104. 13230, 2021.
[54] Ali Shafahi, W Ronny Huang, Mahyar Najibi, Octavian Suciu, Christoph Studer, Tudor Du- mitras, and Tom Goldstein. Poison frogs! targeted clean-label poisoning attacks on neural networks. In Advances in Neural Information Processing Systems (NeurIPS), 2018.
[55] Reza Shokri and Vitaly Shmatikov. Privacy-preservie deep learning. In Proceedings of the 22nd ACM SIGSAC conference on computer and communications security, pages 1310-1321, 2015.
[56] Ilia Shumailov, Zakhar Shumaylov, Dmitry Kazhdan, Yiren Zhao, Nicolas Papernot, Murat Erdogdu, and Ross Anderson. Manipulating sgd with data ordering attacks. arXiv preprint arXiv:2104. 09667, 2021.
[57] Jacob Steinhardt, Pang Wei Koh, and Percy Liang. Certified defenses for data poisoning attacks. In Advances in Neural Information Processing Systems (NeurIPS), 2017.
[58] Octavian Suciu. Radu Marginean, Yigitcan Kaya, Hal Daume III, and Tudor Dumitras. When does machine learning fail? generalized transferability for evasion and poisoning attacks. In USENIX Security Symposium, pages 1299-1331 1316, 2018.
[59] Mingjie Sun, Siddhant Agarwal, and J Zico Kolter. Poisoned classifiers not only backdoored, they are fundamentally broken. arXiv preprint arXiv:2010.09088 2020.
[60] Christian Szegedy, Wojeiech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Good- fellow, and Rob Fergus. Intriguing properties of neural networks. In International Conference on Learning Representations (ICLR), 2014.
[61] Alexander Turner, Dimitris Tsipras, and Aleksander Madry. Label-consistent backdoor attacks. arXiv preprint urXiv: 2019.
[62] Jasmine Valera, Jacinto Valera, and Yvette Gelogo. A review on facial recognition for on- line learning authentication. In 2015 8th International Conference on Bio-Science and Bio- Technology (BSBT), pages 16-19. IEEE, 2015.
[63] Yizhen Wang and Kamalika Chaudhuri. Data poisoning attacks against online learning. arXiv preprint arXiv: 1808. 08994, 2018
[64] Yuxin Wu and Kaiming He. Group normalization. In Proceedings of the European conference on computer vision (ECCV), pages 3-19, 2018
[65] Huang Xiao, Battista Biggio, Gavin Brown, Giorgio Fumera, Claudia Eckert, and Fabio Roli. Is feature selection secure against training data poisoning? In International Conference on Machine Learning (ICML). pages 1689-1698, 2015.
[66] Xuezhou Zhang, Xiaojin Zhu, and Laurent Lessard. Online data poisoning attacks. In Learning for Dynamics and Control, pages 201-210. PMLR, 2020.
[67] Zhihao Zheng and Pengyu Hong. Robust detection of adversarial attacks by modeling the intrinsic properties of deep neural networks. In Advances in Neural Information Processing Systems (NeurIPS), 2018
[68] Shengyan Zhou, Jianwei Gong, Guangming Xiong, Huiyan Chen, and Karl Iagnemma. Road detection using support vector machine based on online learning and evaluation. In 2010 IEEE intelligent vehicles symposium, pages 256-261. IEBE, 2010.
[69] Chen Zhu, W Ronny Huang. Hengduo Li. Gavin Taylor, Christoph Studer, and Tom Goldstein. Transferable laben poisoning attacks on deep neural nets. In International Conference on Machine Learning (ICML). PMLR, 2019.
14


Reviews:
[{"review_id": "kWzMffosqq", "reviewer": "Reviewer_94w7", "summary": "The paper proposes a poisoning attack for online/federated learning systems with the inclusion of an accumulation phase which then makes a trigger batch of poisons even more deleterious to the model. ", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "While I have experience with poisoning literature, I will add a disclaimer that I am not an expert at poisoning for online learning/federated learning. \n\n### Strengths:\n* The idea of an abrupt degradation of accuracy of an online model seems like an interesting and novel threat-model. \n* The idea of \"accumulating\" the effects of poisons is also quite interesting. \n* The drop in accuracy of the victim model is impressive.\n\n### Weaknesses: \n* The method should be explained more clearly and motivated in a better way. Heuristically, what's the \"goal\" of the accumulation poisons? \n* It's not clear what capacity the attacker has. You mention that they have an l-infinity bound on the perturbations they are allowed to make, but are they allowed to perturb each client's update? If so this seems like a very strict assumption. \n* To this end, if they are allowed to poison every update, a comparison to adaptations of other poisoning methods like the one found in Geiping et al. (cited in your work) would be in order. \n* It would be nice if you described your online learning setup a bit further. You mention a \"burn in\" phase for the CIFAR-10 experiments involving 40 epochs of training. Could you describe how this differs from a standard supervised routine? \n\n\n### Random Comments:\nThere are several grammatical errors that should be cleaned up. \nFor example: \n* line 95: *becomes*\n* line 115: *researches*\n\nAnd a few others I forgot to mark down.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "While the authors do briefly mention limitations in the appendix, it would be nice if they included a further discussion of this in the main body. They do mention that the attacker is assumed to have access to the model's parameters in the main body, but this isn't further discussed as a realistic or unrealistic assumption. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "1"}, "extra_text": {}}, {"review_id": "YcE5so9mUKL", "reviewer": "Reviewer_Gua5", "summary": "Data poisoning attacks usually target the offline setting and do not fully utilize the sequential nature of data utilization in the online learning scenario. This work aims to optimize the damage of a trigger batch during the training phase (akin to an availability attack) by adding selective perturbations to data leading up to the trigger batch. At that point, the trigger batch can cause significantly more damage than a vanilla trigger batch inserted in the data stream. The authors derive optimization requirements for the perturbations in this 'accumulative phase' and the trigger batch, and empirically demonstrate how the proposed attack can outperform other methods, all while being more robust to heuristics like gradient-clipping.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "## Positive Feedback\n\n1. The derivation of the proposed method is laid out quite well - with functional annotations (e.g., Equation 9) and color-coding (e.g., Equations 6,7) help follow the proposed algorithm's process.\n\n2. The attack is robust enough against different hyper-parameters, like lambda values (Figure 3(b)). Performance stability is an excellent property to have and may suggest the proposed method being technically sound.\n\n3. The benefit of the accumulative phase is most evident in the presence of gradient-clip operations. Performance drops are significantly better when an accumulative phase is included, further highlighting the proposed method's advantage. Although better performance in the standard scenario is a good sign, accumulative data to bypass gradient-clipping operations more effectively is valuable.\n\n---\n\n## Criticism\n\n1. My main concern is with an adversary's power that can accumulate poisoned data, compared to one that inserts poisoned data at some given iteration. Although the former is better at utilizing the online nature of data being used by the model, it ultimately uses more poison points. Not only does this make it more likely for the attack to be detected, but it makes comparison with a simple slip-in baseline poisoning attack like an apples-to-oranges comparison. Although the proposed method objectively performs better, it is unclear how an adversary with a similar strength (perhaps insert poisoned data twice at specific intervals?) would hold against this. I understand that it is hard to quantify the adversary's strength in this case, but it is pretty clear that the adversary in the proposed work's threat model is more potent than one that only adds poison data once.\n\n2. The inclusion of only one dataset makes empirical findings less convincing - it would be great if the authors could also include results for another dataset (could be even from the same domain, but a different task). \n\n3. Assuming Whitebox access to random seeds feels like a powerful assumption for federated learning. If the adversary can access the random seeds, it can probably go a step further (in data manipulation or code control) or even adversarially pick seeds. \n\n4. This work assumes that the victim relies on using the parameters at any given iteration as the primary model. The victim could have a basic validation set to spot sudden drops in performance and sound the alarm (while keeping track of the model with the best validation loss). Even more simply, it could keep track of the two most recent parameters, and reject data from batch $i + 1$ if it leads to a performance drop, and use parameters from state $i$. It would be nice if the authors can analyze the effect of skipping the trigger batch in such a setting.\n\n---\n\n## Minor comments\n\n1. Equation 4: are the gradient updates scaled with $\\frac{1}{N}$? If not, should the gradient update not be averaged?\n\n2. Figure 2: What is the scale of values ere? For instance, how difference is 6.7 from 6.5 for metric A or 1500 v/s 1800 on Metric B. It would be great if the authors could include a  sentence or two to give the reader a better scope about how relevant values changes are for these metrics.\n\n3. Figure 4 (and similar figures with image grids) are too small to notice any difference, even after zooming in. Please consider using a 5x5 (or lower) grid of images with a higher DPI.\n\n4. This work seems quite similar to [1], in the sense of how both of them aim to exploit the sequential nature of data being fed in the training pipeline for a machine-learning model. Although this work is pretty recent, it would be worthwhile to talk about it (and perhaps in some future version add comparison experiments). \n---\n\n## References\n\n[1] Shumailov, Ilia, et al. \"Manipulating SGD with data ordering attacks.\" arXiv preprint arXiv:2104.09667 (2021).", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "### Limitations\n\n1. As mentioned above, the authors should also consider a victim that keeps track of the best model based on validation loss (or even training loss) and see how effective the proposed attack can be in such a setting.\n\n2. The lack of evaluation on any other datasets makes it hard to convince the reader of the effectiveness of the proposed attack.\n\n---\n\n### Ideas for future work\n\n1. I would like to see in a future version how performance changes if the trigger batch is rejected by the victim at the batch at which it is inserted and how its effectiveness decreases with more data being fed into the model.\n\n2. The method itself is pretty clever, but the victim can very quickly identify the trigger batch here. It would be exciting to see a variant of this method that can create a 'time-bomb' of sorts - causing a sudden drop in performance after some data batches. Hence, the victim would have a more challenging time identifying culprit data.\n\n\n## Update\n\nMost concerns raised have been addressed - changed rating accordingly ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "S0AJe8abCKj", "reviewer": "Reviewer_C5BF", "summary": "This paper proposes a poisoning attack in settings where data is acquired iteratively. Two concrete settings are examined: online learning and federated learning. Under the assumed threat model, an attacker may observe the state of the learning algorithm and poison the data it consumes in each iteration. In order to evade detection, a novel attack strategy is proposed where the learned model is primed over several iterations in a so-called _accumulative phase_ without causing harm. After the accumulative phase, the attack can be activated by a special trigger data batch to cause a sudden drop in model accuracy. The attack is formulated as an optimization problem, which is ultimately relaxed and solved using projected gradient descent. Experiments are performed using CIFAR-10 to understand the power of the attack for different parameter settings and variations of the accumulative phase. The results show that the attack is effective and can more easily bypass anomaly detection and gradient clipping defenses.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper makes a valuable contribution to the literature on data poisoning attacks, where there has been limited work in online settings. I agree with the observation that existing online attacks are easy to detect when there is a steady drop in model performance. The proposed strategy – of discretely preparing for a sudden and powerful attack – effectively exploits the online nature of the target. And the experiments show that it is surprisingly powerful.\n\nThe paper is generally well-executed. The attacks are defendable – they are derived by relaxing a formulation of the attacker’s objective as an optimization problem. However, I did find it difficult to follow the derivation in some places – e.g. seeing how eq (9) is obtained from eqs (7) and (8). I would like to see some of the missing steps/assumptions spelled out in an appendix. I appreciated the fact that online learning _and_ federated learning were both covered – effectively as two variations of the same framework.\n\nOne area in which the paper could be improved is in clarifying the threat model. I found that assumptions were mentioned in different places, making them difficult to piece together. My understanding is that a white box setting is assumed, where the attacker has full access to the model architecture, model parameters and the ability to poison all of the data received by the learning algorithm in each batch. In other words, a powerful attacker – although this is not out of the ordinary for data poisoning papers.\n\nI found the experiments to be comprehensive, especially given the limited space. An ablation study on attack variants for the accumulation phase shows significant differences in the power of the attack. However to get a complete picture, I’d like to see the computational resources required for each variant. I was pleased to see an evaluation of the attack under an anomaly detection defense, however I think it’s difficult to judge the results in the absence of a benign reference. In particular, I’d like to see how the metrics compare with the values computed on unperturbed (benign) data. \nFinally, I’d be interested to see an additional baseline to match the “vanilla” online poisoning referenced at the beginning of Section 3. It’s my understanding that the first row for each poisoned method in Table 1 is for a baseline where only a _single_ batch is poisoned. But what happens if an attacker continues to poison batches for the same number of iterations as the attacker running the accumulative attack? This would put the attackers on equal footing in terms of the horizon of the attack. The accumulative attack should be less powerful because its attempting to evade detection, but by how much?\n\nOther comments:\n- Line 108: Higher-order derivatives must apparently be computed for this attack, but how high? Are they capped at second-order?\n- There are some unusual word choices in places e.g. retracement (L217)\n- In various places, the model is said to be “irreparably broken” after an attack is launched. This sounds a little strange to me, as practical implementations of online learning would likely be able to revert to a previous checkpoint.\n- For future work: it’d be interesting to develop intuition for how the model is affected by the accumulative phase of the attack. This may help in developing a defense for this attack.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Societal impact is not discussed at length, but new types of attacks are usually disclosed so that defenses can be developed.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "5zA25mpATPq", "reviewer": "Reviewer_cbZK", "summary": "This paper presents a novel poisoning attack strategy for online learning settings, where the parameters of the machine learning model are updated as new batches of data are collected. For this, the authors propose an accumulative poisoning attack strategy with two phases. In the first phase, the attacker poisons the model in a stealthy way, so that the performance of the model is not affected, so that it magnifies the destructive effect of the poisoning attack conducted in the second phase, where the attacker injects poisoning points in a batch of samples to decrease the performance of the model. The authors shows that this attack strategy can also be applied for compromising federated learning. ", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The idea and the algorithm proposed in the paper is very interesting and, to my best knowledge, novel. I really like the idea of having a preliminary phase where the attacker, in a stealthy way, manipulates the learning algorithm without affecting the performance, so that the impact of the attack is maximized in the second phase. In this sense, I think that the differences with respect to backdoor attacks are clear and adequately justified in Section 3.4. \n\nThe presentation and justification of the method are fine. However, following some of the steps in the derivation of the attack are not straightforward (see for example the derivation of (9)) and I think that more details about the steps (in the appendix) are needed to help readers to understand and follow better the derivation of the attack strategy. With respect to the attack formulation, I miss the analysis with the computational complexity of the proposed attack strategy. \n\nMy main concerns about the paper are on the experimental evaluation: \n\n(1) The authors just considered a single dataset, CIFAR, for the experimental evaluation. In this context, CIFAR is an easier dataset to compromise, given that the dataset is quite multi-modal with low-resolution images and the accuracy of the baseline model is less than 85%, which makes it more unstable against data poisoning attacks. It would be interesting to analyse the performance of this attack strategy in other benchmarks. For example, in MNIST, where poisoning with stealthy strategies can be more difficult to achieve. \n\n(2) The assumptions about the attacker are very strong in most of the experiments. In most cases, except for Table 2, the authors assume that the attacker is in control of 100% of the training points (please, correct me if I am wrong), which is quite unrealistic in most cases and provides a misleading view of the effectiveness of the attack. In the case of federated learning, it is unclear what is the fraction of malicious clients present in the platform. From the previous experiments I assume that is 100% of the clients, which explains the results reported in Tables 3 and 4. \n\n(3) Defensive techniques are not properly analyzed. For example, Figure 2 just report the metric values (which ones?) for the different anomaly detectors to show that the data points generated during the accumulative phase are stealthier. But is the performance of the model affected in the presence of such anomaly detectors and what is their effectiveness to detect the attack points generated using the vanilla and the proposed strategy (for example by looking at the AUC, false positives/negatives or the F1 score of the detector). Similarly, it would be interesting to analyze to test the attack against robust optimization methods such as Sever (Diakonikolas et al. “Sever: A Robust Meta-Algorithm for Stochastic Optimization”) or Rubinstein et al. “Antidote: Understanding and Defending against Poisoning of Anomaly Detectors” by, for example, performing a transfer attack. Similarly, in federated learning settings, it would be interesting to test the performance of the model against robust aggregation methods such as Krum (Blanchard et al. “Machine learning with adversaries: Byzantine tolerant gradient descent”) or median-based robust aggregation techniques (Yin et al. “Byzantine-Robust Distributed Learning: Towards Optimal Statistical Rates”), just to cite some. \n\nSide comment: The pictures in Figure 4 are too small to appreciate differences. It would also be good to show some pictures of poisoning points generated with vanilla approaches to highlight the differences between attacks. \n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "No comments. ", "review_text": "", "extra_scores": {"ethical_concerns": "No ethical concerns. ", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5.5"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers were overall positive regarding the paper, and they were satisfied by the rebuttal. After the discussion period, the reviewers requested for the following things to be included in the paper:

1. More description of the optimization problem in Algorithm 1.
2. A  table about the capacity of the attacker in the federated learning setting.
3. Include the additional empirical evaluations mentioned mentioned by the authors during the rebuttal period. 
Decision: Accept (Poster)