Accumulative Poisoning Attacks on Real-time Data

1 Introduction . In practice , machine learning services usually collect their training data from the outside world , and automate the training processes . However , untrusted data sources leave the services vulnerable to poisoning attacks [ 5 , 28 ] , where adversaries can inject malicious training data to degrade model accuracy . To this end , early studies mainly focus on poisoning offline datasets [ 36 , 45 , 46 , 65 ] , where the poisoned training batches are fed into models in an unordered manner , due to the usage of stochastic algorithms ( e.g. , SGD ) . In this setting , poisoning operations are executed before training , and adversaries are not allowed to intervene anymore after training begins . On the other hand , recent work studies a more practical scene of poisoning real-time data streaming [ 63 , 66 ] , where the model is updated on user feedback or newly captured images . In this case , the adversaries can interact with the training process , and dynamically poison the data batches according to the model states . What ’ s more , collaborative paradigms like federated learning [ 30 ] share the model with distributed clients , which facilitates white-box accessibility to model parameters . To alleviate the threat of poisoning attacks , several defenses have been proposed , which aim to detect and filter out poisoned data via influence functions or feature statistics [ 6 , 11 , 17 , 53 , 57 ] . However , to timely update the model on real-time data streaming , on-device applications like facial recognition [ 62 ] and automatic driving [ 68 ] , large-scale services like recommendation systems [ 24 ] may use some heuristic detection strategies ( e.g. , monitoring the accuracy or recall ) to save computation [ 32 ] . In this paper , we show that in real-time data streaming , the negative effect of a poisoned or even clean data batch can be amplified by an accumulative phase , where a single update step can break down the model from 82.09 % accuracy to 27.66 % , as shown in our simulation experiments on CIFAR-10 [ 31 ] ∗Equal contribution . †Corresponding author . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . Poisoning Attacks Accumulative Poisoning Attacks ( demo in the right panel of Fig . 1 ) . Specifically , previous online poisoning attacks [ 63 ] apply a greedy strategy to lower down model accuracy at each update step , which limits the step-wise destructive effect as shown in the left panel of Fig . 1 , and a monitor can promptly intervene to stop the malicious behavior before the model is irreparably broken down . In contrast , our accumulative phase exploits the sequentially ordered property of real-time data streaming , and induces the model state to be sensitive to a specific trigger batch by a succession of model updates . By design , the accumulative phase will not affect model accuracy to bypass the heuristic detection monitoring , and later the model will be suddenly broken down by feeding the trigger batch . The operations used in the accumulative phase can be efficiently computed by applying the reverse-mode automatic differentiation [ 21 , 50 ] . This accumulative attacking strategy gives rise to a new threat for real-time systems , since the destruction happens only after a single update step before a monitor can perceive and intervene . Intuitively , the mechanism of the accumulative phase seems to be analogous to backdoor attacks [ 38 , 54 ] , while in Sec . 3.4 we discuss the critical differences between them . Empirically , we conduct experiments on MNIST and CIFAR-10 by simulating different training processes encountered in two typical real-time streaming settings , involving online learning [ 8 ] and federated learning [ 30 ] . We demonstrate the effectiveness of accumulative poisoning attacks , and provide extensive ablation studies on different implementation details and tricks . We show that accumulative poisoning attacks can more easily bypass defenses like anomaly detection and gradient clipping than vanilla poisoning attacks . While previous efforts primarily focus on protecting the privacy of personal/client data [ 42 , 44 , 55 ] , much less attention is paid to defend the integrity of the shared online or federated models . Our results advocate the necessity of embedding more robust defense mechanisms against poisoning attacks when learning from real-time data streaming . 2 Backgrounds . In this section , we will introduce three attacking strategies , and two typical paradigms of learning from real-time data streaming . For a classifier f ( x ; θ ) with model parameters θ , the training objective is denoted as L ( x , y ; θ ) , where ( x , y ) is the input-label pair . For notation compactness , we denote the empirical training objective on a data set or batch S = { xi , yi } Ni=1 as L ( S ; θ ) . 2.1 Attacking strategies . Below we briefly introduce the concepts of poisoning attacks [ 5 ] , backdoor attacks [ 9 ] , and adversarial attacks [ 19 ] . Although they may have different attacking goals , the applied techniques are similar , e.g. , solving certain optimization problems by gradient-based methods . Poisoning attacks . There is extensive prior work on poisoning attacks , especially in the offline settings against SVM [ 5 ] , logistic regression [ 45 ] , collaborative filtering [ 36 ] , feature selection [ 65 ] , and neural networks [ 13 , 28 , 29 , 46 , 56 , 58 ] . In the threat model of poisoning attacks , the attacking goal is to degrade the model performance ( e.g. , test accuracy ) , while adversaries only have access to training data . Let Strain be the clean training set and Sval be a separate validation set , a poisoner will modify Strain into a poisoned P ( Strain ) , and the malicious objective is formulated as max P L ( Sval ; θ∗ ) , and θ∗ ∈ argmin θ L ( P ( Strain ) ; θ ) , ( 1 ) where Strain and Sval are sampled from the same underlying distribution . The minimization problem is usually solved by stochastic gradient descent ( SGD ) , where feeding data is randomized . Backdoor attacks . As a variant of poisoning attacks , a backdoor attack aims to mislead the model on some specific target inputs [ 1 , 16 , 25 , 54 , 69 ] , or inject trigger patterns [ 9 , 22 , 38 , 52 , 59 , 61 ] , without affecting model performance on clean test data . Backdoor attacks have a similar formulation as poisoning attacks , except that Sval in Eq . ( 1 ) is sampled from a target distribution or with trigger patterns . Compared to the threat model of poisoning attacks , backdoor attacks assume additional accessibility in inference , where the test inputs could be specified or embedded with trigger patches . Adversarial attacks . In recent years , adversarial vulnerability has been widely studied [ 12 , 14 , 19 , 41 , 47 , 48 , 60 ] , where human imperceptible perturbations can be crafted to mislead image classifiers . Adversarial attacks usually only assume accessibility to test data . Under ` p-norm threat model , adversarial examples are crafted as x∗ ∈ argmax x′ L ( x′ , y ; θ ) , such that ‖x′ − x‖p ≤ , ( 2 ) where is the allowed perturbation size . In the adversarial literature , the constrained optimization problem in Eq . ( 2 ) is usually solved by projected gradient descent ( PGD ) [ 41 ] . 2.2 Learning on real-time data streaming . This paper considers two typical real-time learning paradigms , i.e. , online learning and federated learning , as briefly described below . Online learning . Many practical services like recommendation systems rely on online learning [ 3 , 8 , 24 ] to update or refine their models , by exploiting the collected feedback from users or data in the wild . After capturing a new training batch St at round t , the model parameters are updated as θt+1 = θt − β∇θL ( St ; θt ) , ( 3 ) where β is the learning rate of gradient descent . The optimizer can be more advanced ( e.g. , with momentum ) , while we use the basic form of gradient descent in our formulas to keep compactness . Federated learning . Recently , federated learning [ 30 , 43 , 55 ] become a popular research area , where distributed devices ( clients ) collaboratively learn a shared prediction model , and the training data is kept locally on each client for privacy . At round t , the server distributes the current model θt to a subset It of the total N clients , and obtain the model update as θt+1 = θt − β ∑ n∈It Gnt , ( 4 ) where β is the learning rate and G1t , · · · , GNt are the updates potentially returned by the N clients . Compared to online learning that captures semantic data ( e.g. , images ) , the model updates received in federated learning are non-semantic for a human monitor , and harder to execute anomaly detection . 3 Accumulative poisoning attacks . Conceptually , a vanilla online poisoning attack [ 63 ] greedily feeds the model with poisoned data , and a monitor could stop the training process after observing a gradual decline of model accuracy . In contrast , we propose accumulative poisoning attacks , where the model states are secretly ( i.e. , keeping accuracy in a reasonable range ) activated towards a trigger batch by the accumulative phase , and the model is suddenly broken down by feeding in the trigger batch , before the monitor gets conscious of the threat . During the accumulative phase , we need to calculate higher-order derivatives , while by using the reverse-mode automatic differentiation in modern libraries like PyTorch [ 50 ] , the extra computational burden is usually constrained up to 2 ∼ 5 times more compared to the forward propagation [ 20 , 21 ] , which is still efficient . This section formally demonstrates the connections between a vanilla online poisoner and the accumulative phase and provides empirical algorithms for evading the models trained by online learning and federated learning , respectively . 3.1 Poisoning attacks in real-time data streaming . Recent work takes up research on poisoning attacks in real-time data streaming against online SVM [ 7 ] , autoregressive models [ 2 , 10 ] , bandit algorithms [ 27 , 37 , 40 ] , and classification [ 35 , 63 , 66 ] . In this paper , we focus on the classification tasks under real-time data streaming , e.g. , online learning and federated learning , where the minimization process in Eq . ( 1 ) is usually substituted by a series of model update steps [ 11 ] . Assuming that the poisoned data/gradients are fed into the model at round T , then according to Eq . ( 3 ) and Eq . ( 4 ) , the real-time poisoning problems can be formulated as max P L ( Sval ; θT+1 ) , where θT+1 = { θT − β∇θL ( P ( ST ) ; θT ) , online learning ; θT − β ∑ n∈IT P ( G n T ) , federated learning , ( 5 ) where IT is the subset of clients selected at round T . The poisoning operation P acts on the data points in online learning , while acting on the gradients in federated learning . To quantify the ability of poisoning attackers , we define the poisoning ratio asR ( P , S ) = |P ( S ) \S||S| , where S could be a set of data or model updates , and |S| represents the number of elements in S. Note that in Eq . ( 5 ) , the poisoning operation P optimizes a shortsighted goal , i.e. , greedily decreasing model accuracy at each update step , which limits the destructive effect induced by every poisoned batch , and only causes a gradual descent of accuracy . Moreover , this kind of poisoning behavior is relatively easy to perceive by a heuristic monitor [ 51 ] , and set aside time for intervention before the model being irreparably broken . In contrast , we propose an accumulative poisoning strategy , which can be regarded as indirectly optimizing θT in Eq . ( 5 ) via a succession of updates , as detailed below .

Summary: This paper proposes a poisoning attack in settings where data is acquired iteratively. Two concrete settings are examined: online learning and federated learning. Under the assumed threat model, an attacker may observe the state of the learning algorithm and poison the data it consumes in each iteration. In order to evade detection, a novel attack strategy is proposed where the learned model is primed over several iterations in a so-called _accumulative phase_ without causing harm. After the accumulative phase, the attack can be activated by a special trigger data batch to cause a sudden drop in model accuracy. The attack is formulated as an optimization problem, which is ultimately relaxed and solved using projected gradient descent. Experiments are performed using CIFAR-10 to understand the power of the attack for different parameter settings and variations of the accumulative phase. The results show that the attack is effective and can more easily bypass anomaly detection and gradient clipping defenses.