Certified Robustness of Nearest Neighbors against Data Poisoning Attacks

1 INTRODUCTION . Data poisoning attacks ( Barreno et al. , 2006 ; Nelson et al. , 2008 ; Biggio et al. , 2012 ; 2013a ; Xiao et al. , 2015b ; Steinhardt et al. , 2017 ; Shafahi et al. , 2018 ) aim to corrupt the training phase of a machine learning system via carefully poisoning its training dataset including modifying , adding , and/or removing some training examples . The corrupted model predicts incorrect labels for testing examples . Data poisoning attacks pose severe security concerns to machine learning in critical application domains such as autonomous driving ( Gu et al. , 2017 ) , cybersecurity ( Rubinstein et al. , 2009 ; Suciu et al. , 2018 ; Chen et al. , 2017 ) , and healthcare analytics ( Mozaffari-Kermani et al. , 2014 ) . Unlike adversarial examples ( Szegedy et al. , 2014 ; Goodfellow et al. , 2014 ; Carlini & Wagner , 2017 ) , which add perturbation to each testing example to induce misclassification , data poisoning attacks corrupt the model such that it misclassifies many clean testing examples . Multiple certifiably robust learning algorithms ( Ma et al. , 2019 ; Rosenfeld et al. , 2020 ; Levine & Feizi , 2020 ; Jia et al. , 2020 ) against data poisoning attacks were recently developed . A learning algorithm is certifiably robust against data poisoning attacks if it can learn a classifier on a training dataset that achieves a certified accuracy on a testing dataset when the number of poisoned training examples is no more than a threshold ( called poisoning size ) . The certified accuracy of a learning algorithm is a lower bound of the accuracy of its learnt classifier no matter how an attacker poisons the training examples with the given poisoning size . The key idea of state-of-the-art certifiably robust learning algorithms ( Levine & Feizi , 2020 ; Jia et al. , 2020 ) is to create a majority vote mechanism to predict the label of a testing example . In particular , each voter votes a label for a testing example and the final predicted label is the majority vote among multiple voters . For instance , Deep Partition Aggregation ( DPA ) ( Levine & Feizi , 2020 ) divides the training dataset into disjoint partitions and learns a base classifier ( i.e. , a voter ) on each partition . Bagging ( Jia et al. , 2020 ) also learns multiple base classifiers ( i.e. , voters ) , but each of them is learnt on a random subsample of the training dataset . We denote by a and b the labels with the largest and second largest number of votes , respectively . Moreover , sa and sb respectively are the number of votes for labels a and b when there are no corrupted voters . The corrupted voters change their votes from a to b in the worst-case scenario . Therefore , the majority vote result ( i.e. , the predicted label for a testing example ) remains to be a when the number of corrupted voters is no larger than d sa−sb2 e − 1 . In other words , the number of corrupted voters that a majority vote mechanism can tolerate depends on the gap sa − sb between the largest and the second largest number of votes . However , state-of-the-art certifiably robust learning algorithms achieve suboptimal certified accuracies due to two key limitations . First , each poisoned training example leads to multiple corrupted voters in the worse-case scenarios . In particular , modifying a training example corrupts two voters ( i.e. , two base classifiers ) in DPA ( Levine & Feizi , 2020 ) and corrupts the voters whose training subsamples include the modified training example in bagging ( Jia et al. , 2020 ) . Therefore , given the same gap sa − sb between the largest and the second largest number of votes , the majority vote result is robust against a small number of poisoned training examples . Second , they can only certify robustness for each testing example individually because it is hard to quantify how poisoned training examples corrupt the voters for different testing examples jointly . Suppose the classifier learnt by a learning algorithm can correctly classify testing inputs x1 and x2 . An attacker can poison e training examples such that the learnt classifier misclassifies x1 or x2 , but the attacker can not poison e training examples such that both x1 and x2 are misclassified . When the poisoning size is e , existing certifiably robust learning algorithms would produce a certified accuracy of 0 for the two testing examples . However , the certified accuracy can be 1/2 if we consider them jointly . k nearest neighbors ( kNN ) and radius nearest neighbors ( rNN ) ( Fix & Hodges , 1951 ; Cover & Hart , 1967 ) have intrinsic majority vote mechanisms . Specifically , given a testing example , kNN ( or rNN ) predicts its label via taking a majority vote among the labels of its k nearest neighbors ( or neighbors within radius r ) in the training dataset . Our major contribution in this work is that we show the intrinsic majority vote mechanisms in kNN and rNN make them certifiably robust against data poisoning attacks . Moreover , kNN and rNN address the limitations of state-of-the-art certifiably robust learning algorithms . Specifically , each poisoned training example leads to only one corrupted voter in the worse-case scenario in kNN and rNN . Therefore , given the same gap sa− sb , the majority vote result ( i.e. , predicted label for a testing example ) is robust against more poisoned training examples in kNN and rNN . Furthermore , we show that rNN enables joint certification of multiple testing examples . Figure 1 illustrates an example of individual certification and joint certification with two testing examples in rNN . When we treat the two testing examples individually , an attacker can poison 3 training examples such that rNN misclassifies each of them . However , when we treat them jointly , an attacker can not poison 3 training examples to misclassify both of them . We propose such joint certification to derive a better certified accuracy for rNN . Specifically , we design methods to group testing examples in a testing dataset such that we can perform joint certification for each group of testing examples . We evaluate our methods on MNIST and CIFAR10 datasets . We use ` 1 distance metric to calculate nearest neighbors . First , our methods substantially outperform state-of-the-art certifiably robust learning algorithms . For instance , when an attacker can arbitrarily poison 1,000 training examples on MNIST , the certified accuracy of rNN with r = 4 is 40.8 % and 33.5 % higher than those of DPA ( Levine & Feizi , 2020 ) and bagging ( Jia et al. , 2020 ) , respectively . Second , our joint certification improves certified accuracy . For instance , our joint certification improves the certified accuracy of rNN by 15.1 % when an attacker can arbitrarily poison 1,000 training examples on MNIST . In summary , we make the following contributions : • We derive the intrinsic certified robustness guarantees of kNN and rNN against data poisoning attacks . • We propose joint certification of multiple testing examples to derive a better certified robustness guarantee for rNN . rNN is the first method that supports joint certification of multiple testing examples against data poisoning attacks . • We evaluate our methods and compare them with state-of-the-art on MNIST and CIFAR10 . 2 PROBLEM SETUP . Learning setting : Assuming we have a training dataset Dtr with n training examples . We denote by M a learning algorithm . Moreover , we denote byM ( Dtr , x ) the label predicted for a testing input x by a classifier learnt byM on the training dataset Dtr . For instance , given a training dataset Dtr and a testing input x , kNN finds the k training examples in Dtr that are the closest to x as the nearest neighbors , while rNN finds the training examples in Dtr whose distances to x are no larger than r as the nearest neighbors . The distance between a training input and a testing input can be measured by any distance metric . Then , kNN and rNN use majority vote among the nearest neighbors to predict the label of x . Specifically , each nearest neighbor is a voter and votes its label for the testing input x ; and the label with the largest number of votes is the final predicted label for x . Data poisoning attacks : We consider data poisoning attacks ( Rubinstein et al. , 2009 ; Biggio et al. , 2012 ; Xiao et al. , 2015a ; Li et al. , 2016 ; Muñoz-González et al. , 2017 ; Jagielski et al. , 2018 ) that aim to poison ( i.e. , modify , add , and/or remove ) some carefully selected training examples in Dtr such that the corrupted classifier has a low accuracy for testing examples indiscriminately . For simplicity , we use D∗tr to denote the poisoned training dataset . Moreover , we define the poisoning size e as the minimal number of modified/added/removed training examples that can turn Dtr into D∗tr . We use S ( Dtr , e ) to denote the set of poisoned training datasets whose poisoning sizes are at most e. Formally , we define S ( Dtr , e ) as follows : S ( Dtr , e ) = { D∗tr|max { |D∗tr| , |Dtr| } − |D∗tr ∩Dtr| ≤ e } , ( 1 ) where max { |D∗tr| , |Dtr| } − |D∗tr ∩Dtr| is the poisoning size of D∗tr . Note that modifying a training example is equivalent to removing a training example and adding a new one . Given a training dataset Dtr and a poisoning size e , an attacker aims to craft a poisoned training dataset D∗tr to minimize the testing accuracy of the classifier learnt by algorithmM on D∗tr . Certified accuracy : Given a training dataset Dtr and a learning algorithmM , we use certified accuracy on a testing dataset Dte = { ( xi , yi ) } ti=1 to measure the algorithm ’ s performance . Specifically , we denote certified accuracy at poisoning size e as CA ( e ) and formally define it as follows : CA ( e ) = min D∗tr∈S ( Dtr , e ) ∑ ( xi , yi ) ∈Dte I ( M ( D ∗ tr , xi ) = yi ) |Dte| , ( 2 ) where I is the indicator function andM ( D∗tr , xi ) is the label predicted for a testing input xi by the classifier learnt by the algorithmM on the poisoned training dataset D∗tr . CA ( e ) is the least testing accuracy on Dte that the learning algorithmM can achieve no matter how an attacker poisons the training examples when the poisoning size is at most e. Our goal is to derive lower bounds of CA ( e ) for learning algorithms kNN and rNN . 3 CERTIFIED ACCURACY OF KNN AND RNN . We first derive a lower bound of the certified accuracy via individual certification , which treats testing examples in Dte individually . Then , we derive a better lower bound of the certified accuracy for rNN via joint certification , which treats testing examples jointly . 3.1 INDIVIDUAL CERTIFICATION . Given a poisoning size at most e , our idea is to certify whether the predicted label stays unchanged or not for each testing input individually . If the predicted label of a testing input x stays unchanged ( i.e. , M ( Dtr , x ) =M ( D∗tr , x ) ) and it matches with the testing input ’ s true label , then kNN or rNN certifiably correctly classifies the testing input when the poisoning size is at most e. Therefore , we can obtain a lower bound of the certified accuracy at poisoning size e as the fraction of testing inputs in Dte which kNN or rNN certifiably correctly classifies . Next , we first discuss how to certify whether the predicted label stays unchanged or not for each testing input individually . Then , we show our lower bound of the certified accuracy at poisoning size e. Certifying the predicted label of a testing input : Our goal is to certify that M ( Dtr , x ) = M ( D∗tr , x ) for a testing input x when the poisoning size is no larger than a threshold . Given a training dataset Dtr ( or a poisoned training dataset D∗tr ) and a testing input x , we use N ( Dtr , x ) ( or N ( D∗tr , x ) ) to denote the set of nearest neighbors of x in Dtr ( or D∗tr ) for kNN or rNN . We note that there may exist ties when determining the nearest neighbors for kNN , i.e. , multiple training examples may have the same distance to the testing input . Usually , kNN breaks such ties uniformly at random . However , such random ties breaking method introduces randomness , i.e. , the difference of nearest neighbors before and after poisoned training examples ( i.e. , N ( Dtr , x ) vs. N ( D∗tr , x ) ) depends on the randomness in breaking ties . Such randomness makes it challenging to certify the robustness of the predicted label against poisoned training examples . To address the challenge , we propose to define a deterministic ranking of training examples and break ties via choosing the training examples with larger ranks . Moreover , such ranking between clean training examples does not depend on poisoned ones . For instance , we can use a cryptographic hash function ( e.g. , SHA-1 ) that is very unlikely to have collisions to hash each training example based on its input feature vector and label , and then we rank the training examples based on their hash values . We use sl to denote the number of votes in N ( Dtr , x ) for label l , i.e. , the number of nearest neighbors in N ( Dtr , x ) whose labels are l. Formally , we have sl = ∑ ( xj , yj ) ∈N ( Dtr , x ) I ( yj = l ) , where l = 1 , 2 , · · · , c and I is an indicator function . kNN or rNN essentially predicts the label of the testing input x as the label with the largest number of votes , i.e. , M ( Dtr , x ) = argmaxl∈ { 1,2 , ··· , c } sl . Suppose a and b are the labels with the largest and second largest number of votes , i.e. , sa and sb are the largest and second largest ones among { s1 , s2 , · · · , sc } , respectively . We note that there may exist ties when comparing the labels based on their votes . We define a deterministic ranking of labels in { 1 , 2 , · · · , c } and take the label with the largest rank when such ties happen . In the worse-case scenario , each poisoned training example leads to one corrupted voter in kNN or rNN , which changes its vote from label a to label b . Therefore , kNN or rNN still predicts label a for the testing input x when the number of poisoned training examples is no more than d sa−sb2 e − 1 ( without considering the ties breaking ) . Formally , we have the following theorem : Theorem 1 . Assuming we have a training dataset Dtr , a testing input x , and a nearest neighbor algorithmM ( i.e. , kNN or rNN ) . a and b respectively are the two labels with the largest and second largest number of votes among the nearest neighbors N ( Dtr , x ) of x in Dtr . Moreover , sa and sb are the number of votes for a and b , respectively . Then , we have the following : M ( D∗tr , x ) = a , ∀D∗tr ∈ S ( Dtr , e ) , e ≤ d sa − sb + I ( a > b ) 2 e − 1 . ( 3 ) Proof . When an attacker poisons at most e training examples , the number of changed nearest neighbors in N ( Dtr , x ) is at most e. We denote by s∗l = ∑ ( xj , yj ) ∈N ( D∗tr , x ) I ( l = yj ) the number of votes for label l among the nearest neighbors N ( D∗tr , x ) in the poisoned training dataset , where l = 1 , 2 , · · · , c. Then , we have sl − e ≤ s∗l ≤ sl + e for each l = 1 , 2 , · · · , c. Therefore , when e ≤ d sa−sb+I ( a > b ) 2 e−1 , we have s ∗ a−s∗b ≥ sa−sb−2·e > 0 if a < b and s∗a−s∗b ≥ sa−sb−2·e ≥ 0 if a > b . Thus , the nearest neighbor algorithm still predicts label a for x in both cases based on our way of breaking ties , i.e. , we haveM ( D∗tr , x ) = a when e ≤ d sa−sb+I ( a > b ) 2 e − 1 . Deriving a lower bound of CA ( e ) : kNN or rNN certifiably correctly classifies a testing input x if it correctly predicts its label before attacks and the predicted label stays unchanged after an attacker poisons the training dataset . Therefore , the fraction of testing inputs that kNN or rNN certifiably correctly classifies is a lower bound of CA ( e ) . Formally , we have the following theorem : Theorem 2 ( Individual Certification ) . Assuming we have a training dataset Dtr , a testing dataset Dte = { ( xi , yi ) } ti=1 , and a nearest neighbor algorithmM ( i.e. , kNN or rNN ) . ai and bi respectively are the two labels with the largest and second largest number of votes among the nearest neighbors N ( Dtr , xi ) of xi in Dtr . Moreover , sai and sbi are the number of votes for ai and bi , respectively . Then , we have the following lower bound of CA ( e ) : CA ( e ) ≥ ∑ ( xi , yi ) ∈Dte I ( ai = yi ) · I ( e ≤ e ∗ i ) |Dte| , ( 4 ) where e∗i = d sai−sbi+I ( ai > bi ) 2 e − 1 . Proof . See Appendix A .

Summary: First, the paper identifies k-Nearest Neighbor (kNN) and radius Nearest Neighbor (rNN) to be naturally effective baseline certified defenses against data poisoning attack. It is easy to see that kNN and rNN are resistant to poison attacks, since to flip the prediction of a test example, one would need to insert/delete enough examples to change the majority vote. Second, the paper proposes a joint certificate that further improves certified accuracy for rNN. Specifically, it uses the fact that for any given poison removal budget, it can only decrease the vote for a single label. Even though the idea is simple, the experimental result is quite impressive significantly outperforming the previous more sophisticated certified defense methods.