A Separation Result Between Data-oblivious and Data-aware Poisoning Attacks

1 Introduction . Traditional approaches to supervised machine learning focus on a benign setting where honestly sampled training data is given to a learner . However , the broad use of these learning algorithms in safety-critical applications makes them targets for sophisticated attackers . Consequently , machine learning has gone through a revolution of studying the same problem , but this time under so-called adversarial settings . Researchers have investigated several types of attacks , including testtime ( a.k.a. , evasion attacks to find adversarial examples ) [ 62 , 6 , 32 , 55 ] , training-time attacks ( a.k.a. , poisoning or causative attacks ) [ 3 , 8 , 51 ] , backdoor attacks [ 67 , 33 ] , membership inference attacks [ 57 ] , etc . In response , other works have put forth several defenses [ 52 , 43 , 9 ] followed by adaptive attacks [ 15 , 2 , 65 ] that circumvent some of the proposed defenses . Thus , developing approaches that are based on solid theoretical foundations ( that prevent further adaptive attacks ) has stood out as an important area of investigation . Poisoning Attacks . In a poisoning attack , an adversary changes a training set S of examples into a “ close ” training set S ′ ( The difference is usually measured by Hamming distance ; i.e. , the number of examples injected and/or removed. ) . Through these changes , the goal of the adversary , generally speaking , is to degrade the “ quality ” of the learned model , where quality here could be interpreted in different ways . In a recent industrial survey [ 39 ] , poisoning attacks were identified as the most important threat model against applications of machine learning . The main reason behind the importance of poisoning attacks are the feasibility of performing the attack for adversary . As the 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . data is usually gathered from multiple sources , the adversary can perform the poisoning attacks by corrupting one of the sources . Hence , it is extremely important to fundamentally understand this threat model . In particular , we need to investigate the role of design choices that are made in both poisoning attacks and defenses . Does the attacker know the training data ? The role of knowledge of the clean training set is one of the less investigated aspects of poisoning attacks . Many previous work on theoretical analysis of poisoning attacks implicitly , or explicitly , assume that the adversary has full knowledge of the training data S before choosing what examples to add or delete from S [ 38 , 58 , 44 , 61 ] . In several natural scenarios , an adversary might not have access to the training data before deciding on how to tamper with it . This has led researchers to study poisoning attacks that do not use the knowledge of the training set to craft the poison points . In this work , we explore the following question : What is the role of the knowledge of training set in the success of poisoning adversaries ? Can the knowledge of training set help the attacks ? Or alternatively , can hiding the training set from adversaries help the defenses ? 1 In this work , as a first step to understand this question , we show a separation result between dataoblivious and data-aware poisoning adversaries . In particular , we show that there exist a learning setting ( Feature selection with LASSO on Gaussian data ) where poisoning adversaries that know the distribution of data but are oblivious to specific training samples that are used to train the model are provably weaker than the adversaries with the knowledge of both training set and the distribution . To the best of our knowledge , this is the first separation result for poisoning attacks . Implications of our separation result : Here , we mention some implications of our separation result . • Separation of threat models : The first implication of our result is the separation of dataoblivious and data-aware poisoning threat models . Our result shows that data-oblivious attacks are strictly weaker than data-aware attacks . In other words , it shows that we can not expect the defenses to have the same effectiveness in both scenarios . This makes the knowledge of data a very important design choice that should be clearly stated when designing defenses or attacks . • Possibility of designing new defenses : Although data-oblivious poisoning is a weaker attack model , it might still be the right threat model for many applications . For instance , if data providers use cryptographically secure multi-party protocols to train the model [ 68 ] , then each participant can only observe their own data . Note that each party might still have access to some data pool from the true distribution of training set and that still fits in our data-oblivious threat model . In these scenarios , it is natural to use defenses that are only secure against data-oblivious attacks . Our results shows the possibility of designing defense mechanisms that leverage the secrecy of training data and can provide much stronger security guarantees in this threat mode . In particular , our result shows the provable robustness of LASSO algorithm in defending against data-oblivious attacks . Note that this approach is distinct from the demoted notion of “ security through obscurity ” as the attacker knows every detail of the algorithm as well as the data distribution . The only unknown to the adversary is the randomness involved in the process of sampling training examples from the training distribution . This is exactly similar to how secret randomness helps security in cryptography . • A new motive for privacy : privacy is often viewed as a utility for data owners in the machine learning pipeline . Due to the trade-offs between privacy and the efficiency/utility , data-users often ignore the privacy of data owners while doing their analysis , especially when there is no incentive to enforce the privacy of the learning protocol . The possibility of improving the security against poisoning attacks by enforcing the ( partial ) dataobliviousness of the adversary could create a new incentive for keeping training datasets secret . Specifically , the users of data would now have more motivation to try to keep training dataset private , with the goal of securing their models against poisoning and increasing their utility in scenarios where part of data is coming from potentially malicious sources . 1This question was independently asked as an open question in the survey of Goldblum et al . [ 31 ] . 1.1 Our Contributions . In this work , we provide theoretical evidence that obliviousness of attackers to the training data can indeed help robustness against poisoning attacks . In particular , we provide a provable difference between : ( i ) an adversary that is aware of the training data as well as the distribution of training data , before launching the attack ( data-aware adversary ) and ( ii ) an adversary that only knows the distribution of training data and does not know the specific clean examples in the training set ( dataoblivious adversary ) . We start by formalizing what it means mathematically for the poisoning adversary to be dataoblivious or data-aware . Separations for feature selection with Lasso . We then prove a separation theorem between the data-aware and data-oblivious poisoning threat models in the context of feature selection . We study data-aware and data-oblivious attackers against the Lasso estimator and show that if certain natural properties holds for the distribution of dataset , the power of optimal data-aware and data-oblivious poisoning adversaries differ significantly . We emphasize that in our data-oblivious setting , the adversary fully knows the data distribution , and hence it implicitly has access to a lot of auxiliary information about the data set , yet the very fact that it does not know the actual sampled dataset makes it harder for adversary to achieve its goal . Experiments . To further investigate the power of data-oblivious and data-aware attacks in the context of feature selection , we experiment on synthetic datasets sampled from Gaussian distributions , as suggested in our theoretical results . Our experiments confirm our theoretical findings by showing that the power of data-oblivious and poisoning attacks differ significantly . Furthermore , we experimentally evaluate the power of partially-aware attackers who only know part of the data . These experiments show the gradual improvement of the attack as the knowledge of data grows . In our experimental studies we go beyond Gaussian setting and show that the the power of dataoblivious attacks could be significantly lower on real world distributions as well . In our experiments , sometimes ( depending on the noise nature of the dataset ) , even an attacker that knows 20 % of the dataset can not have much of improvement over an oblivious attacker . Separation for classification . In addition to our main results in the context of feature selection , in this work , we also take initial steps to study the role of adversary ’ s knowledge ( about the data set ) when the goal of the attacker is to increase the risk of the produced model in the context of classification . These results are presented supplemental material ( Section A ) . 1.2 Related Work . Here , we provide a short version of related prior work . A more comprehensive description of previous work has been provided in Appendix B where we also categorize the existing attacks into data-aware and data-oblivious categories . Beatson et al . [ 4 ] study “ Blind ” attackers against machine learning models that do not even know the distribution of the data . They show that poisoning attacks could be successful in such a restricted setting by studying the minimax risk of learners . They also introduced “ informed ” attacks that see the data distribution , but not the actual training samples and leave the study of these attacks to future work . Interestingly , the “ informed ” setting of [ 4 ] is equivalent to the “ oblivious ” setting in our work . Xiao et al . [ 71 ] empirically examine the robustness of feature selection in the context of poisoning attacks , but their measure of stability is across sets of features . We are distinct in that our paper studies the effect of data-oblivious attacks on individual features and with provable guarantees . We distinguish our work with another line of work that studies the computational complexity of the attacker [ 46 , 29 ] . Here , we study the “ information complexity ” of the attack ; namely , what information the attacker needs to succeed in a poisoning attack , while those works study the computational resources that a poisoning attacker needs to successfully degrade the quality of the learned model . Another recent exciting line of work that studies the computational aspect of robust learning in poisoning contexts , focuses on the computational complexity of the learning process itself [ 18 , 40 , 16 , 20 , 21 , 19 , 53 , 22 ] , and other works have studied the same question about the complexity of the learning process for evasion attacks [ 11 , 10 , 17 ] . Furthermore , our work deals with information complexity and is distinct from works that study the impact of the training set ( e.g. , using clean labels ) on the success of poisoning [ 55 , 73 , 59 , 67 ] . Our work ’ s motivation for data secrecy might seem similar to other works that leverage privacypreserving learning ( and in particular differential privacy [ 23 , 26 , 25 ] ) to limit the power of poisoning attacks by making the learning process less sensitive to poison data [ 42 ] . However , despite seeming similarity , what we pursue here is fundamentally different . In this work , we try to understand the effect of keeping the data secret from adversaries . Whereas the robustness guarantees that come from differential privacy has nothing to do with secrecy and hold even if the adversary gets to see the full training set ( or even select the whole training set in an adversarial way. ) . We also point out some separation results in the context of adversarial examples . The work of Bubeck et al . [ 12 ] studies the separation in the power of computationally bounded v.s . computationally unbounded learning algorithms in learning robust model . Tsipras et al . [ 66 ] studies the separation between benign accuracy and robust accuracy of classifiers showing that they can be even at odds with each other . Schmidt et al . [ 54 ] show the separation between sample complexity of learning algorithms in training an adversarially robust model versus a model with high benign accuracy . Garg et al . [ 29 ] separate the notions of computationally bounded v.s . computationally unbounded attacks in successfully generating adversarial examples . Although all these results are only proven for few ( perhaps unrealistic ) settings , they still significantly helped the understanding of adversarial examples . As opposed to the data poisoning setting , the question of adversary ’ s ( adaptive ) knowledge was indeed previously studied in the line of work on adversarial examples [ 41 , 49 , 62 ] . In a test time evasion attack the adversary ’ s goal is to find an adversarial example , the adversary knows the input x entirely before trying to find a close input x′ that is misclassified . So , this adaptivity aspect already differentiates adversarial examples from random noise .

Summary: This paper studies the (information-theoretic) power of an adversary in poisoning attacks. The authors exhibit a learning model (feature selection with Lasso) in which an adversary that has knowledge of the training set is provably stronger than an adversary that has no knowledge of the training set. (In both cases the adversary also has access to the distribution of the data.) The authors complement their theoretical findings with experiments on both synthetic and real data.  To show their theoretical result, the authors define the notion of unstable and resilient distributions under addition of poisoned data to the original dataset. They show that Gaussian distributions with suitable parameters are both resilient and unstable. This means (acc. to Prop 3.5) that data-aware adversaries can profit from the instability of the distribution to create poisoned data that cause Lasso to incorrectly select certain features with high probability, while the distribution's resiliency ensures that data-oblivious adversaries have to create attacks at random.