Online Adversarial Attacks

Adversarial attacks expose important vulnerabilities of deep learning models , yet little attention has been paid to settings where data arrives as a stream . In this paper , we formalize the online adversarial attack problem , emphasizing two key elements found in real-world use-cases : attackers must operate under partial knowledge of the target model , and the decisions made by the attacker are irrevocable since they operate on a transient data stream . We first rigorously analyze a deterministic variant of the online threat model by drawing parallels to the well-studied ksecretary problem in theoretical computer science and propose VIRTUAL+ , a simple yet practical online algorithm . Our main theoretical result show VIRTUAL+ yields provably the best competitive ratio over all single-threshold algorithms for k < 5— extending previous analysis of the k-secretary problem . We also introduce the stochastic k-secretary—effectively reducing online blackbox transfer attacks to a ksecretary problem under noise—and prove theoretical bounds on the performance of VIRTUAL+adapted to this setting . Finally , we complement our theoretical results by conducting experiments on MNIST , CIFAR-10 and Imagenet classifiers , revealing not only the necessity of online algorithms in achieving near-optimal performance but also the rich interplay of a given attack strategy towards online attack selection , enabling simple strategies like FGSM to outperform classically strong whitebox adversaries . 1 INTRODUCTION . In adversarial attacks , an attacker seeks to maliciously disrupt the performance of deep learning systems by adding small but often imperceptible noise to otherwise clean data ( Szegedy et al. , 2014 ; Goodfellow et al. , 2015 ) . Critical to the study of adversarial attacks is specifying the threat model Akhtar & Mian ( 2018 ) , which outlines the adversarial capabilities of an attacker and the level of information available in crafting attacks . Canonical examples include the whitebox threat model Madry et al . ( 2017 ) , where the attacker has complete access , and the less permissive blackbox threat model where an attacker only has partial information , like the ability to query the target model ( Chen et al. , 2017 ; Ilyas et al. , 2019 ; Papernot et al. , 2016 ) . Previously studied threat models ( e.g. , whitebox and blackbox ) implicitly assume a static setting that permits full access to instances in a target dataset at all times ( Tramèr et al. , 2018 ) . However , such an assumption is unrealistic in many real-world systems . Countless real-world applications involve streaming data that arrive in an online fashion ( e.g. , financial markets or real-time sensor networks ) . Understanding the feasibility of adversarial attacks in this online setting is an essential question . As a motivating example , consider the case where the adversary launches a man-in-the-middle attack depicted in Fig . 1 . Here , data is streamed between two endpoints—i.e. , from sensors on an autonomous car to the actual control system . An adversary , in this example , would intercept the sensor data , potentially perturb it , and then send it to the controller . Unlike classical adversarial attacks , such a scenario presents two key challenges that are representative of all online settings . 1 . Transiency : At every time step , the attacker makes an irrevocable decision on whether to attack and if she fails , or opts not to attack , then that datapoint is no longer available for further attacks . 2 . Online Attack Budget : The adversary—to remain anonymous from stateful defences—is restricted to a small selection budget and must optimally balance a passive exploration phase before selecting high value items in the data stream ( e.g . easiest to attack ) to submit an attack on . To the best of our knowledge , the only existing approaches that craft adversarial examples on streaming data ( Gong et al. , 2019a ; Lin et al. , 2017 ; Sun et al. , 2020 ) require multiple passes through a data stream and thus can not be applied in a realistic online setting where an adversary is forced into irrevocable decisions . Moreover , these approaches do not come with theoretical guarantees . Consequently , assessing the practicality of adversarial attacks— to better expose risks—in a truly online setting is still an open problem , and the focus of this paper . Main Contributions . We formalize the online threat model to study adversarial attacks on streaming data . In our online threat model , the adversary must execute k successful attacks within n streamed data points , where k n. As a starting point for our analysis , we study the deterministic online threat model in which the actual value of an input—i.e. , the likelihood of a successful attack—is revealed along with the input . Our first insight elucidates that such a threat model , modulo the attack strategy , equates to the k-secretary problem known in the field of optimal stopping theory Dynkin ( 1963 ) ; Kleinberg ( 2005 ) , allowing for the application of established online algorithms for picking optimal data points to attack . We then propose a novel online algorithm VIRTUAL+ that is both practical , simple to implement for any pair ( k , n ) , and requires no additional hyperparameters . Besides , motivated by attacking blackbox target models , we also introduce a modified secretary problem dubbed the stochastic k-secretary problem , which assumes the values an attacker observes are stochastic estimates of the actual value . We prove theoretical bounds on the competitive ratio— under mild feasibility assumptions—for VIRTUAL+in this setting . Guided by our theoretical results , we conduct a suite of experiments on both toy and standard datasets and classifiers ( i.e. , MNIST , CIFAR-10 , and Imagenet ) . Our empirical investigations reveal two counter-intuitive phenomena that are unique to the online blackbox transfer attack setting : 1 . ) In certain cases attacking robust models may in fact be easier than non-robust models based on the distribution of values observed by an online algorithm . 2 . ) Simple attackers like FGSM can seemingly achieve higher online attack transfer rates than stronger PGD-attackers when paired with a online algorithm , demonstrating the importance of carefully selecting which data points to attack . We summarize our key contributions : • We formalize the online adversarial attack threat model as an online decision problem and rigorously connect it to a generalization of the k-secretary problem . • We introduce and analyze VIRTUAL+ , an extension of VIRTUAL for the k-secretary problem yielding a significant practical improvement ( 60 % ) . We then provide , via novel techniques , a tractable formula for its competitive ratio , partially answering one of Albers & Ladewig ( 2020 ) ’ s open questions ( see footnote 1 ) and achieving a new state-of-the-art competitive ratio for k < 5 . • We propose Alg . 2 that leverages ( secretary ) online algorithms to perform efficient online adversarial attacks . We compare different online algorithms including VIRTUAL+ on MNIST , CIFAR-10 , and Imagenet in the challenging Non-Interactive BlackBox transfer ( NoBox ) setting . 2 BACKGROUND AND PRELIMINARIES . Classical Adversarial Attack Setup . We are interested in constructing adversarial examples against some fixed target classifier ft : X → Y which consumes input data points x ∈ X and labels them with a class label y ∈ Y . The goal of an adversarial attack is then to produce an adversarial example x′ ∈ X , such that ft ( x′ ) 6= y , and where the distance d ( x , x′ ) ≤ γ . Then , equipped with a loss ` used to evaluate ft , an attack is said to be optimal if ( Carlini & Wagner , 2017 ; Madry et al. , 2017 ) , x′ ∈ argmaxx′∈X ` ( ft ( x′ ) , y ) , s.t . d ( x , x′ ) ≤ γ . ( 1 ) Note that the formulation above makes no assumptions about access and resource restrictions imposed upon the adversary . Indeed , if the parameters of ft are readily available , we arrive at the familiar whitebox setting , and problem in Eq . 1 is solved by following the gradient∇xft that maximizes ` . k-Secretary Problem . The secretary problem is a well-known problem in theoretical computer science Dynkin ( 1963 ) ; Ferguson et al . ( 1989 ) . Suppose that we are tasked with hiring a secretary from a randomly ordered set of n potential candidates to select the secretary with maximum value . The secretaries are interviewed sequentially and reveal their actual value on arrival . Thus , the decision to accept or reject a secretary must be made immediately , irrevocably , and without knowledge of future candidates . While there exist many generalizations of this problem , in this work , we consider one of the most canonical generalizations known as the k-secretary problem Kleinberg ( 2005 ) . Here , instead of choosing the best secretary , we are tasked with choosing k candidates to maximize the expected sum of values . Typically , online algorithms that attempt to solve secretary problems are evaluated using the competitive ratio , which is the value of the objective achieved by an online algorithm compared to an optimal value of the objective that is achieved by an ideal “ offline algorithm , ” i.e. , an algorithm with access to the entire candidate set . Formally , an online algorithm A that selects a subset of items SA is said to be C-competitive to the optimal algorithm OPT which greedily selects a subset of items S∗ while having full knowledge of all n items , if asymptotically in n Eπ∼Sn [ V ( SA ) ] ≥ ( C + o ( 1 ) ) V ( S∗ ) , ( 2 ) where V is a set-value function that determines the sum utility of each algorithm ’ s selection , and the expectations are over permutations sampled from the symmetric group of n elements , Sn , acting on the data . In §4 , we shall further generalize the k-secretary problem to its stochastic variant where the online algorithm is no longer privy to the actual values but must instead choose under uncertainty . 3 ONLINE ADVERSARIAL ATTACKS . Motivated by our more realistic threat model , we now consider a novel adversarial attack setting where the data is no longer static but arrives in an online fashion . 3.1 ADVERSARIAL ATTACKS AS SECRETARY PROBLEMS . The defining feature of the online threat model—in addition to streaming data and the fact that we may not have access to the target model ft—is the online attack budget constraint . Choosing when to attack under a fixed budget in the online setting can be related to a secretary problem . We formalize this online adversarial attack problem in the boxed online threat model below . In the online threat model we are given a data stream D = { ( x1 , y1 ) , . . . , ( xn , yn ) } of n samples ordered by their time of arrival . In order to craft an attack against the target model ft , the adversary selects , using its online algorithm A , a subset SA ⊂ D of items to maximize : V ( SA ) : = ∑ ( x , y ) ∈SA ` ( ft ( ATT ( x ) ) , y ) s.t . |SA| ≤ k , ( 3 ) where ATT ( x ) denotes an attack on x crafted by a fixed attack method ATT that might or might not depend on ft. From now on we define x′i = ATT ( xi ) . Intuitively , the adversary chooses k instances that are the “ easiest '' to attack , i.e . samples with the highest value . Note that selecting an instance to attack does not guarantee a successful attack . Indeed , a successful attack vector may not exist if the perturbation budget γ is too small . However , stating the adversarial goal as maximizing the value of SA leads to the measurable objective of calculating the ratio of successful attacks in SA versus S∗ . If the adversary knows the true value of a datapoint then the online attack problem reduces to the original k-secretary . On the other hand , the adversary might not have access to ft and instead , the adversary ’ s value function may be an estimate of the true value—e.g . the loss of a surrogate classifier , and the adversary must make selection decisions in the face of uncertainty . The theory developed in this paper will tackle both the case where values vi : = ` ( ft ( x′i ) , yi ) for i ∈ { 1 , . . . , n } : = [ n ] are known ( §3.2 ) , as well as the richer stochastic setting with only estimates of vi , i ∈ [ n ] ( §4 ) . Practicality of the Online Threat Model . It is tempting to consider whether in practice the adversary should forego the online attack budget and instead attack every instance . However , such a strategy poses several critical problems when operating in real-world online attack scenarios . Chiefly , attacking any instance in D incurs a non-trivial risk that the adversary is detected by a defense mechanism . Indeed , when faced with stateful defense strategies ( e.g . Chen et al . ( 2020 ) ) every additional attacked instance further increases the risk of being detected and rendering future attacks impotent . Moreover , attacking every instance may be infeasible computationally for large n or impractical based on other real-world constraints . Generally speaking , as conventional adversarial attacks operate by restricting the perturbation to a fraction of the maximum possible change ( e.g. , ` ∞-attacks ) online attacks analogously restrict the time window to a fraction of possible instances to attack . Similarly , knowledge of n is also a factor that the adversary can easily control in practice . For instance , in the autonomous control system example , the adversary can choose to be active for a short interval—e.g. , when the autonomous car is a certain geospatial location—and thus set the value for n. Online Threat Model . The online threat model relies on the following key definitions : • The target model ft . The adversarial goal is to attack some target model ft : X → Y , through adversarial examples that respect a chosen distance function , d , with tolerance γ . • The data stream D. The data stream D contains the n examples ( xi , yi ) ordered by their time of arrival . At any timestep i , the adversary receives the corresponding item in D and must decide whether to execute an attack or forever forego the chance to attack this item . • Online attack budget k. The adversary is limited to a maximum of k attempts to craft attacks within the online setting thus imposing that each attack is on a unique item in D. • A value function V . Each item in the dataset is assigned a value on arrival by the value function V : X × Y → R+ which represents the utility of selecting the item to craft an attack . This can be the likelihood of a successful attack under ft ( true value ) or a stochastic estimate of the incurred loss given by a surrogate model fs ≈ ft . The online threat model corresponds to the setting where the adversary seeks to craft adversarial attacks ( i ) against a target model ft ∈ F , ( ii ) by observing items in D that arrive online , ( iii ) and choosing k optimal items to attack by relying on ( iv ) an available value function V . The adversary ’ s objective is then to use its value function towards selecting items inD that maximize the sum total value of selections V ( Eq . 3 ) .

Summary:  This paper studies the online adversarial attacks on deep learning (DL) models. Unlike other works that focused on adversarial attacks on  DL models, this work studies, on half of the attacker, how to choose a subset of data points to attack in an online fashion. Since the data arrives as a stream, the attacker has to make an irrevocable decision on whether to attack when each data point comes. This setting aligns with the situation given in the well-studied k-secretary problem. Built on top of a famous algorithm in the k secretary problem called VIRTUAL, an algorithm called VIRTUAL+ is proposed by the authors, which provably generates the best competitive ratio among all the single threshold-based algorithms (when the number of data points being attacked is less than 5). The attacker may not have access to the target deep learning model but can estimate the query value. Hence, the authors obtain theoretical bounds on the performance of the proposed VIRTUAL+ algorithm for the stochastic setting. This paper conducts experiments on both MNIST and CIFAR-10 with both vanilla and robust classifiers.