Observation-Free Attacks on Stochastic Bandits

1 Introduction . Multi-armed bandit problems provide a foundational framework for understanding sequential decision making . In the classical setting , on each round of the decision process a learner selects an action ( arm ) from various alternatives and , upon making this choice , receives some scalar-valued feedback/reward for the chosen action but no additional information . Algorithms for such multi-armed bandits have been widely adopted in various applications , including recommender systems Bouneffouf et al . [ 2012 ] , Li et al . [ 2011a ] , Kawale et al . [ 2015 ] , Li et al . [ 2011b ] and in numerous modern industry and business applications Villar et al . [ 2015 ] , Schwartz et al . [ 2017 ] . A frequent model assumption for bandit problems is that the reward associated with an arm is a stochastic quantity drawn from fixed distribution associated with each arm , and that this random variable is independent of the learner ’ s previous actions . An alternative assumption , which takes a worst-case perspective and has also been widely studied , is that on every round the reward released by each arm is instead chosen by an adversary which may aim to hurt the learner ’ s learning objective . The stochastic model is often criticized for being unrealistic : data collected in a sequence rarely satisfy the IID assumption , and it would be naïve to think that corruptions never occur . The adversarial model , on the other hand , is considered highly pessimistic in contexts where we expect learning to be reasonably possible . Researchers have begun to consider intermediate model assumptions , where the input data is generally assumed to be stochastic for the most part , yet a small fraction of malicious 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . corruptions will occur . One does not have to look hard to find pertinent examples , e.g . click fraud in online advertising [ Haddadi , 2010 ] , and fake reviews in online recommendation systems [ Wilbur and Zhu , 2009 , Kshetri , 2010 , Lappas , 2012 , Lappas et al. , 2016 ] to name a few . Understanding adversarial attacks against machine learning algorithms is critical for helping to design robust systems that can be deployed in the wild . There is a long line of work on understanding adversarial data-poisoning attacks against deep learning algorithms [ Madry et al. , 2017 , Akhtar and Mian , 2018 , Yuan et al. , 2019 ] , supervised learning algorithms [ Dai et al. , 2018 , Liu et al. , 2019 ] , and more recently for multi-armed bandit problems . Perhaps the most popular algorithm for the stochastic multi-armed bandit setting , UCB [ Auer et al. , 2002 ] , has a tight theoretical guarantee on its performance ( i.e . its regret ) . Despite all this , it has been shown indeed that UCB is highly vulnerable to data corruption attacks [ Jun et al. , 2018 , Liu and Shroff , 2019 ] . In short , with only a handful of corruptions on the reward feedback given to the learning , UCB can be tricked into directing most of its choices onto a sub-optimal arm . Adversarial corruptions for multi arm bandit strategies have been studied across two axes : one line of work focus on designing and analysing different techniques to attack existing bandit algorithms [ Jun et al. , 2018 , Liu and Shroff , 2019 , Ma et al. , 2018 , Garcelon et al. , 2020 ] , while the other focuses on designing robust algorithms that can perform well under various levels of data corruption [ Lykouris et al. , 2018 , Gupta et al. , 2019 , Kapoor et al. , 2019 ] . Notwithstanding these prior lines of work , there remains a major gap in the corruption models considered for such adversarial attacks on bandit algorithms . Most existing results assume that the adversary ( corruption agent ) is given full knowledge of the arm chosen by the learner and can perform a targeted corruption on just the reward selected by the algorithm . It has indeed been shown that all no-regret stochastic bandit algorithms are vulnerable to such powerful adversaries [ Liu and Shroff , 2019 ] . On the other hand , the development of robust algorithms ( e.g . Lykouris et al . [ 2018 ] , Gupta et al . [ 2019 ] ) have obtained guarantees only under a weaker adversary , one that can only corrupt the reward feedback before observing the arm selected by the learner . There has been no work , to our knowledge , that has tried to design adversarial attacks against popular stochastic bandit algorithms under the weaker adversary . For algorithms that are deterministic , which select each arm via a non-random function of prior observations , there is no relevant distinction between the strong and weak adversarial models . But given that randomization is a common and important tool in algorithm design , in this work we consider attacks against both randomized and non-randomized algorithms . With this in mind , the goal of the present paper is to design a method of adversarial attack which ( a ) is effective against a very broad range of multi-armed bandit algorithms and ( b ) fits within the weaker adversary model . More specifically , we show that if a stochastic bandit algorithm makes its decisions as a function of a natural statistic , the empirical mean reward and the number of pulls of each arm , then such an algorithms is fully vulnerable to the corruption attacks . This family of bandit algorithms is indeed quite broad , and we show that most of the popular classical strategies—UCB , -greedy , and Thompson sampling [ Agrawal and Goyal , 2012 ] , all of which we analyze—fall within this framework and are thus similarly vulnerable . We further show that using by corrupting only o ( T ) rounds , our attack can force these algorithms to select a specific arm preferred by the adversary ( target arm ) for all but o ( T ) rounds . We believe this reveals what is a core flaw inherent in many bandit algorithms , and these insights can thus help to design more robust learning algorithms in this and other settings . Compared to the most related works of Jun et al . [ 2018 ] , Liu and Shroff [ 2019 ] , Garcelon et al . [ 2020 ] which also study adversarial attacks against bandit algorithms , there are three fundamental differences . The first difference is that this line of work assumes that the adversary can observe the actions of the bandit algorithms . This allows the adversary to attack the algorithms based on whether a particular arm is selected or not . Without such ability , to simulate their attack , the adversary need to corrupt all rounds if the bandit algorithm is randomized . The second difference is the corruption model . In their model , the corruption is counted only for the arm which is selected , while in our model , if in a round an arm is corrupted but not selected by the bandit algorithm , we still count it as a corrupted rounds . Based on our notion of corruption , the corruption budget is T for the attackers who need to corruption every round even if most of its corruption is not observed by the algorithm . The third difference is that the attacks in this line of works never apply corruptions on the target arm . Although this makes the target arm more preferred by the bandit algorithm , the negative side effect is that the amount of corruption they can apply on the non target arms is limited because they are picked less often , and thus they may have to keep attacking the algorithm . In our attack , the adversary corrupts all arms at the beginning , making all arms look similar , thus even non-target arms are picked often enough in the early phase of the attack . This allows the adversary to apply enough corruptions on the non-target arms so that the estimates can not recover even after the attack stops . Through this attack , we show that all mean based algorithms which make decisions only based on estimates of empirical means are vulnerable to adversarial data corruption attacks . Liu and Shroff [ 2019 ] provide a similar conclusion for the offline setting by analysing a few specific algorithms . Also note that in the offline setting considered in Liu and Shroff [ 2019 ] , the algorithm receives a batch of data with size T at once , and goal of the adversary is to manipulate the algorithm ’ s choice at the T + 1 round , which is very different to the online setting . 2 Preliminaries . Let ’ s begin by recalling the stochastic multi arm bandit setting . A principal ( or learner ) faces a sequential decision making problem where it needs to select one out of K actions or arms at each of the T rounds . The principal gets a reward in each round based on the arm chosen in that round . Formally , at each round t , the environment generates a reward vector rt = ( rt1 , . . . , r t K ) ( not observable to the principal ) where rti ∈ [ 0 , 1 ] is the reward the principal will receive if arm i is picked , and for each arm i , rti sampled from a fixed arm dependant distribution with mean µi which is unknown to the principal . Let µ = ( µ1 , . . . , µK ) be the mean reward vector that includes mean rewards of all arms . The principal then selects an arm It and receives the corresponding reward rtIt and does not observe the rest of the values in r t. To characterize the performance of a bandit algorithm , the notion of regret is introduced . The regret of a bandit algorithm is defined as the gap between the total expected reward of the algorithm and the expected reward of the algorithm that always selects the arm with the highest mean reward in each round . Definition 1 ( Regret ) . R ( T ) = T ·max i µi − T∑ t=1 µIt where It is the arm chosen by the algorithm in round t. Let arm i∗ be the optimal arm , i.e . i∗ = argmaxi µi . Next we introduce the notion of adversarial attacks in the stochastic bandit setting . The adversarial attack is a form of data corruption where a malicious agent intends to manipulate the behavior of the bandit algorithm by corrupting the reward vector rt generated by the environment . Specifically , the adversary can change the reward vector rt to another corrupted reward vector r̂t = ( r̂t1 , . . . , r̂ t K ) such that r̂ t i ∈ [ 0 , 1 ] for all i . We say that the round t is corrupted if the adversary changes the reward for at least one of the arms , i.e . ‖rt − r̂t‖1 > 0 . Let C be the total number of rounds that the adversary corrupts , that is C = ∑T t=1 1 { ‖rt − r̂t‖1 > 0 } . We call C the corruption level of the bandit algorithm . Importantly , We assume that the adversary corrupts the reward without observing the arm selected by the adversary . Formally , the protocol between the learner and the adversary at each round t = 1 , . . . , T is as follows : 1 . The learner decides a distribution πt ∈ ∆K over K arms . 2 . The environment generates a stochastic reward rt . 3 . The adversary corrupts the reward , and the corrupted reward becomes r̂t 4 . The learner picks an arm It from the distribution πt and receives corrupted reward r̂tIt Next we give definitions to measure the robustness of an algorithm against adversarial data corruption attacks and the power of attack methods . To characterize the performance of an algorithm under any possible adversarial attack , we introduce the definition of vulnerable algorithms . Definition 2 ( Vulnerable bandit algorithms ) . We say a bandit algorithm is vulnerable if there exists an instance and an adversary such that the adversary with C = o ( T ) corruption level can induce linear regret R ( T ) = Ω ( T ) on the bandit algorithm in expectation . To characterize the performance of an adversarial attack , we need to consider the bandit algorithm it attempts to attack as well . The adversarial attacks that we consider in this work have a goal which is one step harder than just making the bandit algorithm obtain linear regret . The adversary has a favorite arm ( that we call the target arm ) and the adversary ’ s goal is ensure that the bandit algorithm selects the target arm for most of the rounds of the algorithm . We say a bandit algorithm B is completely vulnerable to an adversarial attack A , if with probability at least 1 − δ ( T ) , with δ ( T ) = o ( 1 ) , the adversary can make the algorithm pick the target arm specified by the adversary for all but o ( T ) rounds by using only C = o ( T ) corruption level . We now introduce a framework that is typically employed by a large class of traditional stochastic multi arm bandit algorithms . Since the goal of the bandit algorithm is to incur low regret , to do so , it needs to figure out which arms lead to high expected rewards and then it also needs to ensure that it selects the arm with highest expected reward in most rounds . This leads to an exploration vs exploitation trade-off in the goals of the algorithm . In most cases , bandit algorithms rely on two statistics of each arm to balance the trade-off between explore and exploit : the empirical estimates on mean rewards and the corresponding variance on the estimates . The empirical means indicate which arm is likely to be the optimal , and the variances indicate how much confidence the algorithm has about its estimates . The variance of the estimate can be characterized by the number of samples the algorithm has access to for estimating the empirical means . The number of samples for each arm is exactly equal to the number of times that arm is selected by the learner in the stochastic setting . So typically , a wide class of stochastic multi arm bandit algorithms make decisions based on the empirical mean and number of selections for each arm . We call this class of algorithms as Mean based algorithms . Before introducing the formal definition , let us characterize the information the bandit algorithm has access to when making decisions in a round t. Let It denote the information the algorithm has access to while making decisions in round t. Using the information It , the algorithm generates a probability distribution πt over the arms where for each arm i , πt ( i|It ) is the probability that the arm i is selected in the current round t when the information available is It . Since in each round t , the algorithm chooses an arm It and then obtains the corresponding reward rtIt , the information obtained by the algorithm in round t is ( I t , rtIt ) . Thus before making a decision in round t , the algorithm has access to all the information received in the rounds so far . Let us denote Ht = { ( I1 , r1I1 ) , . . . , ( It−1 , r t−1 It−1 ) } as the history up till round t and it is exactly the information that the bandit algorithm has access to when making the decision in this round , i.e . It = Ht . Thus for the bandit algorithm , the decisions made in round t can be characterized by πt ( i|It ) = πt ( i|Ht ) . Let nt−1i = ∑t−1 τ=1 1 { Iτ = i } denote the number of rounds arm i gets picked by the algorithm before round t , and let µ̄t−1 = ∑t−1 τ=1 r τ i 1 { I τ=i } nt−1i be the empirical mean of the arm i by round t. We can define Mean based algorithms as follows . Definition 3 ( Mean based algorithms ) . We say an algorithm is a mean based algorithm if

Summary: This paper studies adversarial attacks against no-regret stochastic bandit algorithms, including UCB, epsilon-greedy and the Thompson sampling algorithms. The attacker has the ability to perturb the reward function, but such perturbation must happen before observing the action selected by the bandit player. The paper developed a three-phase attack strategy, where in the first phase, the attacker sets the reward to 0 for every arm. As a result, every arm can be selected often enough and the confidence intervals of all arms are small, while the empirical mean are all 0 as well. Then in the second phase, the attacker boosts the reward for the target arm to the maximum, and the authors show that during the second phase, the target arm will always be selected. Finally, in the third phase, no attack happens, and due to the increase of reward for the target arm in phase two, the authors proved that in the third phase, the target arm still maintains its superiority and the other arms cannot recover. Therefore, the target arm will be pulled in almost every round with high probability. The empirical results in this paper demonstrated the effectiveness of the proposed attack algorithm.