Mean-based Best Arm Identification in Stochastic Bandits under Reward Contamination

1 Introduction . Overview . This paper investigates the problem of best arm identification ( BAI ) in stochastic multiarmed bandits ( MABs ) , under the key assumption that the reward samples are vulnerable to adversarial corruptions . Specifically , consider a K-armed stochastic MAB , where the reward from arm i ∈ [ K ] , { 1 , · · · , K } is drawn from a sub-Gaussian distribution Pi . At each round t , the observed reward may be corrupted with probability ε . When a reward sample is corrupted at time t , it is replaced by a sample drawn from a contamination model with distribution Qi ( t ) , which is distinct from Pi . Such a setup is a non-trivial generalization of the canonical BAI problem due to the arbitrary corruption that may be introduced by the corruption model . This setup has far superior practical implications . For instance , consider the example of measuring drug responses , in which several compounds are evaluated in order to determine which one of them renders the maximal efficacy . It is natural to assume that a fraction of the test results are reported incorrectly , or a fraction of the samples may be contaminated [ 1 ] . Both possibilities affect the measurements ( or rewards ) that are used to identify the most effective drug . In a different example , consider a recommendation system in which the goal is to recommend the most “ interesting ” articles to users based on their feedback on previous recommendations . In this case , likewise , user feedbacks are prone to be imprecise or even malicious in a fraction of the recommendations . Such examples motivate investigating the BAI problem under the additional consideration that there exists the possibility that a fraction of the reward samples are 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . corrupted and they obscure the process of identifying the true best arm . The problem of contaminated best arm identification ( CBAI ) has been studied under two main settings , namely the fixed-budget setting and the fixed-confidence setting . The objective in the fixed-budget setting is to identify the best arm within a given sampling budget such that the misclassification rate is minimized [ 2 ] . On the other hand , that of the fixed-confidence setting is to obtain a prescribed confidence with the fewest samples possible , on average [ 3 ] . In this paper , we investigate CBAI in the second setting , in which we are prescribed a fixed guarantee on the misclassification probability and the goal is to identify the arm with the highest true mean , while , in parallel , minimizing the sample complexity . The main contributions of this paper are the following . • This paper is the first to analyze the CBAI problem for sub-Gaussian bandits in the fixed confidence setting , without altering the canonical definition of the best arm which is defined as the arm with the highest mean reward . The existing literature on CBAI aims to identify arms with the largest median [ 4 ] , which is distinct from the definition of the best arm in the canonical BAI problem . • We provide two algorithms to address the CBAI problem , one of them being a procedure based on successive elimination , and the other being a confidence gap-based procedure . Both algorithms use the trimmed mean as an estimator . We provide closed-form decision rules for dynamically selecting the arms over time , estimating the means corresponding to every arm of the bandit instance , stopping the arm-selection process , and finally identifying the true best arm . • We analyze the attendant performance guarantees of the proposed algorithms . Specifically , we establish the decision reliability and the sample complexity . While the algorithm based on successive elimination is shown to be optimal up to logarithmic factors , the one based on confidence gap is shown to be asymptotically optimal up to constant factors . Both algorithms can identify the best arm even when a fraction of the reward samples are corrupted . • Finally , we conduct numerical simulations to demonstrate the efficacy of the proposed algorithms . The experiments show that both algorithms outperform the existing methods for CBAI in both synthetic as well as real-world datasets ( content recommendation and drug testing ) . Related Literature . We review the existing literature in two parts – first , the literature on BAI in the fixed-confidence setting , and next , the literature on contaminated stochastic MABs . The problem of BAI for stochastic MABs has a rich literature , dating back to the study in [ 5 ] . Specifically , the existing BAI algorithms can be broadly grouped into two categories : ( i ) algorithms that are guided by confidence bounds , and ( ii ) those that involve successively distilling the search space . The former class of algorithms involve constructing confidence sets that capture the deviation of the empirical mean of an arm from its true mean . These algorithms terminate and recommend a predicted best arm once the accumulated data is sufficient for forming a decision that meets a prescribed confidence level . Some of the representative studies in this class include [ 6 ] , [ 7 ] , [ 8 ] , and [ 9 ] . The other category of algorithms , also known as racing algorithms , involve successively identifying and rejecting the suboptimal arms until only one arm is left in the active arms set . One of the first of these algorithms was proposed in [ 10 ] . Other examples of racing algorithms for stochastic MABs include the studies in [ 8 ] , [ 11 ] , [ 12 ] , and [ 13 ] . The literature on contaminated stochastic MABs includes analyzing adversarial corruptions in stochastic MABs for regret minimization , first studied in [ 14 ] . In this study , the adversarial power is characterized by the total number of corrupted samples C that can be introduced by the adversary until the specified time horizon . The algorithm proposed by this study degrades linearly with C , which is the optimal rate for any algorithm that achieves the optimal performance in the stochastic setting . The regret bound obtained in this study is further improved in [ 15 ] . More recent studies in this direction include [ 16 , 17 , 18 , 19 , 20 ] . The problem , however , is less-investigated in the context of BAI . In the fixed-budget setting , the problem of CBAI is studied in [ 21 ] . This study proposes a probabilistic sequential algorithm , which allows for trading off the suboptimality gap ( in the fully stochastic regime ) with the success probability using a tunable parameter . In the fixed-confidence setting , CBAI is studied in [ 4 ] , which modifies the definition of the “ best arm ” by redefining it to be the arm that has the largest median . A general adversarial model is assumed in this study , where any reward sample can be contaminated by the adversary with probability ε . Furthermore , three different attack models are considered , namely , the oblivious adversary , the prescient adversary , and the malicious adversary , depending on the level of adaptivity that the adversary employs to obfuscate the true reward samples . While the investigation proposes instance-adaptive algorithms matching up to logarithmic factors , in many models , the median and the mean could be considerably different , especially for the case of heavy-tailed distributions . Furthermore , the study considers a considerably restrictive class of statistical models ( Definition 2 , [ 4 ] ) , which does not include several popular and widely-analyzed classes of bandit instances ( e.g. , heavy-tailed continuous models and all discrete models such as the class of Bernoulli bandits ) . In this paper , we investigate these questions under the conventional definition of the best arm , where the best arm is defined as the arm having the largest mean . Furthermore , we investigate the entire class of sub-Gaussian bandits . 2 Contaminated best arm identification . Setting . Consider the canonical bandit model ν ∈ SGK ( σ ) , where SGK ( σ ) denotes the class of K-armed σ-sub-Gaussian bandits . The reward of arm i ∈ [ K ] is generated from a probability measure Pi with mean µi ∈ R. The means { µi : i ∈ [ K ] } are assumed to be unknown . At each time t , a learner interacts with the bandit instance by pulling one of the arms At ∈ [ K ] according to a control policy , generating the random reward XAt , t ∈ R. Based on the observed reward , a decision is made about the next arm to be pulled . Additionally , the setting assumes an adversary that is capable of contaminating the reward before it is observed by the learner . Specifically , it is assumed that at each time t , the adversary flips a coin , whose outcome is denoted by the random variable Dt , where Dt ∼ Bern ( ε ) . Depending on the outcome of the coin toss , the adversary sends the true reward sample , or an adversarial sample drawn from a corruption model . Thus , with a fixed probability ε ∈ ( 0 , 1 ) , the adversary replaces the true reward with a corrupt sample drawn from an adversarial distribution Qi ( t ) distinct from Pi . We consider an oblivious adversary which is responsible for the contamination of the reward samples . Let us define Xi , t as the reward generated by arm i ∈ [ K ] at time t according to the true distribution Pi . The adversarial samples for each arm i ∈ [ K ] at each time instant t is denoted by X ′i , t . An oblivious adversary is defined as follows . Definition 2.1 ( Oblivious adversary ) . An adversary is said to be oblivious if for all i ∈ [ K ] , the sequence of triples ( Xi , t , X ′i , t , Dt ) t≥1 are assumed to be independent . Hence , at each time t , the adversarial action can be characterized by a Bernoulli random variable Dt ∼ Bern ( ε ) , based on which the observed reward takes the form RAt = { XAt , t ∼ PAt , if Dt = 0 X ′At , t ∼ QAt ( t ) , if Dt = 1 . ( 1 ) This contamination model was first proposed in [ 22 ] in the canonical estimation framework under a single control action . Thus , the contaminated model corresponding to each arm of the bandit instance is characterized by the mixture distribution P̃i ( t ) = ( 1− ε ) Pi + εQi ( t ) . ( 2 ) The learner ’ s sequence of actions and rewards are denoted by At , [ A1 , · · · , At ] and Rt , [ RA1 , · · · , RAt ] . ( 3 ) Partially Identifiable Best Arm Identification ( PIBAI ) . Based on the sequence of rewards accumulated over time , the goal of the learner is to identify the best arm with a high probability . Let us denote the index of the best and second-best arms of the bandit instance by a ? and b ? , respectively . Due to the action of the adversary , it is impossible to identify the true mean of any arm i ∈ [ K ] , even with an infinite number of samples from that arm [ 4 ] . Hence , we consider the notion of partially identifiable best arm identification ( PIBAI ) . Let us consider arm i ∈ [ K ] of the bandit instance ν . Under the PIBAI setting , we can only estimate the mean of an arm up to a constant uncertainty interval Ui around the true mean , i.e. , the interval Ii , [ µi − Ui , µi + Ui ] . Subsequently , in the context of PIBAI , the goal of identifying the best arm can be presented using the following guarantee on the identification performance . Definition 2.2 ( δ-PAC ) . In the context of PIBAI , for a given set of uncertainties { Ui : i ∈ [ K ] } , we say that a procedure is δ-PAC if it satisfies the following guarantee . P ( µâτ + Uâτ < µa ? − Ua ? ) ≤ δ , ( 4 ) where τ denotes the stopping time of the procedure . We note that CBAI is a special case of the PIBAI framework . Specifically , for a fixed level of contamination ε , the bias in estimation is the same for each arm i ∈ [ K ] , that is , Ui = U for every i ∈ [ K ] . Here , U is a function of the level of contamination ε and the variance σ of the arms [ 23 ] . Key assumptions . Now , we formalize the key technical assumptions that are necessary for the algorithms proposed in Section 3 . The first assumption ensures the feasibility of being able to identify the true best arm . Essentially , there always exists adversarial models that could render the goal of CBAI infeasible , even if we have an infinite number of samples . We assume that the algorithms operate in a regime in which it is possible to identify the true best arm . The next assumption is a standard assumption in the robust statistics literature , and it provides an upper bound on the maximum level of corruption that the adversary can inflict up on the rewards . The assumptions are formally stated below . ( i ) The index of the best arm does not change as a result of contamination , i.e. , ( µa ? − Ua ? ) > ( µa + Ua ) for every a ∈ [ K ] \ a ? . ( ii ) We do not require to know the precise value of the level of contamination ε . Rather , it is sufficient to assume that we know an upper bound on it , such that at each time t , the fraction of contaminated samples falls below the upper bound . Henceforth , for the rest of the paper , ε denotes an upper bound on the probability of adversarial replacement of rewards . We focus on the regime ε < 1/2 .

Summary: This work studies the BAI problem with the fixed-confidence setting in the contaminated stochastic bandits, where the rewards can be corrupted with probability $\varepsilon$. It proposes two algorithms and derives upper bounds on their sample complexity. It also derives a lower bound to show the tightness of the bounds. Moreover, it shows numerical results in the end.