Mean-based Best Arm Identification in Stochastic Bandits under Reward Contamination

1 Introduction . Overview . This paper investigates the problem of best arm identification ( BAI ) in stochastic multiarmed bandits ( MABs ) , under the key assumption that the reward samples are vulnerable to adversarial corruptions . Specifically , consider a K-armed stochastic MAB , where the reward from arm i ∈ [ K ] , { 1 , · · · , K } is drawn from a sub-Gaussian distribution Pi . At each round t , the observed reward may be corrupted with probability ε . When a reward sample is corrupted at time t , it is replaced by a sample drawn from a contamination model with distribution Qi ( t ) , which is distinct from Pi . Such a setup is a non-trivial generalization of the canonical BAI problem due to the arbitrary corruption that may be introduced by the corruption model . This setup has far superior practical implications . For instance , consider the example of measuring drug responses , in which several compounds are evaluated in order to determine which one of them renders the maximal efficacy . It is natural to assume that a fraction of the test results are reported incorrectly , or a fraction of the samples may be contaminated [ 1 ] . Both possibilities affect the measurements ( or rewards ) that are used to identify the most effective drug . In a different example , consider a recommendation system in which the goal is to recommend the most “ interesting ” articles to users based on their feedback on previous recommendations . In this case , likewise , user feedbacks are prone to be imprecise or even malicious in a fraction of the recommendations . Such examples motivate investigating the BAI problem under the additional consideration that there exists the possibility that a fraction of the reward samples are 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . corrupted and they obscure the process of identifying the true best arm . The problem of contaminated best arm identification ( CBAI ) has been studied under two main settings , namely the fixed-budget setting and the fixed-confidence setting . The objective in the fixed-budget setting is to identify the best arm within a given sampling budget such that the misclassification rate is minimized [ 2 ] . On the other hand , that of the fixed-confidence setting is to obtain a prescribed confidence with the fewest samples possible , on average [ 3 ] . In this paper , we investigate CBAI in the second setting , in which we are prescribed a fixed guarantee on the misclassification probability and the goal is to identify the arm with the highest true mean , while , in parallel , minimizing the sample complexity . The main contributions of this paper are the following . • This paper is the first to analyze the CBAI problem for sub-Gaussian bandits in the fixed confidence setting , without altering the canonical definition of the best arm which is defined as the arm with the highest mean reward . The existing literature on CBAI aims to identify arms with the largest median [ 4 ] , which is distinct from the definition of the best arm in the canonical BAI problem . • We provide two algorithms to address the CBAI problem , one of them being a procedure based on successive elimination , and the other being a confidence gap-based procedure . Both algorithms use the trimmed mean as an estimator . We provide closed-form decision rules for dynamically selecting the arms over time , estimating the means corresponding to every arm of the bandit instance , stopping the arm-selection process , and finally identifying the true best arm . • We analyze the attendant performance guarantees of the proposed algorithms . Specifically , we establish the decision reliability and the sample complexity . While the algorithm based on successive elimination is shown to be optimal up to logarithmic factors , the one based on confidence gap is shown to be asymptotically optimal up to constant factors . Both algorithms can identify the best arm even when a fraction of the reward samples are corrupted . • Finally , we conduct numerical simulations to demonstrate the efficacy of the proposed algorithms . The experiments show that both algorithms outperform the existing methods for CBAI in both synthetic as well as real-world datasets ( content recommendation and drug testing ) . Related Literature . We review the existing literature in two parts – first , the literature on BAI in the fixed-confidence setting , and next , the literature on contaminated stochastic MABs . The problem of BAI for stochastic MABs has a rich literature , dating back to the study in [ 5 ] . Specifically , the existing BAI algorithms can be broadly grouped into two categories : ( i ) algorithms that are guided by confidence bounds , and ( ii ) those that involve successively distilling the search space . The former class of algorithms involve constructing confidence sets that capture the deviation of the empirical mean of an arm from its true mean . These algorithms terminate and recommend a predicted best arm once the accumulated data is sufficient for forming a decision that meets a prescribed confidence level . Some of the representative studies in this class include [ 6 ] , [ 7 ] , [ 8 ] , and [ 9 ] . The other category of algorithms , also known as racing algorithms , involve successively identifying and rejecting the suboptimal arms until only one arm is left in the active arms set . One of the first of these algorithms was proposed in [ 10 ] . Other examples of racing algorithms for stochastic MABs include the studies in [ 8 ] , [ 11 ] , [ 12 ] , and [ 13 ] . The literature on contaminated stochastic MABs includes analyzing adversarial corruptions in stochastic MABs for regret minimization , first studied in [ 14 ] . In this study , the adversarial power is characterized by the total number of corrupted samples C that can be introduced by the adversary until the specified time horizon . The algorithm proposed by this study degrades linearly with C , which is the optimal rate for any algorithm that achieves the optimal performance in the stochastic setting . The regret bound obtained in this study is further improved in [ 15 ] . More recent studies in this direction include [ 16 , 17 , 18 , 19 , 20 ] . The problem , however , is less-investigated in the context of BAI . In the fixed-budget setting , the problem of CBAI is studied in [ 21 ] . This study proposes a probabilistic sequential algorithm , which allows for trading off the suboptimality gap ( in the fully stochastic regime ) with the success probability using a tunable parameter . In the fixed-confidence setting , CBAI is studied in [ 4 ] , which modifies the definition of the “ best arm ” by redefining it to be the arm that has the largest median . A general adversarial model is assumed in this study , where any reward sample can be contaminated by the adversary with probability ε . Furthermore , three different attack models are considered , namely , the oblivious adversary , the prescient adversary , and the malicious adversary , depending on the level of adaptivity that the adversary employs to obfuscate the true reward samples . While the investigation proposes instance-adaptive algorithms matching up to logarithmic factors , in many models , the median and the mean could be considerably different , especially for the case of heavy-tailed distributions . Furthermore , the study considers a considerably restrictive class of statistical models ( Definition 2 , [ 4 ] ) , which does not include several popular and widely-analyzed classes of bandit instances ( e.g. , heavy-tailed continuous models and all discrete models such as the class of Bernoulli bandits ) . In this paper , we investigate these questions under the conventional definition of the best arm , where the best arm is defined as the arm having the largest mean . Furthermore , we investigate the entire class of sub-Gaussian bandits . 2 Contaminated best arm identification . Setting . Consider the canonical bandit model ν ∈ SGK ( σ ) , where SGK ( σ ) denotes the class of K-armed σ-sub-Gaussian bandits . The reward of arm i ∈ [ K ] is generated from a probability measure Pi with mean µi ∈ R. The means { µi : i ∈ [ K ] } are assumed to be unknown . At each time t , a learner interacts with the bandit instance by pulling one of the arms At ∈ [ K ] according to a control policy , generating the random reward XAt , t ∈ R. Based on the observed reward , a decision is made about the next arm to be pulled . Additionally , the setting assumes an adversary that is capable of contaminating the reward before it is observed by the learner . Specifically , it is assumed that at each time t , the adversary flips a coin , whose outcome is denoted by the random variable Dt , where Dt ∼ Bern ( ε ) . Depending on the outcome of the coin toss , the adversary sends the true reward sample , or an adversarial sample drawn from a corruption model . Thus , with a fixed probability ε ∈ ( 0 , 1 ) , the adversary replaces the true reward with a corrupt sample drawn from an adversarial distribution Qi ( t ) distinct from Pi . We consider an oblivious adversary which is responsible for the contamination of the reward samples . Let us define Xi , t as the reward generated by arm i ∈ [ K ] at time t according to the true distribution Pi . The adversarial samples for each arm i ∈ [ K ] at each time instant t is denoted by X ′i , t . An oblivious adversary is defined as follows . Definition 2.1 ( Oblivious adversary ) . An adversary is said to be oblivious if for all i ∈ [ K ] , the sequence of triples ( Xi , t , X ′i , t , Dt ) t≥1 are assumed to be independent . Hence , at each time t , the adversarial action can be characterized by a Bernoulli random variable Dt ∼ Bern ( ε ) , based on which the observed reward takes the form RAt = { XAt , t ∼ PAt , if Dt = 0 X ′At , t ∼ QAt ( t ) , if Dt = 1 . ( 1 ) This contamination model was first proposed in [ 22 ] in the canonical estimation framework under a single control action . Thus , the contaminated model corresponding to each arm of the bandit instance is characterized by the mixture distribution P̃i ( t ) = ( 1− ε ) Pi + εQi ( t ) . ( 2 ) The learner ’ s sequence of actions and rewards are denoted by At , [ A1 , · · · , At ] and Rt , [ RA1 , · · · , RAt ] . ( 3 ) Partially Identifiable Best Arm Identification ( PIBAI ) . Based on the sequence of rewards accumulated over time , the goal of the learner is to identify the best arm with a high probability . Let us denote the index of the best and second-best arms of the bandit instance by a ? and b ? , respectively . Due to the action of the adversary , it is impossible to identify the true mean of any arm i ∈ [ K ] , even with an infinite number of samples from that arm [ 4 ] . Hence , we consider the notion of partially identifiable best arm identification ( PIBAI ) . Let us consider arm i ∈ [ K ] of the bandit instance ν . Under the PIBAI setting , we can only estimate the mean of an arm up to a constant uncertainty interval Ui around the true mean , i.e. , the interval Ii , [ µi − Ui , µi + Ui ] . Subsequently , in the context of PIBAI , the goal of identifying the best arm can be presented using the following guarantee on the identification performance . Definition 2.2 ( δ-PAC ) . In the context of PIBAI , for a given set of uncertainties { Ui : i ∈ [ K ] } , we say that a procedure is δ-PAC if it satisfies the following guarantee . P ( µâτ + Uâτ < µa ? − Ua ? ) ≤ δ , ( 4 ) where τ denotes the stopping time of the procedure . We note that CBAI is a special case of the PIBAI framework . Specifically , for a fixed level of contamination ε , the bias in estimation is the same for each arm i ∈ [ K ] , that is , Ui = U for every i ∈ [ K ] . Here , U is a function of the level of contamination ε and the variance σ of the arms [ 23 ] . Key assumptions . Now , we formalize the key technical assumptions that are necessary for the algorithms proposed in Section 3 . The first assumption ensures the feasibility of being able to identify the true best arm . Essentially , there always exists adversarial models that could render the goal of CBAI infeasible , even if we have an infinite number of samples . We assume that the algorithms operate in a regime in which it is possible to identify the true best arm . The next assumption is a standard assumption in the robust statistics literature , and it provides an upper bound on the maximum level of corruption that the adversary can inflict up on the rewards . The assumptions are formally stated below . ( i ) The index of the best arm does not change as a result of contamination , i.e. , ( µa ? − Ua ? ) > ( µa + Ua ) for every a ∈ [ K ] \ a ? . ( ii ) We do not require to know the precise value of the level of contamination ε . Rather , it is sufficient to assume that we know an upper bound on it , such that at each time t , the fraction of contaminated samples falls below the upper bound . Henceforth , for the rest of the paper , ε denotes an upper bound on the probability of adversarial replacement of rewards . We focus on the regime ε < 1/2 .

Summary: The authors consider the problem of best arm identification in contaminated stochastic bandits. More precisely, in this framework, the reward probability distribution of each arm may be arbitrarily corrupted with probability $\varepsilon$. The authors focus on the fixed budget setting and propose two different algorithms with theoretical guarantees. Both algorithms use an $\alpha$-trimmed mean estimator to tackle the corruption problem.  The first algorithm (see Algorithm 1) is a gap-based algorithm that relies on the overlap between the confidence intervals of the best empirical arms and the remaining arms in its sampling, stopping, and decision rules. The second algorithm is an elimination-based algorithm, that eliminates arms as soon as they are deemed suboptimal. The authors complement their theoretical findings with experimental results.