Online Sign Identification: Minimization of the Number of Errors in Thresholding Bandits

1 Introduction and related work . In a stochastic multi-armed bandit problem , a decision maker sequentially samples from different distributions in order to optimize a loss that depends on the unknown parameters of those distributions . As a consequence , a tradeoff arises between gathering more samples from any possible distribution ( to enhance the estimation of relevant parameters ) and optimizing the allocation to minimize the final loss . We can distinguish two main categories of losses , focusing on “ exploitation ” vs “ exploration ” . The former directly depends on the whole allocation of samples and the typical example is regret minimization ( we refer to the recent monographs [ 27 , 6 , 34 ] that cover this setting almost exhaustively ) . The later is a bit different ; after the budget of samples is exhausted , the algorithms must answer one or several “ questions ” ( on the different distribution ) and its loss is related to the number of mistakes made ; the typical application being best-arm identification and variants [ 3 , 24 ] . We investigate a class of pure exploration problems , called “ thresholding bandit ” [ 28 , 35 ] . The key property of this class is that a question is asked about each distribution , and the probability of making a mistake decreases with the total information gathered on that distribution solely . The typical question the algorithm must answer is “ is the mean of the distribution above or below some threshold ? '' ( say , 0 , for simplicity ) ; giving the wrong answer can either incur a unit cost - independently from the distribution - , or a data-dependent cost ( say , the distance to the threshold that represents the “ risk ” of that distribution ) . A typical application of thresholding bandits is crowdsourcing [ 7 ] where the 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . objective is to distinguish workers with positive ( vs. negative ) efficiency ; another one is bandit binary classification [ 18 ] . Some care must be taken when designing a performance criterion for a thresholding bandit problem , since any non-stupid algorithm will eventually answer all questions correctly ( hence have a 0 loss ) if it has enough samples . Furthermore , if distributions are sub-Gaussian ( a rather mild assumption that we are going to make ) , the probability of making a single mistake decreases exponentially fast with the number of samples . As a consequence , the focus must be on controlling the exponential decay constant . We illustrate that issue on the unit cost problem described as follows . There are K different σ-sub-Gaussian distributions ; the mean of distribution k is denoted by µk and the ( variancenormalized ) gap of distribution k to the threshold 0 is denoted by ∆k : = |µk|/ √ 2σ2 . The algorithm has a budget of T samples to ( sequentially ) allocate to those distributions and , based on the Nk , T samples of distribution k , it must decide the sign of µk ; any mistake has a cost of one . We denote by Ek ∈ { 0 , 1 } an indicator of a wrong sign prediction of µk after exhausting the budget of T samples . The loss is then L1T : = ∑ k Ek . It is not difficult to see that the expected number of mistakes could be of order ∑K k=1 exp ( −Nk , T∆2k ) . In particular , sampling evenly across distributions ( Nk , T = T/K ) gives E [ L1T ] ≈ ∑ k exp ( − T K∆ 2 k ) , which has an exponential decay in T . However , this uniform allocation is far from being optimal in term of the exponential decay constant . Computing an ( approximate ) optimal fixed allocation in hindsight is not difficult : just optimize the upper-bound of E [ L1T ] . Since even the uniform allocation has a loss decaying exponentially , the performance of an algorithm should be measured not with respect to E [ L1T ] ( see [ 24 ] ) but rather in terms of − log ( E [ L1T ] ) /T . The oracle that uses knowledge of the gaps ∆k to optimize its fixed allocation verifies lim sup T→∞ 1 T log ( E [ L1T ] ) ≤ − 1∑ k 1/∆ 2 k . This unit cost framework has been investigated recently [ 35 ] with a simple yet effective algorithm called LSA ( Logarithmic-Sample Algorithm ) designed exclusively for this problem ; it samples the distribution with the smallest current index defined as αNk , t∆̂2k , t + logNk , t , where ∆̂k , t is the empirical estimate of ∆k and α is some parameter to be chosen . LSA is `` optimal up to a constant '' , but the constant is unfortunately in the exponential decay , as it was proved that1 lim sup T→∞ 1 T log ( E [ L1T ] ) ≤ − 1 16020 1∑ k 1/∆ 2 k for LSA . As we shall see , this result can be drastically improved with our more refined and general analysis ( that implies choosing a totally different input parameter α = 1 instead of 1/10 as suggested originally ) . 1.1 Contributions . We investigate the thresholding bandit problem with a weighted number of errors loss . Our contributions are twofold : 1 ) a generic method to design algorithms , with a generic proof , showing good performance on the weighted number of errors loss . 2 ) new lower-bounds and counter-intuitive results for the unit cost problem . A generic algorithm with performance guarantees We propose a Frank-Wolfe inspired method to design bandit algorithms . We develop a proof technique to obtain loss bounds for the type of algorithms that our method produces , which we apply to the thresholding bandit with losses LT = K∑ k=1 akEk or L∆T = K∑ k=1 ∆kEk , ( 1 ) where ( ak ) k∈ [ K ] are known costs . The class of algorithms we analyze includes both LSA and APT ( Anytime Parameter-free Thresholding ) [ 35 , 28 ] . We obtain precise non-asymptotic loss bounds for E [ LT ] ; for instance , we improve the original bound of LSA by a factor 4005 ( and APT by a factor 8 ) . More importantly , we get a new algorithm whose expected error for the unit cost problem is within a 1See Remark 1 [ 35 ] . This bound implies that LSA - with the specified choice of α = 0.1 needs 16000 times more samples than the oracle to achieve the same performances . factor 4 of the oracle . We emphasize again than those “ constant ” factors are in the exponential ( and are not mere multiplicative constants ) . Interestingly , this class of algorithms are not driven either by the “ optimism under uncertainty ” principle , a standard technique in multi-armed bandit [ 4 ] nor “ Explore-then-commit / Successive Elimination ” [ 31 , 12 ] . New insights on the thresholding bandit problem First , the optimal allocation provided by the oracle of [ 35 ] in the unit cost problem has a M-shape ( see Figure 1 ) because of two concurrent phenomena . On the one hand , the arms close to the threshold should not be pulled too much because their sign is difficult ( if not impossible ) to identify and it is a waste of budget . On the other hand , the signs of the arms far from the threshold are quickly well estimated and therefore should not be chosen too often either . The middle arms are the ones that need to be pulled the most frequently . As T gets larger , more and more budget is allocated to difficult arms . In section 2.2 , we provide a lower-bound that shows that this M shape is actually impossible to achieve for a sequential algorithm . Typically , the hollow inside of the M shape corresponds to arms whose sign can not be well-estimated . In particular , it is not possible to distinguish arms that are very close to the threshold from the arms that are at the top of the M and should be pulled the most frequently according to the oracle . Our second insight is corroborated by numerical simulations in Section 4 . We show empirically that our algorithms not only match but also surpass the optimal non-adaptive sampling of the oracle . We conjecture that our algorithms take advantage of the chance due to noise that can move its estimate of the arm away from the threshold . In particular , when all the gaps ∆k are equal , the non-adaptive optimal allocation should be uniform , which is significantly outperformed by adaptive algorithms . This suggests that adaptivity is crucial for this problem and may inspire future research directions to the multi-armed bandit community in order to prove theoretical guarantees for such phenomena . 1.2 Additional related work . Zero-one loss Most of the literature on thresholding bandits [ 28 , 30 , 8 ] aims at minimizing the probability of making any sign error , i.e. , minimizing the loss L∗T = I { ∃k ∈ [ K ] , Ek = 1 } = max k Ek . ( 2 ) We already mentioned the algorithm APT [ 28 ] , that gets an exponential decay of that loss ( variants include variance estimation [ 36 ] and/or delayed feedbacks ) . Other algorithms exist , but based on the optimism principle [ 23 , 30 ] . Unfortunately they suffer from a degraded exponential decay constant ( by a factor bigger than 1000 ) . Another part of the literature focuses on the fixed confidence framework , where the objective is to answer some questions with some fixed probability of mistake ( and obviously with a minimal sample budget ) . For instance , an objective could be to return any arm above some threshold as soon as possible [ 21 , 9 ] , or the one closest to the threshold [ 16 ] , or just identifying that one arm is above that threshold [ 25 ] , or even to control false discovery rates and variants [ 19 , 18 ] . Global loss , dynamic allocation and outliers detection The loss considered in thresholding bandits can be seen as a variant of a “ global loss ” ( i.e. , essentially non-linear ) that has been extensively studied in the bandit literature [ 1 , 2 , 29 ] . However , the major difference is , again , that the optimal allocation is time dependent and that the loss converges exponentially fast to zero ( no matter the algorithm ) . Similarly , Frank Wolfe algorithms have been introduced in this setting [ 5 , 13 ] ; even though our algorithms share some similarities , they are intrinsically different for the same reasons . Similarly , the problem investigated could be seen as a special case of bandit resource allocations [ 26 , 7 , 32 , 11 , 14 ] but where the loss is always decreasing with respect to the budget allocated per resource ( hence again leading to a zero loss exponentially fast ) . Finally the global objective of thresholding bandits is to obtain a synthetic view of how the means of distributions are spread on the real line ( which ones are above/below some threshold ) . In that aspect , this problem sheds some similarities with outlier detection in multi-armed bandits [ 22 , 38 , 37 ] .

Summary: This paper works on the problem of fixed-budget sign identification problem in stochastic bandits. Given a set of arms where each sample on an arm returns a noisy result according to this arm's unknown latent distribution, a threshold, and the limited time horizon $T$, the learner needs to adaptively sample the arms and wants to minimize the error probability of determining whether the mean rewards of the arms are above or below the threshold.   The authors did a thorough and careful analysis on the performance of a large class of algorithms as well as the lower bound of this problem. By the analysis, the authors managed to get a gap 4 in the exponential part of the upper bound and the lower bound of the error probability, outperforming the existing results.   The results in this paper help the researchers to understand the problem of online fixed-budget sign identification problem, and the new bounds are promising both in theory and in practice.