Online Sign Identification: Minimization of the Number of Errors in Thresholding Bandits

1 Introduction and related work . In a stochastic multi-armed bandit problem , a decision maker sequentially samples from different distributions in order to optimize a loss that depends on the unknown parameters of those distributions . As a consequence , a tradeoff arises between gathering more samples from any possible distribution ( to enhance the estimation of relevant parameters ) and optimizing the allocation to minimize the final loss . We can distinguish two main categories of losses , focusing on “ exploitation ” vs “ exploration ” . The former directly depends on the whole allocation of samples and the typical example is regret minimization ( we refer to the recent monographs [ 27 , 6 , 34 ] that cover this setting almost exhaustively ) . The later is a bit different ; after the budget of samples is exhausted , the algorithms must answer one or several “ questions ” ( on the different distribution ) and its loss is related to the number of mistakes made ; the typical application being best-arm identification and variants [ 3 , 24 ] . We investigate a class of pure exploration problems , called “ thresholding bandit ” [ 28 , 35 ] . The key property of this class is that a question is asked about each distribution , and the probability of making a mistake decreases with the total information gathered on that distribution solely . The typical question the algorithm must answer is “ is the mean of the distribution above or below some threshold ? '' ( say , 0 , for simplicity ) ; giving the wrong answer can either incur a unit cost - independently from the distribution - , or a data-dependent cost ( say , the distance to the threshold that represents the “ risk ” of that distribution ) . A typical application of thresholding bandits is crowdsourcing [ 7 ] where the 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . objective is to distinguish workers with positive ( vs. negative ) efficiency ; another one is bandit binary classification [ 18 ] . Some care must be taken when designing a performance criterion for a thresholding bandit problem , since any non-stupid algorithm will eventually answer all questions correctly ( hence have a 0 loss ) if it has enough samples . Furthermore , if distributions are sub-Gaussian ( a rather mild assumption that we are going to make ) , the probability of making a single mistake decreases exponentially fast with the number of samples . As a consequence , the focus must be on controlling the exponential decay constant . We illustrate that issue on the unit cost problem described as follows . There are K different σ-sub-Gaussian distributions ; the mean of distribution k is denoted by µk and the ( variancenormalized ) gap of distribution k to the threshold 0 is denoted by ∆k : = |µk|/ √ 2σ2 . The algorithm has a budget of T samples to ( sequentially ) allocate to those distributions and , based on the Nk , T samples of distribution k , it must decide the sign of µk ; any mistake has a cost of one . We denote by Ek ∈ { 0 , 1 } an indicator of a wrong sign prediction of µk after exhausting the budget of T samples . The loss is then L1T : = ∑ k Ek . It is not difficult to see that the expected number of mistakes could be of order ∑K k=1 exp ( −Nk , T∆2k ) . In particular , sampling evenly across distributions ( Nk , T = T/K ) gives E [ L1T ] ≈ ∑ k exp ( − T K∆ 2 k ) , which has an exponential decay in T . However , this uniform allocation is far from being optimal in term of the exponential decay constant . Computing an ( approximate ) optimal fixed allocation in hindsight is not difficult : just optimize the upper-bound of E [ L1T ] . Since even the uniform allocation has a loss decaying exponentially , the performance of an algorithm should be measured not with respect to E [ L1T ] ( see [ 24 ] ) but rather in terms of − log ( E [ L1T ] ) /T . The oracle that uses knowledge of the gaps ∆k to optimize its fixed allocation verifies lim sup T→∞ 1 T log ( E [ L1T ] ) ≤ − 1∑ k 1/∆ 2 k . This unit cost framework has been investigated recently [ 35 ] with a simple yet effective algorithm called LSA ( Logarithmic-Sample Algorithm ) designed exclusively for this problem ; it samples the distribution with the smallest current index defined as αNk , t∆̂2k , t + logNk , t , where ∆̂k , t is the empirical estimate of ∆k and α is some parameter to be chosen . LSA is `` optimal up to a constant '' , but the constant is unfortunately in the exponential decay , as it was proved that1 lim sup T→∞ 1 T log ( E [ L1T ] ) ≤ − 1 16020 1∑ k 1/∆ 2 k for LSA . As we shall see , this result can be drastically improved with our more refined and general analysis ( that implies choosing a totally different input parameter α = 1 instead of 1/10 as suggested originally ) . 1.1 Contributions . We investigate the thresholding bandit problem with a weighted number of errors loss . Our contributions are twofold : 1 ) a generic method to design algorithms , with a generic proof , showing good performance on the weighted number of errors loss . 2 ) new lower-bounds and counter-intuitive results for the unit cost problem . A generic algorithm with performance guarantees We propose a Frank-Wolfe inspired method to design bandit algorithms . We develop a proof technique to obtain loss bounds for the type of algorithms that our method produces , which we apply to the thresholding bandit with losses LT = K∑ k=1 akEk or L∆T = K∑ k=1 ∆kEk , ( 1 ) where ( ak ) k∈ [ K ] are known costs . The class of algorithms we analyze includes both LSA and APT ( Anytime Parameter-free Thresholding ) [ 35 , 28 ] . We obtain precise non-asymptotic loss bounds for E [ LT ] ; for instance , we improve the original bound of LSA by a factor 4005 ( and APT by a factor 8 ) . More importantly , we get a new algorithm whose expected error for the unit cost problem is within a 1See Remark 1 [ 35 ] . This bound implies that LSA - with the specified choice of α = 0.1 needs 16000 times more samples than the oracle to achieve the same performances . factor 4 of the oracle . We emphasize again than those “ constant ” factors are in the exponential ( and are not mere multiplicative constants ) . Interestingly , this class of algorithms are not driven either by the “ optimism under uncertainty ” principle , a standard technique in multi-armed bandit [ 4 ] nor “ Explore-then-commit / Successive Elimination ” [ 31 , 12 ] . New insights on the thresholding bandit problem First , the optimal allocation provided by the oracle of [ 35 ] in the unit cost problem has a M-shape ( see Figure 1 ) because of two concurrent phenomena . On the one hand , the arms close to the threshold should not be pulled too much because their sign is difficult ( if not impossible ) to identify and it is a waste of budget . On the other hand , the signs of the arms far from the threshold are quickly well estimated and therefore should not be chosen too often either . The middle arms are the ones that need to be pulled the most frequently . As T gets larger , more and more budget is allocated to difficult arms . In section 2.2 , we provide a lower-bound that shows that this M shape is actually impossible to achieve for a sequential algorithm . Typically , the hollow inside of the M shape corresponds to arms whose sign can not be well-estimated . In particular , it is not possible to distinguish arms that are very close to the threshold from the arms that are at the top of the M and should be pulled the most frequently according to the oracle . Our second insight is corroborated by numerical simulations in Section 4 . We show empirically that our algorithms not only match but also surpass the optimal non-adaptive sampling of the oracle . We conjecture that our algorithms take advantage of the chance due to noise that can move its estimate of the arm away from the threshold . In particular , when all the gaps ∆k are equal , the non-adaptive optimal allocation should be uniform , which is significantly outperformed by adaptive algorithms . This suggests that adaptivity is crucial for this problem and may inspire future research directions to the multi-armed bandit community in order to prove theoretical guarantees for such phenomena . 1.2 Additional related work . Zero-one loss Most of the literature on thresholding bandits [ 28 , 30 , 8 ] aims at minimizing the probability of making any sign error , i.e. , minimizing the loss L∗T = I { ∃k ∈ [ K ] , Ek = 1 } = max k Ek . ( 2 ) We already mentioned the algorithm APT [ 28 ] , that gets an exponential decay of that loss ( variants include variance estimation [ 36 ] and/or delayed feedbacks ) . Other algorithms exist , but based on the optimism principle [ 23 , 30 ] . Unfortunately they suffer from a degraded exponential decay constant ( by a factor bigger than 1000 ) . Another part of the literature focuses on the fixed confidence framework , where the objective is to answer some questions with some fixed probability of mistake ( and obviously with a minimal sample budget ) . For instance , an objective could be to return any arm above some threshold as soon as possible [ 21 , 9 ] , or the one closest to the threshold [ 16 ] , or just identifying that one arm is above that threshold [ 25 ] , or even to control false discovery rates and variants [ 19 , 18 ] . Global loss , dynamic allocation and outliers detection The loss considered in thresholding bandits can be seen as a variant of a “ global loss ” ( i.e. , essentially non-linear ) that has been extensively studied in the bandit literature [ 1 , 2 , 29 ] . However , the major difference is , again , that the optimal allocation is time dependent and that the loss converges exponentially fast to zero ( no matter the algorithm ) . Similarly , Frank Wolfe algorithms have been introduced in this setting [ 5 , 13 ] ; even though our algorithms share some similarities , they are intrinsically different for the same reasons . Similarly , the problem investigated could be seen as a special case of bandit resource allocations [ 26 , 7 , 32 , 11 , 14 ] but where the loss is always decreasing with respect to the budget allocated per resource ( hence again leading to a zero loss exponentially fast ) . Finally the global objective of thresholding bandits is to obtain a synthetic view of how the means of distributions are spread on the real line ( which ones are above/below some threshold ) . In that aspect , this problem sheds some similarities with outlier detection in multi-armed bandits [ 22 , 38 , 37 ] .

Summary: This paper studies the thresholding bandit problem with the goal which can be seen as a weighted version of aggregate regret. The main contributions of this paper are two-fold:  1. It introduces an algorithm that is inspired by the Frank-Wolfe algorithm regarding this generic goal. A corresponding upper bound is derived regarding this algorithm. 2. This paper discovers new insights on the thresholding bandit problem i.e., a good algorithm must pull all arms and the “optimal” non-adaptive algorithm may be worse than adaptive algorithms.  After rebuttal -------------------- The rebuttal looks good to me and I will keep my positive score.