Selective Sampling for Online Best-arm Identification

1 Introduction . In this work we consider selective sampling for online best-arm identification . In this setting , at every time step t = 1 , 2 , . . . , Nature reveals a potential measurement xt ∈ X ⊂ Rd to the learner . The learner can choose to either query xt ( ξt = 1 ) or abstain ( ξt = 0 ) and immediately move on to the next time . If the learner chooses to take a query ( ξt = 1 ) , then Nature reveals a noisy linear measurement of an unknown θ∗ ∈ Rd , i.e . yt = 〈xt , θ∗〉+ t where t is mean zero sub-Gaussian noise . Before the start of the game , the learner has knowledge of a set Z ⊂ Rd . The objective of the learner is to identify z∗ : = arg maxz∈Z〈z , θ∗〉 with probability at least 1− δ at a learner specified stopping time U . It is desirable to minimize both the stopping time U which counts the total number of unlabeled or labeled queries and the number of labeled queries requested L : = ∑Ut=1 1 { ξt = 1 } . In this setting , at each time t the learner must make the decision of whether to accept the available measurement xt , or abstain and wait for an even more informative measurement . While abstention may result in a smaller total labeled sample complexity L , the stopping time U may be very large . This paper characterizes the set of feasible pairs ( U , L ) that are necessary and sufficient to identify z∗ with probability at least 1 − δ when xt are drawn IID at each time t from a distribution ν . Moreover , we propose an algorithm that nearly obtains the minimal information theoretic label sample complexity L for any desired unlabeled sample complexity U . While characterizing the sample complexity of selective sampling for online best arm identification is the primary theoretical goal of this work , the study was initially motivated by fundamental questions about how to optimally trade-off the value of information versus time . Even for this idealized linear setting , it is far from obvious a priori what an optimal decision rule ξt looks like and if it can even be succinctly described , or if it is simply the solution to an opaque optimization problem . Remarkably , ∗Equal contribution . Alphabetical order . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . we show that for every feasible , optimal operating pair ( U , L ) there exists a matrix A ∈ Rd×d such that the optimal decision rule takes on the form ξt = 1 { x > Ax ≥ 1 } when xt ∼ ν iid . The fact that for any smooth distribution ν the decision rule is a hard decision equivalent to xt falling outside a fixed ellipse or not , and not a stochastic rule that varies complementarily with the density of ν over space is perhaps unexpected . To motivate the problem description , suppose on each day t = 1 , 2 , . . . a food blogger posts the Cocktail of the Day with a recipe described by a feature vector xt ∈ Rd . You have the ingredients ( and skills ) to make any possible cocktail in the space of all cocktails Z , but you don ’ t know which one you ’ d like the most , i.e. , z∗ : = arg maxz∈Z〈z , θ∗〉 , where θ∗ captures your preferences over cocktail recipes . You decide to use the Cocktail of the Day to inform your search . That is , each day you are presented with the cocktail recipe xt ∈ Rd , and if you choose to make it ( ξt = 1 ) you observe your preference for the cocktail yt with E [ yt ] = 〈xt , θ∗〉 . Of course , making cocktails can get costly , so you don ’ t want to make each day ’ s cocktail , but rather you will only make the cocktail if xt is informative about θ∗ ( e.g. , uses a new combination of ingredients ) . At the same time , waiting too many days before making the next cocktail of the day may mean that you never get to learn ( and hence drink ) the cocktail z∗ you like best . The setting above is not limited to cocktails , but rather naturally generalizes to discovering the efficacy of drugs and other therapeutics where blood and tissue samples come to the clinic in a stream and the researcher has to choose whether to take a potentially costly measurement . Our results hold for arbitrary θ∗ ∈ Rd , sets X ⊂ Rd and Z ⊂ Rd , and measures ν ∈ 4X 1 for which we assume xt ∼ ν is drawn IID . The assumption that each xt is IID allows us to make very strong statements about optimality . To summarize , our contributions are as follows : • We present fundamental limits on the trade-off between the amount of unlabelled data and labelled data in the form of ( the first ) information theoretic lower bounds for selective sampling problems that we are aware of . Naturally , they say that there is an absolute minimum amount of unlabelled data that is necessary to solve the problem , but then for any amount of unlabelled data beyond this critical value , the bounds say that the amount of labelled data must exceed some value as a function of the unlabelled data used . • We propose an algorithm that nearly matches the lower bound at all feasible trade-off points in the sense that given any unlabelled data budget that exceeds the critical threshold , the algorithm takes no more labels than the lower bound suggests . Thus , the upper and lower bounds sketch out a curve of all possible operating points , and the algorithm achieves any point on this curve . • We characterize the optimal decision rule of whether to take a sample or not , based on any critical point is a simple test : Accept xt ∈ Rd if x > t Axt ≥ 1 for some matrix A that depends on the desired operating point and geometry of the task . Geometrically , this is equivalent to xt falling inside or outside an ellipsoid . • Our framework is also general enough to capture binary classification , and consequently , we prove results there that improve upon state of the art . 1.1 Related Work . Selective Sampling in the Streaming Setting : Online prediction , the setting in which the selective sampling framework was introduced , is a closely related problem to the one studied in this paper and enjoys a much more developed literature [ 6 , 9 , 1 , 7 ] . In the linear online prediction setting , for t = 1 , 2 , . . . Nature reveals xt ∈ Rd , the learner predicts ŷt and incurs a loss ` ( ŷt , yt ) , and then the learner decides whether to observe yt ( i.e. , ξt = 1 ) or not ( ξt = 0 ) , where yt is a label generated by a composition of a known link function with a linear function of xt . For example , in the classification setting [ 1 , 6 , 9 ] , one setting assumes yt ∈ { −1 , 1 } with E [ yt|xt ] = 〈xt , θ∗〉 for some unknown θ∗ ∈ Rd , and ` ( ŷt , yt ) = 1 { ŷt 6= yt } . In the regression setting [ 7 ] , one observes yt ∈ [ −1 , 1 ] with E [ yt|xt ] = 〈xt , θ∗〉 again , and ` ( ŷt , yt ) = ( ŷt − yt ) 2 . After any amount of time U , the learner is incentivized to minimize both the amount of requested labels ∑U t=1 1 { ξt = 1 } and the cumulative loss ∑U t=1 ` ( yt , ŷt ) ( or some measure of regret which compares to predictions using the unknown θ∗ ) . If every label yt is requested then L = U and this is just the classical online learning setting . 1We denote the set of probability measures over X as 4X . These works give a guarantee on the regret and labeled points taken in terms of the hardness of the stream relative to a learner which would see the label at every time . Most do not give the learner the ability to select an operating point that provides a trade-off between the amount of unlabeled versus labeled data taken . Those few works that propose algorithms that do provide this functionality do not provide lower bounds that match their given upper bounds , leaving it unclear whether their algorithm optimally negotiates this trade-off . In contrast , our work fully characterizes the trade-off between the amount of unlabeled and labeled data through an information-theoretic lower bound and a matching upper bound . Specifically , our algorithm includes a tuning parameter , call it τ , that controls the trade-off between the evaluation metric of interest ( for us , the quality of the recommended z ∈ Z ) , the label complexity L , and the amount of unlabelled data U that is necessary before the metric of interest can be non-trivial . We prove that each possible setting of τ parametrizes all possible trade-offs between unlabeled and labeled data . Our work is perhaps closest to the streaming setting for agnostic active classification [ 8 , 15 ] where each xs is drawn i.i.d . from an underlying distribution ν on X , and indeed our results can be specialized to this setting as we discuss in Section 3 . These papers also evaluate themselves at a single point on the tradeoff curve , namely the number of samples needed in passive supervised learning to obtain a learner with excess risk at most . They provide minimax guarantees on the amount of labeled data needed in terms of the disagreement coefficient [ 12 ] . In contrast , again , our results characterize the full trade-off between the amount of unlabeled data seen , and the amount of labeled data needed to achieve the target excess risk . We note that using online-to-batch conversion methods , [ 9 , 1 , 6 ] also provide results on the amount of labeled data needed but they assume a very specific parametric form to their label distribution unlike our setting which is agnostic . Other works have characterized selective sampling for classification in the realizable setting that assumes there exists a classifer among the set under consideration that perfectly labels every yt [ 13 ] –our work addresses the agnostic setting where no such assumption is made . Finally , our results apply under the more general setting of domain adaptation under covariate shift where we are observing data drawn from the stream ν , but we will evaluate the excess risk of our resulting classifier on a different stream π [ 22 , 23 , 26 ] . Best-Arm Identification and Online Experimental Design . Our techniques are based on experimental design methods for best-arm identification in linear bandits , see [ 24 , 11 , 5 ] . In the setting of these works , there exists a pool of examples X and at each time any x ∈ X can be selected with replacement . The goal is to identify the best arm using as few total selections ( labels ) as possible . Their algorithms are based on arm-elimination . Specifically , they select examples with probability proportional to an approximate G-optimal design with respect to the current remaining arms . Then , during each round after taking measurements , those arms with high probability of being suboptimal will be eliminated . Remarkably , near-optimal sample complexity has been achieved under this setting . While we apply these techniques of arm-elimination and sampling through G-optimal design , the major difference is that we are facing a stream instead of a pool of examples . Finally , [ 10 ] considers a different online experiment design setup where ( adversarially chosen ) experiments arrive sequentially and a primal-dual algorithm decides whether to choose each , subject to a total budget . [ 10 ] studies the competitive ratio of such algorithms ( in the manner of online packing algorithms ) for problems such as D-optimal experiment design .

Summary: This paper considers the problem of best-arm identification in a set Z. The environment samples a vector x from a set X and provides it to the learner. The learner may either choose to observe the reward corresponding to x (given by theta^T x + noise) or ignore it. The authors derive lower and upper bounds on U (number of samples provided by the environment) and L (number of rewards requested by the learner) in terms of a tuning parameter tau and the error probability delta.  The algorithm runs in rounds. In each round, there is an active set of arms that can potentially be the best arm. The learner solves an optimization problem to find a function P: X -> [0, 1], where P(x) is the probability with which it observes the reward for x. The goal of this optimization problem is to minimize the expected number of observations made in tau steps, while ensuring that P(x) does not reject so many samples that the problem of identifying the best arm with probability delta in the first tau steps becomes infeasible.  The authors also show how their approach can be used for binary classification as well, again with selective sampling of outputs.