Selective Sampling for Online Best-arm Identification

1 Introduction . In this work we consider selective sampling for online best-arm identification . In this setting , at every time step t = 1 , 2 , . . . , Nature reveals a potential measurement xt ∈ X ⊂ Rd to the learner . The learner can choose to either query xt ( ξt = 1 ) or abstain ( ξt = 0 ) and immediately move on to the next time . If the learner chooses to take a query ( ξt = 1 ) , then Nature reveals a noisy linear measurement of an unknown θ∗ ∈ Rd , i.e . yt = 〈xt , θ∗〉+ t where t is mean zero sub-Gaussian noise . Before the start of the game , the learner has knowledge of a set Z ⊂ Rd . The objective of the learner is to identify z∗ : = arg maxz∈Z〈z , θ∗〉 with probability at least 1− δ at a learner specified stopping time U . It is desirable to minimize both the stopping time U which counts the total number of unlabeled or labeled queries and the number of labeled queries requested L : = ∑Ut=1 1 { ξt = 1 } . In this setting , at each time t the learner must make the decision of whether to accept the available measurement xt , or abstain and wait for an even more informative measurement . While abstention may result in a smaller total labeled sample complexity L , the stopping time U may be very large . This paper characterizes the set of feasible pairs ( U , L ) that are necessary and sufficient to identify z∗ with probability at least 1 − δ when xt are drawn IID at each time t from a distribution ν . Moreover , we propose an algorithm that nearly obtains the minimal information theoretic label sample complexity L for any desired unlabeled sample complexity U . While characterizing the sample complexity of selective sampling for online best arm identification is the primary theoretical goal of this work , the study was initially motivated by fundamental questions about how to optimally trade-off the value of information versus time . Even for this idealized linear setting , it is far from obvious a priori what an optimal decision rule ξt looks like and if it can even be succinctly described , or if it is simply the solution to an opaque optimization problem . Remarkably , ∗Equal contribution . Alphabetical order . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . we show that for every feasible , optimal operating pair ( U , L ) there exists a matrix A ∈ Rd×d such that the optimal decision rule takes on the form ξt = 1 { x > Ax ≥ 1 } when xt ∼ ν iid . The fact that for any smooth distribution ν the decision rule is a hard decision equivalent to xt falling outside a fixed ellipse or not , and not a stochastic rule that varies complementarily with the density of ν over space is perhaps unexpected . To motivate the problem description , suppose on each day t = 1 , 2 , . . . a food blogger posts the Cocktail of the Day with a recipe described by a feature vector xt ∈ Rd . You have the ingredients ( and skills ) to make any possible cocktail in the space of all cocktails Z , but you don ’ t know which one you ’ d like the most , i.e. , z∗ : = arg maxz∈Z〈z , θ∗〉 , where θ∗ captures your preferences over cocktail recipes . You decide to use the Cocktail of the Day to inform your search . That is , each day you are presented with the cocktail recipe xt ∈ Rd , and if you choose to make it ( ξt = 1 ) you observe your preference for the cocktail yt with E [ yt ] = 〈xt , θ∗〉 . Of course , making cocktails can get costly , so you don ’ t want to make each day ’ s cocktail , but rather you will only make the cocktail if xt is informative about θ∗ ( e.g. , uses a new combination of ingredients ) . At the same time , waiting too many days before making the next cocktail of the day may mean that you never get to learn ( and hence drink ) the cocktail z∗ you like best . The setting above is not limited to cocktails , but rather naturally generalizes to discovering the efficacy of drugs and other therapeutics where blood and tissue samples come to the clinic in a stream and the researcher has to choose whether to take a potentially costly measurement . Our results hold for arbitrary θ∗ ∈ Rd , sets X ⊂ Rd and Z ⊂ Rd , and measures ν ∈ 4X 1 for which we assume xt ∼ ν is drawn IID . The assumption that each xt is IID allows us to make very strong statements about optimality . To summarize , our contributions are as follows : • We present fundamental limits on the trade-off between the amount of unlabelled data and labelled data in the form of ( the first ) information theoretic lower bounds for selective sampling problems that we are aware of . Naturally , they say that there is an absolute minimum amount of unlabelled data that is necessary to solve the problem , but then for any amount of unlabelled data beyond this critical value , the bounds say that the amount of labelled data must exceed some value as a function of the unlabelled data used . • We propose an algorithm that nearly matches the lower bound at all feasible trade-off points in the sense that given any unlabelled data budget that exceeds the critical threshold , the algorithm takes no more labels than the lower bound suggests . Thus , the upper and lower bounds sketch out a curve of all possible operating points , and the algorithm achieves any point on this curve . • We characterize the optimal decision rule of whether to take a sample or not , based on any critical point is a simple test : Accept xt ∈ Rd if x > t Axt ≥ 1 for some matrix A that depends on the desired operating point and geometry of the task . Geometrically , this is equivalent to xt falling inside or outside an ellipsoid . • Our framework is also general enough to capture binary classification , and consequently , we prove results there that improve upon state of the art . 1.1 Related Work . Selective Sampling in the Streaming Setting : Online prediction , the setting in which the selective sampling framework was introduced , is a closely related problem to the one studied in this paper and enjoys a much more developed literature [ 6 , 9 , 1 , 7 ] . In the linear online prediction setting , for t = 1 , 2 , . . . Nature reveals xt ∈ Rd , the learner predicts ŷt and incurs a loss ` ( ŷt , yt ) , and then the learner decides whether to observe yt ( i.e. , ξt = 1 ) or not ( ξt = 0 ) , where yt is a label generated by a composition of a known link function with a linear function of xt . For example , in the classification setting [ 1 , 6 , 9 ] , one setting assumes yt ∈ { −1 , 1 } with E [ yt|xt ] = 〈xt , θ∗〉 for some unknown θ∗ ∈ Rd , and ` ( ŷt , yt ) = 1 { ŷt 6= yt } . In the regression setting [ 7 ] , one observes yt ∈ [ −1 , 1 ] with E [ yt|xt ] = 〈xt , θ∗〉 again , and ` ( ŷt , yt ) = ( ŷt − yt ) 2 . After any amount of time U , the learner is incentivized to minimize both the amount of requested labels ∑U t=1 1 { ξt = 1 } and the cumulative loss ∑U t=1 ` ( yt , ŷt ) ( or some measure of regret which compares to predictions using the unknown θ∗ ) . If every label yt is requested then L = U and this is just the classical online learning setting . 1We denote the set of probability measures over X as 4X . These works give a guarantee on the regret and labeled points taken in terms of the hardness of the stream relative to a learner which would see the label at every time . Most do not give the learner the ability to select an operating point that provides a trade-off between the amount of unlabeled versus labeled data taken . Those few works that propose algorithms that do provide this functionality do not provide lower bounds that match their given upper bounds , leaving it unclear whether their algorithm optimally negotiates this trade-off . In contrast , our work fully characterizes the trade-off between the amount of unlabeled and labeled data through an information-theoretic lower bound and a matching upper bound . Specifically , our algorithm includes a tuning parameter , call it τ , that controls the trade-off between the evaluation metric of interest ( for us , the quality of the recommended z ∈ Z ) , the label complexity L , and the amount of unlabelled data U that is necessary before the metric of interest can be non-trivial . We prove that each possible setting of τ parametrizes all possible trade-offs between unlabeled and labeled data . Our work is perhaps closest to the streaming setting for agnostic active classification [ 8 , 15 ] where each xs is drawn i.i.d . from an underlying distribution ν on X , and indeed our results can be specialized to this setting as we discuss in Section 3 . These papers also evaluate themselves at a single point on the tradeoff curve , namely the number of samples needed in passive supervised learning to obtain a learner with excess risk at most . They provide minimax guarantees on the amount of labeled data needed in terms of the disagreement coefficient [ 12 ] . In contrast , again , our results characterize the full trade-off between the amount of unlabeled data seen , and the amount of labeled data needed to achieve the target excess risk . We note that using online-to-batch conversion methods , [ 9 , 1 , 6 ] also provide results on the amount of labeled data needed but they assume a very specific parametric form to their label distribution unlike our setting which is agnostic . Other works have characterized selective sampling for classification in the realizable setting that assumes there exists a classifer among the set under consideration that perfectly labels every yt [ 13 ] –our work addresses the agnostic setting where no such assumption is made . Finally , our results apply under the more general setting of domain adaptation under covariate shift where we are observing data drawn from the stream ν , but we will evaluate the excess risk of our resulting classifier on a different stream π [ 22 , 23 , 26 ] . Best-Arm Identification and Online Experimental Design . Our techniques are based on experimental design methods for best-arm identification in linear bandits , see [ 24 , 11 , 5 ] . In the setting of these works , there exists a pool of examples X and at each time any x ∈ X can be selected with replacement . The goal is to identify the best arm using as few total selections ( labels ) as possible . Their algorithms are based on arm-elimination . Specifically , they select examples with probability proportional to an approximate G-optimal design with respect to the current remaining arms . Then , during each round after taking measurements , those arms with high probability of being suboptimal will be eliminated . Remarkably , near-optimal sample complexity has been achieved under this setting . While we apply these techniques of arm-elimination and sampling through G-optimal design , the major difference is that we are facing a stream instead of a pool of examples . Finally , [ 10 ] considers a different online experiment design setup where ( adversarially chosen ) experiments arrive sequentially and a primal-dual algorithm decides whether to choose each , subject to a total budget . [ 10 ] studies the competitive ratio of such algorithms ( in the manner of online packing algorithms ) for problems such as D-optimal experiment design .

Summary: This paper tackles the problem of best-arm identification under linear assumption. The arms are d-dimensional vectors. In addition, the learner has the possiblity to observe or not the (noisy) value of the arm x_t that is proposed by the environment at each time step. The difficulty of this problem lies in the tradeoff between the number L of labeled observations, and the total number U of labeled or unlabeled one. The goal is to minimize both L and U. In the case of i.i.d. x_t's, theoretical lower bounds on the expectation of L and U are provided for any delta-PAC algorithm ; almost matching upper bounds are given as well. A practical elimination-style algorithm is described: it depends on a subroutine solving an optimization problem by stochastic gradient ascent.