Practical, Provably-Correct Interactive Learning in the Realizable Setting: The Power of True Believers

1 Introduction . In this paper , we study interactive learning where an algorithm makes a decision and observes feedback that it then uses to update its behavior . Interactive learning problems are becoming increasingly widespread in the information era . Examples include A/B/n testing where technology companies perform large-scale experiments to adaptively collect data to optimize their products on platforms like websites or smart phone applications [ 1 ] ; active classification where learning algorithms adaptively collect data with the hope of learning high-quality predictive models using a much smaller number of labels than is typically required in supervised learning [ 2 ] ; and environmental monitoring using sensor networks [ 3 ] . At a high-level , there are two main algorithmic paradigms for interactive learning : agnostic algorithms and realizability-based algorithms . Agnostic algorithms may use a model class F to guide learning , but do not assume that the true data-generating process is well-modeled by F . Because of this , agnostic algorithms tend to have the advantages of being robust to model misspecification and noise . Due to these virtues , agnostic algorithms have received much attention in the literature on interactive learning , e.g. , in active classification [ 4 , 5 , 6 ] . By contrast , realizability-based algorithms assume 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . that the model class F accurately models the real-world and leverages the structure in F to guide and potentially accelerate learning . Computationally efficient realizability-based algorithms have only been developed for specific model classes for problems like best arm identification [ 7 , 8 , 9 ] and regret minimization [ 10 ] , and the literature lacks a general framework for developing computationally efficient minimax optimal algorithms for generic function classes in the realizable setting . The starting point of this paper is the basic question : in the realizable setting , can agnostic algorithms compete with realizability-based algorithms ? In this paper , we begin by giving a series of negative results that demonstrate that agnostic algorithms pay a significant cost for their robustness to model mispecification . As an example , we show that any agnostic active classification algorithm is minimax suboptimal for a class of realizable instances and thus has no hope of competing with realizability-based algorithms in the realizable setting . These results motivate us to develop a general framework for computationally efficient and sample-efficient algorithms for generic function classes in the realizable setting . In doing so , we solve an open problem dating back to the work of [ 11 ] on the Haystack Dimension , developing the first computationally efficient algorithm for best arm identification with generic function classes that matches the minimax lower bound up to logarithmic factors . Finally , we empirically demonstrate the generality and practicality of our new approach , GRAILS , for function classes ranging from vanilla kernel methods to kernel methods with side information and to the class of convex functions . 2 Problem Setup . Let X denote the input space and Y ⇢ R the output space . We assume that |Y| < 1 , but will relax this assumption later . Let x1 , . . . , xn 2 X be a fixed pool of n measurements ( or arms ) with associated scores y1 , . . . , yn 2 Y . At each round t , the learner selects ( or queries ) It 2 [ n ] and observes yIt . We assume that the learner is given a function class F ⇢ YX where YX denotes the set of all functions mapping X to Y . We say that realizability holds if there exists f⇤ 2 F such that f ⇤ ( xi ) = yi for all i 2 [ n ] . An algorithm is realizability-based if it assumes realizability . We focus on the noiseless setting here , but it is straightforward to extend our algorithms to handle the case where yIt is perturbed by independent , additive noise by simple repeated sampling . We consider the following three objectives : • Best arm identification : The goal is to identify an element of argmin i2 [ n ] yi using as few queries as possible . • Cumulative Loss Minimization : The goal is to identify an element of argmin i2 [ n ] yi while minimizing the loss P T t=1 yIt incurred where T is the round that the agent identifies an element of argmin i2 [ n ] f ⇤ ( xi ) . • Active Classification : The goal is to identify an element of argmin f2F P n i=1 { yi 6= f ( xi ) } using as few queries as possible . Best-arm identification is a well-studied problem with applications ranging from clinical trials to A/B/n testing . Cumulative loss minimization is a new problem . Applications include running a clinical trial to determine which of a collection of drugs is most effective while , due to ethical concerns about giving patients ineffective drugs , minimizing the number of participants with bad outcomes . It is closely related to regret minimization where the goal is instead to minimize P T0 t=1 yIt mini2 [ n ] yi for a fixed time horizon T0 . Finally , active classification is a mature field whose goal is to minimize the number of labels required to learn a high-quality classifier . Our main focus in this work is on minimax optimality for a fixed F in the realizable setting : • Best arm identification : ⇤best ( F ) is the smallest integer q such that there exists some algorithm A such that for every f⇤ 2 F , A outputs an element of argmin i2 [ n ] f ⇤ ( xi ) after at most q queries . • Cumulative Loss Minimization : ⇤loss ( F ) is the smallest real number q such that there exists some algorithm A such that for every f⇤ 2 F , A outputs an element of argmin i2 [ n ] f ⇤ ( xi ) after incurring a loss of at most q . • Active Classification : ⇤class ( F ) is the smallest integer q such that there exists some algorithm A such that for every f⇤ 2 F , A outputs f⇤ after at most q queries . We emphasize that the above notion of minimax optimality is with respect to the class F . Next , we briefly summarize our contributions : • Best arm identification : Assuming we can sample efficiently from a distribution ⇡ with support F , we give a greedy and computationally efficient algorithm that obtains a sample complexity of O ( log ( n ) log ( 1P⇡ ( Sf⇤ ) ) ⇤ best ) where P⇡ ( Sf⇤ ) is the probability of sampling f⇤ and ⇤best is a combinatorial quantity related to the extended teaching dimension and is a minimax lower bound . This is the first computationally efficient algorithm for best arm identification with generic function classes that matches the minimax lower bound up to logarithmic factors . • Loss minimization : We propose a new algorithm that achieves a loss of O ( ⇤loss log ( |F| ) ) in the worst case where ⇤loss is the minimax lower bound . We show that when applied to the regret minimization setting with general function classes , our algorithm achieves a state-of-the-art regret bound that is always better than the prior state-of-the-art regret bound in [ 11 ] , can be arbitrarily better , and for a large set of function classes matches the minimax lower bound up to logarithmic factors . Furthermore , using our techniques from our algorithm for best arm identification , we make this algorithm computationally efficient by leveraging a sampling oracle . • Active Classification : We show that there exists a class of realizable instances such that any agnostic algorithm must query ⌦̃ ( p n ) arms to identify the true classifier f⇤ , but there is a realizability-based algorithm that requires O ( log ( n ) ) queries . This demonstrates an exponential gap between agnostic and realizability-based algorithms for active classification . 3 Related Work . Best arm identification has received much attention in the literature but predominantly for special classes of functions ( e.g. , linear ) [ 12 , 13 , 7 , 8 , 9 ] . By contrast , our work concerns best arm identification with general function classes , which has received much less attention , and is most closely related to [ 11 ] . [ 11 ] introduce a combinatorial quantity HD ( F ) known as the haystack dimension , and propose a greedy algorithm that achieves a sample complexity of HD ( F ) log ( |F| ) , but is not computationally efficient . We build on this work by designing a greedy algorithm that achieves computational efficiency by appealing to [ 14 ] to sample functions from the function class F . A key technical challenge is that the version space under a greedy algorithm is not convex and therefore standard sampling algorithms like “ hit and run ” can not be directly applied to it . [ 15 ] gives a general computationally efficient algorithm that solves many pure exploration problems such as active classification and clustering . Their algorithm requires specifying a distance d between functions in the function class F and proceeds by shrinking the average diameter of F as measured by d. Unfortunately , for problems such as best arm or ✏-good arm identification , it is unclear how to construct the appropriate distance function to achieve optimal performance or even if such a function exists for a given application . Active Classification : There have been many works on active classification [ 6 , 2 ] , both in the realizable setting ( e.g. , [ 16 , 17 , 18 ] ) and on agnostic algorithms ( e.g. , [ 5 , 19 , 4 , 20 ] ) . Proposition 1 shows that agnostic algorithms are minimax suboptimal in the realizable setting , suggesting that they may never be able to match the performance of realizability-based algorithms . Our work is closely related to the problem of Exact learning where a learning algorithm is required to identify the true f ⇤ with probability 1 [ 21 , 22 ] , in contrast to the PAC requirement where a failure probability of is permitted . While this requirement is certainly strong , this framework enables the design of practical algorithms , as suggested by our experiments . Bayesian Optimization : Our work is also related to the vast field of Bayesian optimization . We review a few relevant theoretical results and refer readers to [ 3 ] for a thorough survey . [ 23 ] propose and analyze GP-UCB . Although GP-UCB is applied often to best arm identification problems , [ 23 ] only give a regret bound and optimality is unclear . Other notable works include [ 24 ] whose results are asymptotic and the recent paper [ 25 ] , which gives a new algorithm for best arm identification for kernel bandits based on experimental design , but its optimality is unclear . 4 No Free Lunch for Agnostic Algorithms . Many agnostic algorithms have been proposed in the active classification literature ( e.g . [ 5 , 19 , 4 ] ) . We say an algorithm A is -agnostic if for any labeling of the data y 2 Yn , A finds the best classifier , argmin f2F P n i=1 { y 6= f ( xi ) } , with probability at least 1 . Despite considering a much larger class of possible labelings , agnostic algorithms have been shown to achieve the lower bound in the realizable setting for the well-studied problem of thresholds where F = { f1 , . . . , fn } and fi ( xj ) = 1 if j i and fi ( xj ) = 0 otherwise [ 26 ] . This raises the question : can agnostic algorithms be optimal over classes of realizable instances in general ? The following result shows that the sample complexity of agnostic algorithms is necessarily worse than the minimax lower bound by an exponential factor . Proposition 1 . Let 2 ( 0 , min ( 120 , 1 n ) ) . Consider the active classification setting . There exists x1 , . . . , xn and F forming a class of instances I = { ( xi , f ( xi ) ) ni=1 : f 2 F } such that • the expected number of samples of any -agnostic algorithm is ⌦ ⇣ log ( 1/ ) p n log ( n/ ) ⌘ on one of the instances in I , and • there exists a realizability-based algorithm that solves each instance in I in O ( log ( n ) ) queries . It turns out agnostic algorithms are also minimax suboptimal for regret minimization , a problem closely related to loss minimization as we will discuss in more detail in Section 6 . Here , we say that a regret minimization algorithm A is ✏-agnostic if for any y 2 Yn such that minj2 [ n ] \ { i⇤ } yj yi⇤ ✏ where i⇤ 2 argmini2 [ n ] yi , A suffers at most bounded regret independent of the time horizon . Proposition 2 . There exists F such that any 1-agnostic algorithm suffers regret at least O ( |F| ) for some instance in F while there exists a realizability-based algorithm suffering regret at most O ( log ( |F| ) ) . For best-arm identification , a fully agnostic algorithm must consider any y 2 Yn and it is therefore trivial that it would need to query every xi . Therefore , we consider a weaker notion of agnostic algorithm . For k 2 N and 2 ( 0 , 1 ) , we say an algorithm A is ( , k ) -agnostic if for any y 2 Yn such that minf2F P n i=1 { f ( xi ) 6= yi } k , A identifies argmini2 [ n ] yi with probability at least 1 . Despite only allowing for small amounts of mispecification , there is still an exponential gap between the performance of agnostic algorithms and the minimax lower bound . Proposition 3 . Let 2 ( 0 , min ( 140 , 1 n ) ) . There exists F such that for any algorithm A that is ( , 1 ) -agnostic with respect to F , A takes ⌦ ⇣ n log ( n/ ) ⌘ queries in expectation on some instance in F , while there exists a realizability-based algorithm requiring O ( 1 ) samples on all instances in F . Techniques : The above results rely on a novel approach for constructing instance-dependent lower bounds for the noiseless setting . The key idea is a reduction of the noiseless setting to a setting where observations are corrupted with a Gaussian random variable , that is , when the arm i 2 [ n ] is queried , the agent observes yi + ⌘ where ⌘ ⇠ N ( 0 , 1 ) , instead of yi . This reduction enables the application of the transportation Lemma from the multi-armed bandit literature [ 27 ] to the noiseless setting and thereby to construct instances where agnostic algorithms necessarily perform poorly .

Summary: Consider the problem of best arm identification: there are $n$ arms, each with a reward, and the goal is to select the best arm  with the best reward with as few pulls as possible. The query complexity of this specific problem in the realizable setting (where rewards correspond to an unknown function from a known family) is well understood (up to log factors), but there is no matching efficient algorithm. This paper provides nearly optimal algorithms for classes of functions that are convex (up to a discretization). Algorithms for related interactive learning problems are also considered. Finally, the authors present lower bounds for such problems that show that agnostic algorithms (not assuming realizability) can be much worse.