Pure Exploration in Kernel and Neural Bandits

1 Introduction . Pure exploration in bandits [ 11 , 12 , 6 ] has been extensively studied in machine learning . Consider a set of arms , where each arm is associated with an unknown reward distribution . The goal is to ∗Equal contribution 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . approximately identify the optimal arm using as few samples as possible . Applications of bandit pure exploration range from medical domains [ 3 ] to online content recommendation [ 40 ] . Despite the popularity of bandit pure exploration , it was previously mainly studied in two relatively restrictive settings : ( 1 ) the standard multi-armed bandit setting [ 22 , 19 , 9 , 18 ] , where the expected rewards among arms are completely unrelated to each other , and ( 2 ) the ( generalized ) linear bandit setting [ 38 , 13 , 10 , 26 ] , where the expected rewards are assumed to be linearly parameterized by some unknown weight vector . The standard multi-armed bandit setting fails to deal with large arm sets , and the linear bandit setting suffers from both model misspecifications ( due to its simplified linear form ) and the curse of dimensionality in the high-dimensional setting . Pure exploration is also studied in continuous spaces . However , guarantees therein scale exponentially with dimension [ 31 , 4 ] . In this paper , we generalize bandit pure exploration to the nonlinear and high-dimensional settings . More specifically , we study the following two settings : ( 1 ) the rewards of arms are parameterized by a function belonging to a Reproducing Kernel Hilbert Space ( RKHS ) , and ( 2 ) the rewards of arms are nonlinear functions that can be approximated by an overparameterized neural network . Problems in these two settings are often high-dimensional in nature . To overcome the curse of dimensionality , we propose to adaptively embed each arm ’ s feature representation in a lower-dimensional space and carefully deal with the induced misspecifications . Note that our approach is conceptually very different from all existing work dealing with model misspecifications : they assume the existence of misspecifications and address it in the original space ( thus dealing with model misspecifications in a passive way ) [ 28 , 7 ] . On the other hand , we deliberately induce ( acceptable ) misspecifications to embed arms into lower-dimensional spaces and thus overcome the curse of dimensionality . 1.1 Contribution and Outline . We make the following main contributions : • In Section 3 , we introduce the idea of adaptive embedding to avoid the curse of dimensionality . The induced model misspecifications are carefully handled , which is novel in the bandit pure exploration setting . The sample complexity is theoretically analyzed and we relate the instancedependent sample complexity to the complexity of a closely-related linear bandit problem without model misspecification . As a by-product , our algorithm can also be applied to constrained highdimensional linear bandit pure exploration to reduce sample complexity . • In Section 4 , we specialize the adaptive embedding scheme to pure exploration in an RKHS . We construct feature mappings from eigenfunctions and eigenvalues of the associated kernel . The effective dimension of the kernel is analyzed , and we provide sample complexity guarantees in terms of the eigenvalue decay of the associated kernel . We rely on a known kernel in this setting . • In Section 5 , we further extend our adaptive embedding scheme to pure exploration with a general nonlinear reward function and model the reward function with an over-parameterized neural network . Sample complexity guarantees are provided with respect to the eigenvalue decay of the associated Neural Tangent Kernel . To the best of our knowledge , this provides the first theoretically founded pure exploration algorithm with a neural network approximation . • In Section 6 , we conduct extensive experiments on both synthetic and real-world datesets to confirm the efficacy of our proposed algorithms . We conclude our paper in Section 7 with open problems . 1.2 Related Work . The bandit pure exploration problem has a long history , dating back to the seminal work by Bechhofer [ 5 ] , Paulson et al . [ 32 ] . One classical objective of pure exploration is Best Arm Identification ( BAI ) , where the goal is to identify the best arm using as few samples as possible [ 22 , 19 , 9 , 15 ] . To make it applicable to a large action space , the BAI problem is also extensively studied as the good arm identification problem , where the goal is to identify an -optimal arm [ 11 , 12 , 20 , 21 , 34 , 23 , 30 ] . The pure exploration problem in linear bandits is initially analyzed in Soare et al . [ 38 ] , where optimal experimental design [ 27 ] is applied to guide the allocation of samples . Other approaches dealing with linear bandits , with various sample complexity guarantees , include adaptive sampling [ 45 ] and an approach called track-and-stop [ 10 ] . Constrained linear bandit pure exploration is also commonly studied with additional assumptions on the reward parameters [ 41 , 10 ] . We note that the track-and-stop approach only achieves optimal instance-dependent sample complexity in the regime where the confidence parameter approaches 0 , but fails to do so in the moderate confidence regime . Fiez et al . [ 13 ] propose an elimination-based algorithm ( with optimal design ) that achieves ( nearly ) instance-dependent sample complexity . such algorithm is further generalized to the combinatorial bandit setting [ 24 ] . Learning with model misspecifications was recently introduced in bandit learning , with the primary emphasis placed on the regret minimization problem [ 16 , 28 , 14 ] . A very recent independent work studies pure exploration in kernel bandits with misspecifications [ 7 ] ; both their and our algorithms follow the framework of RAGE [ 13 ] and draw inspiration from [ 28 ] . Camilleri et al . [ 7 ] propose a robust estimator that works in high-dimensional spaces and also explore the project-then-round idea through regularized least squares . Our algorithms adaptively embed actions into lower dimensional spaces according to some error tolerances ( different embeddings from round to round ) ; our rounding and elimination steps are thus computed only with respect to lower-dimensional embeddings . We additionally study the pure exploration problem with an overparameterized neural network . As mentioned before , our approach is also conceptually different from existing ones : rather than passively dealing with model misspecifications in its original representation , we deliberately and adaptively embed arms into a lower-dimensional space to avoid the curse of dimensionality ; the induced model misspecifications are also carefully dealt with in our algorithms . 2 Problem setting . We introduce the general setting and notations for pure exploration in bandits . Consider a set of arms X ⊆ RD where the number of arms |X | = K is possibly very large . We use an unknown function h : X → [ −1 , 1 ] to represent the true reward of each arm . A noisy feedback h ( x ) + ξ is observed after each sample arm x , where the noise ξ is assumed to be 1-sub-Gaussian . The learner is allowed to allocate her samples based on previously collected information , and the goal is to approximately identify an approximately optimal arm using as few samples as possible . Let x ? = arg maxx∈X h ( x ) denote the optimal arm among X . We aim at developing ( , δ ) -PAC guarantees : for any δ ∈ ( 0 , 1 ) , with probability at least 1− δ , the algorithm outputs an -optimal arm x̂ such that h ( x̂ ) ≥ h ( x ? ) − using a finite number of samples . The performance of the algorithm is measured by its sample complexity , i.e. , the number of samples pulled before it stops and recommends a candidate arm . Notations . We define ∆x = h ( x ? ) − h ( x ) as the sub-optimality gap of arm x . We use the notations Sk : = { x ∈ X : ∆x < 4 · 2−k } ( with S1 = X ) . We consider feature mappings of the form ψd ( · ) : X → Rd , and define ψd ( X ) = { ψd ( x ) : x ∈ X } . We use ΛX = { λ ∈ R|X | : ∑ x∈X λx = 1 , λx ≥ 0 } to denote the ( |X | − 1 ) -dimensional probability simplex over arms in X ; and set Aψd ( λ ) = ∑ x∈X λxψd ( x ) ψd ( x ) > .2 We use ‖x‖A = √ x > Ax to represent the Mahalanobis norm . We also define Y ( V ) = { v − v′ : v , v′ ∈ V } for any set V . For a matrixH ∈ R|X |×|X| , we useH ( x , x′ ) to denote the entry ofH which locates at row x and column x′ . 3 Bandit pure exploration with adaptive embedding . We introduce the idea of bandit pure exploration with adaptive embedding , which can be viewed as an approach that actively trades off sample complexity with accuracy guarantees : we adaptively embed the feature representation into lower-dimensional spaces to avoid the curse of dimensionality , and conduct pure exploration with misspecified linear bandits . The embedding dimensions are carefully selected so that we can identify an -optimal arm . We formalize the idea as follows . For any d ∈ N , we assume the existence of a feature mapping ψd : X → Rd and a unknown reward vector θd ∈ Rd such that , for any x ∈ X , h ( x ) = 〈ψd ( x ) , θd〉+ ηd ( x ) , where ηd ( x ) represents the induced approximation error on arm xwith respect to the low-dimensional embedding ψd ( · ) . Without loss of generality , we assume that the action set X is rich enough so that ψd ( X ) spans Rd for d considered in this paper . Otherwise , one can always project feature 2A generalized inversion is used for singular matrices . We refer to Appendix A.1 for detailed discussion . representations ψd ( X ) into an even lower-dimensional space without losing information in the linear component . We use γ̃ : N → R to represent the misspecification level : an upper bound of the induced approximation error across all arms , i.e. , maxx∈X |ηd ( x ) | ≤ γ̃ ( d ) . We define g ( d , ζ ) : = ( 1 + ζ ) infλ∈ΛX supy∈Y ( ψd ( X ) ) ‖y‖2Aψd ( λ ) −1 , which represents the optimal value of a transductive design among embeddings in Rd . We define γ ( d ) : = ( 16 + 8 √ g ( d , ζ ) ) γ̃ ( d ) , which quantifies the sub-optimality gap of the identified arm in the worst case . One can easily show γ ( d ) ≤ O ( γ̃ ( d ) √ d ) through Kiefer-Wolfowitz theorem [ 27 ] . Remark 1 . We believe such optimality guarantees are un-improvable in general . In fact , a hard instance is constructed in [ 28 ] showing that , even with deterministic feedback , identifying a o ( γ̃ ( d ) √ d ) optimal arm requires sample complexity exponential on d. On the other side , identifying a Ω ( γ̃ ( d ) √ d ) optimal only requires sample complexity polynomially in d. Such a sharp trade-off between optimality and sample complexity motivates our definition of γ ( d ) ( and our sample complexities are polynomially in d ) . We assume the knowledge of both the feature mapping ψd ( · ) and the error function γ̃ ( · ) . This assumption is mild since one can explicitly construct/analyze ψd ( · ) and γ̃ ( · ) in many cases ( as discussed in Section 3.2 , Section 4 and Section 5 ) . We further assume that γ ( d ) can be made arbitrarily small for large enough d. Such assumption trivially holds if the rewards are perfectly explained for d large enough , i.e. , γ̃ ( d ) = 0 . We now define the effective dimension with respect to γ ( d ) ( induced from feature mapping ψd ( · ) ) as follows . Definition 1 . For any > 0 , we define the effective dimension as deff ( ) : = min { d ≥ 1 : γ ( d ) ≤ } . In general , the effective dimension deff ( ) captures the smallest dimension one needs to explore in order to identify an -optimal arm . Similar notions have been previously used in regret minimization settings [ 42 , 43 ] . One can easily see that deff ( 1 ) ≤ deff ( 2 ) as long as 1 ≥ 2 .

Summary: Summary: This paper studies \delta-PAC \epsilon-best arm identification when a kernel or neural representation is used to embed the arms. The paper targets regimes where the number of arms K is really large and the similarity between arms can be exploited to reduce the sample complexity of the tests needed to find the best arm.  General remark: The paper is well written and addresses a hard problem. The algorithms are smart and the empirical results are impressive. Despite a few minor remarks listed below, I recommend accepting this paper.  Disclaimer: Unfortunately, my level of expertise does not allow me to verify Appendix F.