Pure Exploration in Kernel and Neural Bandits

1 Introduction . Pure exploration in bandits [ 11 , 12 , 6 ] has been extensively studied in machine learning . Consider a set of arms , where each arm is associated with an unknown reward distribution . The goal is to ∗Equal contribution 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . approximately identify the optimal arm using as few samples as possible . Applications of bandit pure exploration range from medical domains [ 3 ] to online content recommendation [ 40 ] . Despite the popularity of bandit pure exploration , it was previously mainly studied in two relatively restrictive settings : ( 1 ) the standard multi-armed bandit setting [ 22 , 19 , 9 , 18 ] , where the expected rewards among arms are completely unrelated to each other , and ( 2 ) the ( generalized ) linear bandit setting [ 38 , 13 , 10 , 26 ] , where the expected rewards are assumed to be linearly parameterized by some unknown weight vector . The standard multi-armed bandit setting fails to deal with large arm sets , and the linear bandit setting suffers from both model misspecifications ( due to its simplified linear form ) and the curse of dimensionality in the high-dimensional setting . Pure exploration is also studied in continuous spaces . However , guarantees therein scale exponentially with dimension [ 31 , 4 ] . In this paper , we generalize bandit pure exploration to the nonlinear and high-dimensional settings . More specifically , we study the following two settings : ( 1 ) the rewards of arms are parameterized by a function belonging to a Reproducing Kernel Hilbert Space ( RKHS ) , and ( 2 ) the rewards of arms are nonlinear functions that can be approximated by an overparameterized neural network . Problems in these two settings are often high-dimensional in nature . To overcome the curse of dimensionality , we propose to adaptively embed each arm ’ s feature representation in a lower-dimensional space and carefully deal with the induced misspecifications . Note that our approach is conceptually very different from all existing work dealing with model misspecifications : they assume the existence of misspecifications and address it in the original space ( thus dealing with model misspecifications in a passive way ) [ 28 , 7 ] . On the other hand , we deliberately induce ( acceptable ) misspecifications to embed arms into lower-dimensional spaces and thus overcome the curse of dimensionality . 1.1 Contribution and Outline . We make the following main contributions : • In Section 3 , we introduce the idea of adaptive embedding to avoid the curse of dimensionality . The induced model misspecifications are carefully handled , which is novel in the bandit pure exploration setting . The sample complexity is theoretically analyzed and we relate the instancedependent sample complexity to the complexity of a closely-related linear bandit problem without model misspecification . As a by-product , our algorithm can also be applied to constrained highdimensional linear bandit pure exploration to reduce sample complexity . • In Section 4 , we specialize the adaptive embedding scheme to pure exploration in an RKHS . We construct feature mappings from eigenfunctions and eigenvalues of the associated kernel . The effective dimension of the kernel is analyzed , and we provide sample complexity guarantees in terms of the eigenvalue decay of the associated kernel . We rely on a known kernel in this setting . • In Section 5 , we further extend our adaptive embedding scheme to pure exploration with a general nonlinear reward function and model the reward function with an over-parameterized neural network . Sample complexity guarantees are provided with respect to the eigenvalue decay of the associated Neural Tangent Kernel . To the best of our knowledge , this provides the first theoretically founded pure exploration algorithm with a neural network approximation . • In Section 6 , we conduct extensive experiments on both synthetic and real-world datesets to confirm the efficacy of our proposed algorithms . We conclude our paper in Section 7 with open problems . 1.2 Related Work . The bandit pure exploration problem has a long history , dating back to the seminal work by Bechhofer [ 5 ] , Paulson et al . [ 32 ] . One classical objective of pure exploration is Best Arm Identification ( BAI ) , where the goal is to identify the best arm using as few samples as possible [ 22 , 19 , 9 , 15 ] . To make it applicable to a large action space , the BAI problem is also extensively studied as the good arm identification problem , where the goal is to identify an -optimal arm [ 11 , 12 , 20 , 21 , 34 , 23 , 30 ] . The pure exploration problem in linear bandits is initially analyzed in Soare et al . [ 38 ] , where optimal experimental design [ 27 ] is applied to guide the allocation of samples . Other approaches dealing with linear bandits , with various sample complexity guarantees , include adaptive sampling [ 45 ] and an approach called track-and-stop [ 10 ] . Constrained linear bandit pure exploration is also commonly studied with additional assumptions on the reward parameters [ 41 , 10 ] . We note that the track-and-stop approach only achieves optimal instance-dependent sample complexity in the regime where the confidence parameter approaches 0 , but fails to do so in the moderate confidence regime . Fiez et al . [ 13 ] propose an elimination-based algorithm ( with optimal design ) that achieves ( nearly ) instance-dependent sample complexity . such algorithm is further generalized to the combinatorial bandit setting [ 24 ] . Learning with model misspecifications was recently introduced in bandit learning , with the primary emphasis placed on the regret minimization problem [ 16 , 28 , 14 ] . A very recent independent work studies pure exploration in kernel bandits with misspecifications [ 7 ] ; both their and our algorithms follow the framework of RAGE [ 13 ] and draw inspiration from [ 28 ] . Camilleri et al . [ 7 ] propose a robust estimator that works in high-dimensional spaces and also explore the project-then-round idea through regularized least squares . Our algorithms adaptively embed actions into lower dimensional spaces according to some error tolerances ( different embeddings from round to round ) ; our rounding and elimination steps are thus computed only with respect to lower-dimensional embeddings . We additionally study the pure exploration problem with an overparameterized neural network . As mentioned before , our approach is also conceptually different from existing ones : rather than passively dealing with model misspecifications in its original representation , we deliberately and adaptively embed arms into a lower-dimensional space to avoid the curse of dimensionality ; the induced model misspecifications are also carefully dealt with in our algorithms . 2 Problem setting . We introduce the general setting and notations for pure exploration in bandits . Consider a set of arms X ⊆ RD where the number of arms |X | = K is possibly very large . We use an unknown function h : X → [ −1 , 1 ] to represent the true reward of each arm . A noisy feedback h ( x ) + ξ is observed after each sample arm x , where the noise ξ is assumed to be 1-sub-Gaussian . The learner is allowed to allocate her samples based on previously collected information , and the goal is to approximately identify an approximately optimal arm using as few samples as possible . Let x ? = arg maxx∈X h ( x ) denote the optimal arm among X . We aim at developing ( , δ ) -PAC guarantees : for any δ ∈ ( 0 , 1 ) , with probability at least 1− δ , the algorithm outputs an -optimal arm x̂ such that h ( x̂ ) ≥ h ( x ? ) − using a finite number of samples . The performance of the algorithm is measured by its sample complexity , i.e. , the number of samples pulled before it stops and recommends a candidate arm . Notations . We define ∆x = h ( x ? ) − h ( x ) as the sub-optimality gap of arm x . We use the notations Sk : = { x ∈ X : ∆x < 4 · 2−k } ( with S1 = X ) . We consider feature mappings of the form ψd ( · ) : X → Rd , and define ψd ( X ) = { ψd ( x ) : x ∈ X } . We use ΛX = { λ ∈ R|X | : ∑ x∈X λx = 1 , λx ≥ 0 } to denote the ( |X | − 1 ) -dimensional probability simplex over arms in X ; and set Aψd ( λ ) = ∑ x∈X λxψd ( x ) ψd ( x ) > .2 We use ‖x‖A = √ x > Ax to represent the Mahalanobis norm . We also define Y ( V ) = { v − v′ : v , v′ ∈ V } for any set V . For a matrixH ∈ R|X |×|X| , we useH ( x , x′ ) to denote the entry ofH which locates at row x and column x′ . 3 Bandit pure exploration with adaptive embedding . We introduce the idea of bandit pure exploration with adaptive embedding , which can be viewed as an approach that actively trades off sample complexity with accuracy guarantees : we adaptively embed the feature representation into lower-dimensional spaces to avoid the curse of dimensionality , and conduct pure exploration with misspecified linear bandits . The embedding dimensions are carefully selected so that we can identify an -optimal arm . We formalize the idea as follows . For any d ∈ N , we assume the existence of a feature mapping ψd : X → Rd and a unknown reward vector θd ∈ Rd such that , for any x ∈ X , h ( x ) = 〈ψd ( x ) , θd〉+ ηd ( x ) , where ηd ( x ) represents the induced approximation error on arm xwith respect to the low-dimensional embedding ψd ( · ) . Without loss of generality , we assume that the action set X is rich enough so that ψd ( X ) spans Rd for d considered in this paper . Otherwise , one can always project feature 2A generalized inversion is used for singular matrices . We refer to Appendix A.1 for detailed discussion . representations ψd ( X ) into an even lower-dimensional space without losing information in the linear component . We use γ̃ : N → R to represent the misspecification level : an upper bound of the induced approximation error across all arms , i.e. , maxx∈X |ηd ( x ) | ≤ γ̃ ( d ) . We define g ( d , ζ ) : = ( 1 + ζ ) infλ∈ΛX supy∈Y ( ψd ( X ) ) ‖y‖2Aψd ( λ ) −1 , which represents the optimal value of a transductive design among embeddings in Rd . We define γ ( d ) : = ( 16 + 8 √ g ( d , ζ ) ) γ̃ ( d ) , which quantifies the sub-optimality gap of the identified arm in the worst case . One can easily show γ ( d ) ≤ O ( γ̃ ( d ) √ d ) through Kiefer-Wolfowitz theorem [ 27 ] . Remark 1 . We believe such optimality guarantees are un-improvable in general . In fact , a hard instance is constructed in [ 28 ] showing that , even with deterministic feedback , identifying a o ( γ̃ ( d ) √ d ) optimal arm requires sample complexity exponential on d. On the other side , identifying a Ω ( γ̃ ( d ) √ d ) optimal only requires sample complexity polynomially in d. Such a sharp trade-off between optimality and sample complexity motivates our definition of γ ( d ) ( and our sample complexities are polynomially in d ) . We assume the knowledge of both the feature mapping ψd ( · ) and the error function γ̃ ( · ) . This assumption is mild since one can explicitly construct/analyze ψd ( · ) and γ̃ ( · ) in many cases ( as discussed in Section 3.2 , Section 4 and Section 5 ) . We further assume that γ ( d ) can be made arbitrarily small for large enough d. Such assumption trivially holds if the rewards are perfectly explained for d large enough , i.e. , γ̃ ( d ) = 0 . We now define the effective dimension with respect to γ ( d ) ( induced from feature mapping ψd ( · ) ) as follows . Definition 1 . For any > 0 , we define the effective dimension as deff ( ) : = min { d ≥ 1 : γ ( d ) ≤ } . In general , the effective dimension deff ( ) captures the smallest dimension one needs to explore in order to identify an -optimal arm . Similar notions have been previously used in regret minimization settings [ 42 , 43 ] . One can easily see that deff ( 1 ) ≤ deff ( 2 ) as long as 1 ≥ 2 .

Summary: This paper introduces a new approach to tackle bandit pure exploration in high dimensional setting. Recall that the literature generally deals with a finite number of arms, or scale exponentially with the dimension of the feature space. The idea, developed in Algorithm 1, is as follows. At each iteration of a RAGE-like algorithm, the arms are embedded into a lower dimensional space. The dimension is chosen as the smallest integer such that the induced error does not harm the subsequent computations and guarantees. Guarantees for Algorithm 1 are derived in Theorem 1, and expressed in terms of the effective dimension. An example of sequence of feature maps is given for linear bandits, based on the SVD decomposition of the design matrix. Under boundedness of parameter $\theta_\star$, it is possible to control the induced error by the successive approximations. When the reward function is assumed to be in a RKHS, using the eigendecomposition of the associated integral operator allows to design similarly a series of feature maps with controllable error. The effective dimension can be bounded in terms of the eigendecay of the kernel. The reward function can also be approximated using a neural network. The algorithm (Algorithm 2) then also needs to learn the weights of the network jointly. Guarantees for Algorithm 2 are stated in Theorem 3. Finally, Section 6 presents some experiments on synthetic and real world datasets showing that the proposed Algorithms exhibit smaller sample complexities than Action Elim.