Misspecified Gaussian Process Bandit Optimization

1 Introduction . Bandit optimization has been successfully used in a great number of machine learning and real-world applications , e.g. , in mobile health [ 42 ] , environmental monitoring [ 40 ] , economics [ 27 ] , hyperparameter tuning [ 26 ] , to name a few . To scale to large or continuous domains , modern bandit approaches try to model and exploit the problem structure that is often manifested as correlations in rewards of `` similar '' actions . Hence , the key idea of kernelized bandits is to consider only smooth reward functions of a low norm belonging to a chosen Reproducing Kernel Hilbert Space ( RKHS ) of functions . This permits the application of flexible nonparametric Gaussian process ( GP ) models and Bayesian optimization methods via a well-studied link between RKHS functions and GPs ( see , e.g. , [ 18 ] for a concise review ) . A vast majority of previous works on nonparametric kernelized bandits have focused on designing algorithms and theoretical bounds on the standard notions of regret ( see , e.g. , [ 40 , 9 , 35 ] ) . However , they solely focus on the realizable ( i.e. , well-specified ) case in which one assumes perfect knowledge of the true function class . For example , the analysis of the prominent GP-UCB [ 40 ] algorithm assumes the model to be well-specified and ignores potential misspecification issues . As the realizability assumption may be too restrictive in real applications , we focus on the case where it may only hold approximately . In practice , model misspecifications can arise due to various reasons , such as incorrect choice of kernel , consideration of an overly smooth function class , hyperparameter estimation errors , etc . Hence , an open question is to characterize the impact of model misspecification in the kernelized setting , and to design robust algorithms whose performance degrades optimally with the increasing level of misspecification . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . In this paper , we study the GP bandit problem with model misspecification in which the true unknown function might be ✏-far ( as measured in the max norm ) from a member of the learner ’ s assumed hypothesis class . We propose a novel GP bandit algorithm and regret bounds that depend on the misspecification error , time horizon , and underlying kernel . Specifically , we present an algorithm that is based on the classical uncertainty sampling approach that is frequently used in Bayesian optimization and experimental design . Importantly , our main presented algorithm assumes no knowledge of the misspecification error ✏ and achieves standard regret rates in the realizable case . Related work on GP bandits . GP bandit algorithms have received significant attention in recent years ( e.g. , [ 40 , 11 , 9 , 39 ] ) . While the most popular approaches in the stochastic setting rely on upper confidence bound ( UCB ) and Thompson sampling strategies , a number of works also consider uncertainty sampling procedures ( e.g. , [ 10 , 41 , 6 ] ) . Beyond the standard setting , numerous works have also considered the contextual bandit setting ( e.g. , [ 22 , 44 , 21 ] ) , while the case of unknown kernel hyperparameters and misspecified smoothness has been studied in , e.g. , [ 46 , 3 , 47 ] . [ 46 , 3 ] assume that the reward function is smooth as measured by the known ( or partially known ) kernel , while in our problem , we allow the unknown function to lie outside a given kernel ’ s reproducing space . Related corruption-robust GP bandits in which an adversary can additively perturb the observed rewards are recently considered in [ 4 ] . In Section 2 , we also discuss how the misspecified problem can be seen from this perspective , where the corruption function is fixed and the adversarial budget scales with the time horizon . While the focus of [ 4 ] is on protecting against an adaptive adversary and thus designing randomized algorithms and regret bounds that depend on the adversarial budget , we propose deterministic algorithms and analyze the impact of misspecification . Apart from corruptions , several works have considered other robust aspects in GP bandits , such as designing robust strategies against the shift in uncontrollable covariates [ 5 , 20 , 31 , 38 , 7 ] . While they report robust regret guarantees , they still assume the realizable case only . Our goal of attaining small regret despite the wrong hypothesis class requires very different techniques from these previous works . Related work on misspecified linear bandits . Recently , works on reinforcement learning with misspecified linear features ( e.g. , [ 12 , 45 , 17 ] ) have renewed interest in the related misspecified linear bandits ( e.g. , [ 48 ] , [ 24 ] , [ 30 ] , [ 15 ] ) first introduced in [ 16 ] . In [ 16 ] , the authors show that standard algorithms must suffer ⌦ ( ✏T ) regret under an additive ✏–perturbation of the linear model . Recently , [ 48 ] propose a robust variant of OFUL [ 1 ] that requires knowing the misspecification parameter ✏ . In particular , their algorithm obtains a high-probability Õ ( d p T + ✏ p dT ) regret bound . In [ 24 ] , the authors propose another arm-elimination algorithm based on G-experimental design . Unlike the previous works , their algorithm is agnostic to the misspecification level , and its performance matches the lower bounds . As shown in [ 24 ] , when the number of actions K is large ( K d ) , the “ price ” of ✏-misspecification must grow as ⌦ ( ✏ p dT ) .1 Our main algorithm is inspired by the proposed technique for the finite-arm misspecified linear bandit setting [ 24 ] . It works in the more general kernelized setting , uses a simpler data acquisition rule often used in Bayesian optimization and experimental design , and recovers the same optimal guarantees when instantiated with linear kernel . Several works have recently considered the misspecified contextual linear bandit problem with unknown model misspecification ✏ . [ 14 ] introduce a new family of algorithms that require access to an online oracle for square loss regression and address the case of adversarial contexts . Concurrent work of [ 33 ] solves the case when contexts / action sets are stochastic . Both works ( [ 14 ] and [ 33 ] ) leverage CORRAL-type aggregation [ 2 ] of contextual bandit algorithms and achieve the optimal Õ ( p dT ✏ + d p T ) regret bound . Finally , in [ 32 ] , the authors present a practical master algorithm that plays base algorithms that come with a candidate regret bound that may not hold during all rounds . The master algorithm plays base algorithms in a balanced way and suitably eliminates algorithms whose regret bound is no longer valid . Similarly to the previous works that rely on the CORRAL-type master algorithms , we use the balancing master algorithm of [ 33 ] together with our GP-bandit base algorithm to provide contextual misspecified regret bounds . Around the time of the submission of this work , a related approach that among others also considers misspecified kernel bandits appeared online [ 8 ] . [ 8 , Theorem 3 ] contains the same regret scaling due to misspecification as we obtain in our results . The main difference between the two works is in the proposed algorithms and analysis techniques . Our approach does not require robust estimators and simply uses the standard ones ( i.e. , GP posterior/ kernelized ridge regression mean and variance 1A result by [ 15 ] further shows that this result can be improved to O ( ✏ p KT ) in the small-K regime . estimators ) that can be computed in the closed-form . It can also handle infinite action sets similarly to the classical Bayesian optimization algorithms ; it utilizes a single acquisition function that , in practice , can be maximized via standard off-the-shelf global optimization solvers . We also present a complete treatment of the misspecified problem by showing the failure of standard UCB approaches , algorithms for known and unknown ✏ , an impossibility result , and extensions to the contextual bandit setting . Finally , our main algorithm ( Algorithm 2 ) demonstrates the use of a different acquisition function in comparison to [ 24 , 8 ] that relies on standard and non-robust estimators . Contributions . In this paper , we systematically handle model misspecification in GP bandits . Specifically , this paper makes the following contributions : • We introduce a misspecified kernelized bandit problem , and for known misspecification error ✏ , we present the EC-GP-UCB algorithm with enlarged confidence bounds that achieves cumulative RT = O ( B p TT+ T p T+✏T p T ) regret . Our simple lower bound argument shows that ⌦ ( T ✏ ) regret is unavoidable in the general kernelized setting . • For when ✏ is unknown , we propose another algorithm based on uncertainty sampling and phased exploration that achieves ( up to polylog factors ) the previous regret rates in the misspecified setting , and standard regret guarantees in the realizable case ( when ✏ = 0 ) . • Finally , we consider a misspecified contextual kernelized problem , and show that when action sets are stochastic , our EC-GP-UCB algorithm can be effectively combined with the regret bound balancing strategy from [ 33 ] to achieve previous regret bounds ( up to some additive lower order terms ) . 2 Problem statement . We consider the problem of sequentially maximizing some black-box reward function f⇤ : D ! R over a known compact set of actions D ⇢ Rd . To learn about f⇤ , the learner relies on sequential noisy bandit feedback , i.e. , at every round t , the learner selects xt 2 D and obtains a noisy observation y ⇤ t = f ⇤ ( xt ) + ⌘t , ( 1 ) where we assume independent ( over rounds ) -sub-Gaussian noise ( see Appendix A for definition ) . Let k ( · , · ) denote a positive definite kernel function defined on D ⇥D and Hk ( D ) be its associated Reproducing Kernel Hilbert Space ( RKHS ) of well-behaved functions . Suppose that before making any decision , the learner is provided with a hypothesis class Fk ( D ; B ) = { f 2 Hk ( D ) : kfkk B } , ( 2 ) where every member function has a bounded RKHS norm kfkk = p hf , fi for some B > 0 , that measures the complexity of f with respect to kernel k ( · , · ) . We consider kernel functions such that k ( x , x ) 1 for every x 2 D. Most commonly used kernel functions that satisfy this property are outlined in Appendix A . The standard setting assumes a realizable ( i.e. , well-specified ) scenario in which f⇤ 2 Fk ( D ; B ) , i.e. , the unknown function is a member of the known RKHS with bounded norm . In contrast , in the misspecified setting , we assume that the learner is informed that f⇤ can be uniformly approximated by a member from the given hypothesis class Fk ( D ; B ) , i.e. , min f2Fk ( D ; B ) kf f ⇤ k1 ✏ , ( 3 ) for some ✏ > 0 and max-norm k · k1 . Here , we note that if two functions are close in RKHS norm , then they are also pointwise close but the reverse does not need to be true . Hence , the true function f ⇤ can in principle have a significantly larger RKHS norm than any f from Fk ( D ; B ) , or it might not even belong to the considered RKHS . We also note that in the case of continuous universal kernels [ 28 ] ( i.e. , due to the universal function approximation property of such kernels ) , any continuous function f⇤ on D satisfies the above assumption ( Eq . ( 3 ) ) for any ✏ > 0 and Fk ( D ; B ) with suitably large RKHS norm bound B . This is a difference in comparison to the previously studied misspecified linear setting , and another motivation to study the misspecified kernelized problem . As in the standard setting ( which corresponds to the case when ✏ = 0 ) , we assume that k ( · , · ) and B are known to the learner . The goal of the learner is to minimize the cumulative regret R ⇤ T = TX t=1 max x2D f ⇤ ( x ) f⇤ ( xt ) , ( 4 ) where T is a time horizon . We denote instantaneous regret at round t as r⇤t = f⇤ ( x⇤ ) f⇤ ( xt ) , where x⇤ 2 argmaxx2D f⇤ ( x ) , and note that R⇤T = PT t=1 r ⇤ t . The learner ’ s algorithm knows Fk ( D ; B ) , meaning that it takes D , k ( · , · ) and B as input . Crucially , in our most general considered problem variant , the exact value of misspecification ✏ is assumed to be unknown , and so we seek an algorithm that can adapt to any ✏ > 0 , including the realizable case when ✏ = 0 . Alternatively , one can consider competing with a best-in-class benchmark , i.e. , RT = TX t=1 max x2D f̃ ( x ) f̃ ( xt ) , ( 5 ) where f̃ 2 argmin f2Fk ( D ; B ) kf f ⇤ k1 ( 6 ) and |f̃ ( x ) f⇤ ( x ) | ✏ for every x 2 D. Here , the goal is to minimize cumulative regret in case the true unknown objective is f̃ 2 Fk ( D ; B ) , while noisy observations of f̃ that the learner receives are at most ✏-misspecified , i.e. , y ⇤ t = f̃ ( xt ) +m ( xt ) | { z } f⇤ ( xt ) +⌘t , ( 7 ) where m ( · ) : D ! [ ✏ , ✏ ] is a fixed and unknown function . Since it holds that kf̃ f⇤k1 ✏ , the absolute difference between RT and R⇤T is at most 2✏T , and as will already become clear in Section 3.2 , this difference will have no significant impact on the scalings in our main bound ( since R⇤T = ⌦ ( ✏T ) ) . In our theoretical analysis , we will interchangeably use both regret definitions ( from Eqs . ( 4 ) and ( 5 ) ) .

Summary: The paper focuses on the kernelized bandit problem where the model can be misspecified: the objective function lies outside the given RKHS such that its distance (induced from the max-norm) to any member of the RKHS is at least $\epsilon$. It proposes 3 different algorithms which are capable of addressing the problem in different scenarios: known $\epsilon$, unknown $\epsilon$, and a contextual bandit setting. The performance of these methods are theoretically analyzed via the upper confidence bounds of the cumulative regrets which connect the algorithms’ performance to $\epsilon$, the time horizon, and the kernel.