The Pareto Frontier of model selection for general Contextual Bandits

1 Introduction . Contextual multi-armed bandits are a fundamental problem in online learning [ Auer et al. , 2002 , Langford and Zhang , 2007 , Chu et al. , 2011 , Abbasi-Yadkori et al. , 2011 ] . The contextual bandit problem proceeds as a repeated game between a learner and an adversary . At every round of the game the adversary prepares a pair of a context and a loss over an action space , the learner observes the context and selects an action from the action space and then observes only the loss of the selected action . The goal of the learner is to minimize their cumulative loss . The performance measure , known as regret , is the difference between the learner ’ s cumulative loss and the smallest loss of a fixed policy , belonging to an apriori determined policy class , mapping contexts to actions . Given a single contextual bandit instance with finite sized policy class , the well-known Exp4 algorithm [ Auer et al. , 2002 ] achieves the optimal regret bound of O ( p KT ln ( |⇧| ) . Regret guarantees degrade with the complexity of the policy class , therefore a a learner might want to leverage “ guesses ” about the optimal policy . Given policy classes ⇧1 ⇢ · · · ⇢ ⇧M , a learner would ideally suffer regret scaling only with the complexity of ⇧m⇤ , the smallest policy class containing the optimal policy ⇡⇤ . While these kind of results are obtainable in full-information games , in which the learner gets to observe the loss for all actions , they are impossible for multi-armed bandits [ Lattimore , 2015 ] . In some aspects , contextual bandits are an intermediate setting between full-information and multi-armed bandits and it is unknown if model selection is possible . Foster et al . [ 2020b ] stated model selection in contextual bandits as a relevant open problem in COLT2020 . Any positive result for model selection in contextual bandits would imply a general way to treat multi-armed bandits with a switching baseline . Furthermore any negative result is conjectured to implicate negative results on another unresolved open problem on second order bounds for full-information games [ Freund , 2016 ] . In this paper , we give a fairly complete answer to the questions above . ⇤Author was at Johns Hopkins University during part of this work . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . P.1 We provide a Pareto frontier of upper bounds for model selection in contextual bandits with finite sized policy classes . P.2 We present matching lower bounds that shows that our upper bounds are tight , thereby resolve the motivating open problems [ Foster et al. , 2020b ] . P.3 We present a novel impossibility result for adapting to the number of switch points under adaptive adversaries [ Besbes et al. , 2014 ] . P.4 We negatively resolve an open problem on second order bounds for full-information [ Freund , 2016 ] . Related work . A problem closely related to contextual bandits with finite policy classes are linear contextual bandits . Model selection in linear contextual bandit problems has recently received significant attention , however none of these resuls transfer to the finite policy case . In the linear bandits problem the m-th policy class is a subset of Rdm and the losses ` t , ⇡ ( x ) , ⇡ 2 ⇧m are linear , that is ` t , ⇡ ( x ) = h✓m , m ( x , ⇡ ( x ) ) i+ ⇠ . Here m : X ⇥A ! Rdm is a feature embedding mapping from context-action pairs into Rdm , ⇠ is mean-zero sub-Gaussian noise with variance proxy equal to one and ✓m 2 Rdm is an unknown parameter . Foster et al . [ 2019 ] assume the contexts are also drawn from an unknown distribution x ⇠ D and propose an algorithm which does not incur more than Õ ( 1 3 ( i ⇤ T ) 2/3 ( Mdi⇤ ) 1/3 ) , where 3 is the smallest eigenvalue of the covariance matrix of feature embeddings ⌃ = Ex⇠D ⇥ 1 M P a2A M ( x , a ) M ( x , a ) > ⇤ . Pacchiano et al . [ 2020b ] propose a different approach based on the corralling algorithm of Agarwal et al . [ 2017 ] which enjoys a Õ ( di⇤ p T ) regret bound for finite action sets and Õ ( d2i⇤ p T ) bound for arbitrary action sets A . Later , Pacchiano et al . [ 2020a ] design an algorithm which enjoys a gap-dependent guarantee under the assumption that all of the miss-specified models have regret Ri ( t ) t , 8t 2 [ T ] . Under such an assumption , the authors recover a regret bounds of the order Õ ( di⇤ p T + d4i⇤/ ) for arbitrary action sets . Cutkosky et al . [ 2020 ] also manage to recover the O ( di⇤ p T ) and O ( d2i⇤ p T ) bounds for the model selection problems through their corralling algorithm . Ghosh et al . [ 2021 ] propose an algorithm which enjoys Õ ⇣ d2M 4.65 + p dm⇤T ⌘ in the finite arm setting , where = min { |✓m⇤ , i| : |✓m⇤ , i| > 0 } is the smallest , in absolute value , entry of ✓m⇤ . Their algorithm also enjoys a similar guarantee for arbitrary action sets with p dm⇤T replaced by dm⇤ p T . Zhu and Nowak [ 2021 ] show that it is impossible to achieve the desired regret guarantees of p dm⇤T without additional assumptions by showing a result similar to the one of Lattimore [ 2015 ] . The work of Lattimore [ 2015 ] states that in the stochastic multi-armed bandit problem it is impossible to achieve p T regret to a fixed arm , without suffering at least K p T regret to a different arm . Chatterji et al . [ 2020 ] study the problem of selecting between an algorithm for the linear contextual bandit problem and the simple stochastic multi-armed bandit problem , that is they aim to achieve simultaneously a regret guarantee which is instance-dependent optimal for the stochastic multi-armed bandit problem and optimal for the finite arm stochastic linear bandit problem . The proposed results only hold under additional assumptions . More generally , the study of the corralling problem , in which we are presented with multiple bandit algorithms and would like to perform as well as the best one , was initiated by Agarwal et al . [ 2017 ] . Other works which fall into the corralling framework are that of Foster et al . [ 2020a ] who study the miss-specified linear contextual bandit problem , that is the observed losses are linear up to some unknown ✏ miss-specification , and the work of Arora et al . [ 2021 ] who study the corralling problem for multi-armed stochastic bandit algorithms . Our work also shows an impossibility result for the stochastic bandit problem with non-stationary rewards . Auer [ 2002 ] first investigates the problem under the assumption that there are L distributional changes throughout the game and gives an algorithm with a Õ ( p KLT ) dynamic regret2 bound , under the assumption that L is known . Auer et al . [ 2019 ] achieves similar regret guarantees without assuming that the number if switches ( or changes ) of the distribution is known . A different measurement of switches is the total variation of changes in distribution VT = PT t=2 kE [ ` t ] E [ ` t 1 ] k1 . Multiple works give dynamic regret bounds of the order Õ ( V 1/3T T 2/3 ) ( hiding dependence on the size of the policy class ) when VT is known , including for extensions of the multi-armed bandit problem like 2In dynamic regret the comparator is the best action for the current distribution . contextual bandits and linear contextual bandits [ Besbes et al. , 2014 , Luo et al. , 2018 , Besbes et al. , 2015 , Wei et al. , 2017 ] . Cheung et al . [ 2019 ] , Zhao et al . [ 2020 ] further show algorithms which enjoy a parameter free regret bound of the order Õ ( V 1/4T T 3/4 ) ( hiding dependence on dimensionality ) for the linear bandits problem . The lower bound in Table 1 might seem to contradict such results . In Section 5.1 we carefully explain why this is not the case . Finally , our lower bounds apply to the problem of devising an algorithm which simultaneously enjoys a second order bound over any fraction of experts . Cesa-Bianchi et al . [ 2007 ] first investigate the problem of second order bounds for the experts problem , in which the proposed algorithm maintains a distribution pt over the set of K experts , during every round of the game . The experts are assumed to have stochastic losses ` t and the work shows an algorithm with Õ ( qPT t=1 Vi⇠pt [ ` t , i ] logK ) regret guarantee . Chaudhuri et al . [ 2009 ] , Chernov and Vovk [ 2010 ] , Luo and Schapire [ 2015 ] , Koolen and Van Erven [ 2015 ] study a different experts problem in which the comparator class for the regret changes from the best expert in hindsight to the uniform distribution over the best b✏Kc experts for an arbitrary positive ✏ . The above works propose algorithms which achieve a Õ ( p T log ( 1/✏ ) ) regret bound for all ✏ simultaneously . Freund [ 2016 ] asks if there exists an algorithm which enjoys both guarantees at the same time , that is , does there exist an algorithm with regret bound Õ ( qPT t=1 Vi⇠pt [ ` t , i ] log ( 1/✏ ) ) which holds simultaneously for all positive ✏ . Notation . For any N 2 N , [ N ] denotes the set { 1 , . . . , N } . Õ notation hides poly-logarithmic factors in the horizon T and the number of arms K but not in the size of the policy classes |⇧m| . 2 Problem setting . We consider the contextual bandit problem with general policy classes of finite size . There are K arms and nested policy classes ( ⇧m ) Mm=1 , where a policy ⇡ 2 ⇧m , ⇡ : X ! [ K ] is a mapping from an arbitrary context space X into the set of K arms . The game is played for T rounds and at any time t , the agent observes a context xt 2 X , selects arm At 2 [ K ] and observes the loss ` t , At from an otherwise unobserved loss vector ` t 2 [ K ] . We measure an algorithm ’ s performance in terms of pseudo-regret , which is the expected cumulative regret of the player against following a fixed policy in hindsight Reg ( T , ⇧ ) = max ⇡2⇧ E `` TX t=1 ` t , At ` t , ⇡ ( xt ) # . Environments . We distinguish between stochastic environments and oblivious or adaptive adversaries . In stochastic environments , there are unknown distribution PX , Q such that xt ⇠ PX and ` t ⇠ Q ( ·|xt ) are i.i.d . samples . In the adversarial regime , the distributions can change over time , i.e . xt ⇠ PX , t , ` t ⇠ Qt ( ·|xt ) . When the choices are fixed at the beginning of the game , the adversary is called oblivious , while an adaptive adversary can chose PX , t , Qt based on all observations up to time t 1 . Often the stochastic-adversarial hybrid problem has been studied with adversarially chosen context but stochastic losses . In our work , all upper bounds hold in the stronger notion where both the losses and the contexts are adaptive , while the lower bounds hold for the weaker notion where only the contexts are adaptive . Open problem [ Foster et al. , 2020b ] . The regret upper bounds for all regimes introduced above for a fixed policy class ⇧ of finite size are of the order Õ ( p ln ( |⇧| ) KT ) and can be achieved by the Exp4 algorithm [ Auer et al. , 2002 ] . The question asked by Foster et al . [ 2020b ] : For a nested sequence of policies ⇧1 ⇢ ⇧2 ⇢ · · · ⇢ ⇧M , is there a universal ↵ 2 [ 12 , 1 ) such that a regret bound of Reg ( T , ⇧m ) = PolyLog ( K , M ) Õ ln ( |⇧m| ) 1 ↵ T ↵ ( 1 ) is obtainable for all m 2 [ M ] simultaneously ? W.l.o.g . we can assume that M = O ( ln ln ( |⇧M | ) ) = O ( ln ( T ) ) . Otherwise we take a subset of policy classes that includes ⇧M and where two consequent policy classes at least square in size . Due to nestedness , any guarantees on this subset of models imply up to constants the same bounds on the full set . S-switch A motivating example for studying nested policy classes is the S-switch problem . The context is simply xt = t 2 [ T ] and the set of policies is given by ⇧S = ( ⇡ T 1X t=1 I { ⇡t 6= ⇡t+1 } S ) , the set of policies that changes its action not more than S many times . Any positive result for contextual bandits with finite sized policy classes would provide algorithms that adapt to the number of switch points , since ln |⇧S | = Õ ( S ) . To make clear what problem we are considering , we are using RegSW ( T , S ) to denote the regret in the switching problem . Next , we define the class of proper algorithms which choose their policy at every time step t independently of context xt . Restricting our attention to such algorithms greatly reduces the technicalities for lower bound proofs in the non-adaptive regimes . The lower bound for this class of algorithms is also at the core of the argument for adaptive ( improper ) algorithms in stochastic environments . Definition 1 . We call an algorithm proper , if at any time t , the algorithm follows the recommendation of a policy ⇡it 2 ⇧M , and if the choice of it by the algorithm , is independent of the context xt . Example . EXP4 is proper . The properness assumption intuitively allows us to reduce the model selection problem to a bandit-like problem in the space of all policies ⇧M . We give more details in Section 4.2 and Appendix B.3 .

Summary: This paper studies the model selection for contextual bandits. It approaches two open problems. One is a COLT2020 open problem of Foster et al, which askes if there exists a constant such that a regret bound (1) holds for all m \in [0,M] simultaneously for a nested sequence of a finite set of policies m = 1, ..., M . The other one is a COLT2016 open problem of Freund asking if there is an algorithm for the online full feedback problem with adversarial losses that achieves \sqrt{V_T logK} regret, where V_T is the cumulative variance across experts. The main claim of the paper is negative answers on both open problems.