The Pareto Frontier of model selection for general Contextual Bandits

1 Introduction . Contextual multi-armed bandits are a fundamental problem in online learning [ Auer et al. , 2002 , Langford and Zhang , 2007 , Chu et al. , 2011 , Abbasi-Yadkori et al. , 2011 ] . The contextual bandit problem proceeds as a repeated game between a learner and an adversary . At every round of the game the adversary prepares a pair of a context and a loss over an action space , the learner observes the context and selects an action from the action space and then observes only the loss of the selected action . The goal of the learner is to minimize their cumulative loss . The performance measure , known as regret , is the difference between the learner ’ s cumulative loss and the smallest loss of a fixed policy , belonging to an apriori determined policy class , mapping contexts to actions . Given a single contextual bandit instance with finite sized policy class , the well-known Exp4 algorithm [ Auer et al. , 2002 ] achieves the optimal regret bound of O ( p KT ln ( |⇧| ) . Regret guarantees degrade with the complexity of the policy class , therefore a a learner might want to leverage “ guesses ” about the optimal policy . Given policy classes ⇧1 ⇢ · · · ⇢ ⇧M , a learner would ideally suffer regret scaling only with the complexity of ⇧m⇤ , the smallest policy class containing the optimal policy ⇡⇤ . While these kind of results are obtainable in full-information games , in which the learner gets to observe the loss for all actions , they are impossible for multi-armed bandits [ Lattimore , 2015 ] . In some aspects , contextual bandits are an intermediate setting between full-information and multi-armed bandits and it is unknown if model selection is possible . Foster et al . [ 2020b ] stated model selection in contextual bandits as a relevant open problem in COLT2020 . Any positive result for model selection in contextual bandits would imply a general way to treat multi-armed bandits with a switching baseline . Furthermore any negative result is conjectured to implicate negative results on another unresolved open problem on second order bounds for full-information games [ Freund , 2016 ] . In this paper , we give a fairly complete answer to the questions above . ⇤Author was at Johns Hopkins University during part of this work . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . P.1 We provide a Pareto frontier of upper bounds for model selection in contextual bandits with finite sized policy classes . P.2 We present matching lower bounds that shows that our upper bounds are tight , thereby resolve the motivating open problems [ Foster et al. , 2020b ] . P.3 We present a novel impossibility result for adapting to the number of switch points under adaptive adversaries [ Besbes et al. , 2014 ] . P.4 We negatively resolve an open problem on second order bounds for full-information [ Freund , 2016 ] . Related work . A problem closely related to contextual bandits with finite policy classes are linear contextual bandits . Model selection in linear contextual bandit problems has recently received significant attention , however none of these resuls transfer to the finite policy case . In the linear bandits problem the m-th policy class is a subset of Rdm and the losses ` t , ⇡ ( x ) , ⇡ 2 ⇧m are linear , that is ` t , ⇡ ( x ) = h✓m , m ( x , ⇡ ( x ) ) i+ ⇠ . Here m : X ⇥A ! Rdm is a feature embedding mapping from context-action pairs into Rdm , ⇠ is mean-zero sub-Gaussian noise with variance proxy equal to one and ✓m 2 Rdm is an unknown parameter . Foster et al . [ 2019 ] assume the contexts are also drawn from an unknown distribution x ⇠ D and propose an algorithm which does not incur more than Õ ( 1 3 ( i ⇤ T ) 2/3 ( Mdi⇤ ) 1/3 ) , where 3 is the smallest eigenvalue of the covariance matrix of feature embeddings ⌃ = Ex⇠D ⇥ 1 M P a2A M ( x , a ) M ( x , a ) > ⇤ . Pacchiano et al . [ 2020b ] propose a different approach based on the corralling algorithm of Agarwal et al . [ 2017 ] which enjoys a Õ ( di⇤ p T ) regret bound for finite action sets and Õ ( d2i⇤ p T ) bound for arbitrary action sets A . Later , Pacchiano et al . [ 2020a ] design an algorithm which enjoys a gap-dependent guarantee under the assumption that all of the miss-specified models have regret Ri ( t ) t , 8t 2 [ T ] . Under such an assumption , the authors recover a regret bounds of the order Õ ( di⇤ p T + d4i⇤/ ) for arbitrary action sets . Cutkosky et al . [ 2020 ] also manage to recover the O ( di⇤ p T ) and O ( d2i⇤ p T ) bounds for the model selection problems through their corralling algorithm . Ghosh et al . [ 2021 ] propose an algorithm which enjoys Õ ⇣ d2M 4.65 + p dm⇤T ⌘ in the finite arm setting , where = min { |✓m⇤ , i| : |✓m⇤ , i| > 0 } is the smallest , in absolute value , entry of ✓m⇤ . Their algorithm also enjoys a similar guarantee for arbitrary action sets with p dm⇤T replaced by dm⇤ p T . Zhu and Nowak [ 2021 ] show that it is impossible to achieve the desired regret guarantees of p dm⇤T without additional assumptions by showing a result similar to the one of Lattimore [ 2015 ] . The work of Lattimore [ 2015 ] states that in the stochastic multi-armed bandit problem it is impossible to achieve p T regret to a fixed arm , without suffering at least K p T regret to a different arm . Chatterji et al . [ 2020 ] study the problem of selecting between an algorithm for the linear contextual bandit problem and the simple stochastic multi-armed bandit problem , that is they aim to achieve simultaneously a regret guarantee which is instance-dependent optimal for the stochastic multi-armed bandit problem and optimal for the finite arm stochastic linear bandit problem . The proposed results only hold under additional assumptions . More generally , the study of the corralling problem , in which we are presented with multiple bandit algorithms and would like to perform as well as the best one , was initiated by Agarwal et al . [ 2017 ] . Other works which fall into the corralling framework are that of Foster et al . [ 2020a ] who study the miss-specified linear contextual bandit problem , that is the observed losses are linear up to some unknown ✏ miss-specification , and the work of Arora et al . [ 2021 ] who study the corralling problem for multi-armed stochastic bandit algorithms . Our work also shows an impossibility result for the stochastic bandit problem with non-stationary rewards . Auer [ 2002 ] first investigates the problem under the assumption that there are L distributional changes throughout the game and gives an algorithm with a Õ ( p KLT ) dynamic regret2 bound , under the assumption that L is known . Auer et al . [ 2019 ] achieves similar regret guarantees without assuming that the number if switches ( or changes ) of the distribution is known . A different measurement of switches is the total variation of changes in distribution VT = PT t=2 kE [ ` t ] E [ ` t 1 ] k1 . Multiple works give dynamic regret bounds of the order Õ ( V 1/3T T 2/3 ) ( hiding dependence on the size of the policy class ) when VT is known , including for extensions of the multi-armed bandit problem like 2In dynamic regret the comparator is the best action for the current distribution . contextual bandits and linear contextual bandits [ Besbes et al. , 2014 , Luo et al. , 2018 , Besbes et al. , 2015 , Wei et al. , 2017 ] . Cheung et al . [ 2019 ] , Zhao et al . [ 2020 ] further show algorithms which enjoy a parameter free regret bound of the order Õ ( V 1/4T T 3/4 ) ( hiding dependence on dimensionality ) for the linear bandits problem . The lower bound in Table 1 might seem to contradict such results . In Section 5.1 we carefully explain why this is not the case . Finally , our lower bounds apply to the problem of devising an algorithm which simultaneously enjoys a second order bound over any fraction of experts . Cesa-Bianchi et al . [ 2007 ] first investigate the problem of second order bounds for the experts problem , in which the proposed algorithm maintains a distribution pt over the set of K experts , during every round of the game . The experts are assumed to have stochastic losses ` t and the work shows an algorithm with Õ ( qPT t=1 Vi⇠pt [ ` t , i ] logK ) regret guarantee . Chaudhuri et al . [ 2009 ] , Chernov and Vovk [ 2010 ] , Luo and Schapire [ 2015 ] , Koolen and Van Erven [ 2015 ] study a different experts problem in which the comparator class for the regret changes from the best expert in hindsight to the uniform distribution over the best b✏Kc experts for an arbitrary positive ✏ . The above works propose algorithms which achieve a Õ ( p T log ( 1/✏ ) ) regret bound for all ✏ simultaneously . Freund [ 2016 ] asks if there exists an algorithm which enjoys both guarantees at the same time , that is , does there exist an algorithm with regret bound Õ ( qPT t=1 Vi⇠pt [ ` t , i ] log ( 1/✏ ) ) which holds simultaneously for all positive ✏ . Notation . For any N 2 N , [ N ] denotes the set { 1 , . . . , N } . Õ notation hides poly-logarithmic factors in the horizon T and the number of arms K but not in the size of the policy classes |⇧m| . 2 Problem setting . We consider the contextual bandit problem with general policy classes of finite size . There are K arms and nested policy classes ( ⇧m ) Mm=1 , where a policy ⇡ 2 ⇧m , ⇡ : X ! [ K ] is a mapping from an arbitrary context space X into the set of K arms . The game is played for T rounds and at any time t , the agent observes a context xt 2 X , selects arm At 2 [ K ] and observes the loss ` t , At from an otherwise unobserved loss vector ` t 2 [ K ] . We measure an algorithm ’ s performance in terms of pseudo-regret , which is the expected cumulative regret of the player against following a fixed policy in hindsight Reg ( T , ⇧ ) = max ⇡2⇧ E `` TX t=1 ` t , At ` t , ⇡ ( xt ) # . Environments . We distinguish between stochastic environments and oblivious or adaptive adversaries . In stochastic environments , there are unknown distribution PX , Q such that xt ⇠ PX and ` t ⇠ Q ( ·|xt ) are i.i.d . samples . In the adversarial regime , the distributions can change over time , i.e . xt ⇠ PX , t , ` t ⇠ Qt ( ·|xt ) . When the choices are fixed at the beginning of the game , the adversary is called oblivious , while an adaptive adversary can chose PX , t , Qt based on all observations up to time t 1 . Often the stochastic-adversarial hybrid problem has been studied with adversarially chosen context but stochastic losses . In our work , all upper bounds hold in the stronger notion where both the losses and the contexts are adaptive , while the lower bounds hold for the weaker notion where only the contexts are adaptive . Open problem [ Foster et al. , 2020b ] . The regret upper bounds for all regimes introduced above for a fixed policy class ⇧ of finite size are of the order Õ ( p ln ( |⇧| ) KT ) and can be achieved by the Exp4 algorithm [ Auer et al. , 2002 ] . The question asked by Foster et al . [ 2020b ] : For a nested sequence of policies ⇧1 ⇢ ⇧2 ⇢ · · · ⇢ ⇧M , is there a universal ↵ 2 [ 12 , 1 ) such that a regret bound of Reg ( T , ⇧m ) = PolyLog ( K , M ) Õ ln ( |⇧m| ) 1 ↵ T ↵ ( 1 ) is obtainable for all m 2 [ M ] simultaneously ? W.l.o.g . we can assume that M = O ( ln ln ( |⇧M | ) ) = O ( ln ( T ) ) . Otherwise we take a subset of policy classes that includes ⇧M and where two consequent policy classes at least square in size . Due to nestedness , any guarantees on this subset of models imply up to constants the same bounds on the full set . S-switch A motivating example for studying nested policy classes is the S-switch problem . The context is simply xt = t 2 [ T ] and the set of policies is given by ⇧S = ( ⇡ T 1X t=1 I { ⇡t 6= ⇡t+1 } S ) , the set of policies that changes its action not more than S many times . Any positive result for contextual bandits with finite sized policy classes would provide algorithms that adapt to the number of switch points , since ln |⇧S | = Õ ( S ) . To make clear what problem we are considering , we are using RegSW ( T , S ) to denote the regret in the switching problem . Next , we define the class of proper algorithms which choose their policy at every time step t independently of context xt . Restricting our attention to such algorithms greatly reduces the technicalities for lower bound proofs in the non-adaptive regimes . The lower bound for this class of algorithms is also at the core of the argument for adaptive ( improper ) algorithms in stochastic environments . Definition 1 . We call an algorithm proper , if at any time t , the algorithm follows the recommendation of a policy ⇡it 2 ⇧M , and if the choice of it by the algorithm , is independent of the context xt . Example . EXP4 is proper . The properness assumption intuitively allows us to reduce the model selection problem to a bandit-like problem in the space of all policies ⇧M . We give more details in Section 4.2 and Appendix B.3 .

Summary: The paper studies a COLT 2020 open problem on model selection for contextual bandits (Foster et al. 2020b). The paper presents strong lower bounds which are claimed to hold for all algorithms in the adaptive adversarial regime and hold for "proper" algorithms in the stochastic regime. If the lower bounds are true, then they would lead to the following consequences: 1.  The paper resolves the open problem of Foster et al. (2020b) by providing a negative answer.  2.  As a byproduct, the paper resolves a COLT 2016 open problem on second-order bounds for full-information online learning (Freund 2016).  For the model selection problem, the paper also provides a matching upper bound, thus gives a complete characterization of the pareto frontier of model selection for contextual bandits. The paper also discusses the implications to the $S$-switch non-stationary bandit problem.