Efficient First-Order Contextual Bandits: Prediction, Allocation, and Triangular Discrimination

1 Introduction . In the contextual bandit problem , a learning agent repeatedly makes decisions based on contextual information , with the goal of learning a decision-making policy that minimizes their total loss over time . This model captures simple reinforcement learning tasks in which the agent must learn to make high-quality decisions in an uncertain environment , but does not need to engage in long-term planning or credit assignment . Owing to the availability of high-quality engineered reward metrics , contextual bandit algorithms are now routinely deployed in production for online personalization systems [ 4 , 61 ] . Contextual bandits encompass both the general problem of statistical learning with function approximation ( specifically , cost-sensitive classification ) and the classical multi-armed bandit problem , yet present algorithmic challenges greater than the sum of both parts . In spite of these difficulties , extensive research effort over the past decade has resulted in efficient , general-purpose algorithms , as well as a sharp understanding of the optimal worst-case sample complexity [ 9 , 12 , 3 , 29 , 58 ] . While the algorithmic and statistical foundations for contextual bandits are beginning to take shape , we still lack an understanding of adaptive or data-dependent algorithms that can go beyond the worst case and exploit nice properties of real-world instances for better performance . This is in stark contrast to supervised statistical learning , where adaptivity has substantial theory , and where standard 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . algorithms ( e.g. , empirical risk minimization ) are known to automatically adapt to nice data [ 17 ] . For contextual bandits , adaptivity poses new challenges that seem to require algorithmic innovation , and a major research frontier is to develop algorithmic principles for adaptivity and an understanding of the fundamental limits . To highlight the lack of understanding for adaptive and data-dependent algorithms , a COLT 2017 open problem posed by Agarwal , Krishnamurthy , Langford , Luo , and Schapire [ 5 ] asks whether there exist contextual bandit algorithms that achieve a certain data-dependent first-order regret bound , which scales with the cumulative loss L ? of the best policy , rather than with the time horizon T . For multiarmed bandits , first-order regret bounds ( also known as small-loss bounds or fast rates ) typically scale as √ L ? and imply faster convergence for “ easy ” problems , interpolating between the optimal √ T rate for worst-case instances and constant/logarithmic regret for noise-free instances [ 7 , 31 ] . Agarwal et al . [ 5 ] observed that existing techniques appear to be inadequate to achieve this type of guarantee in contextual bandits . Beyond simply asking whether first-order regret can be achieved , they also asked whether it can be achieved efficiently , which is essential for real-world deployment . Subsequently , Allen-Zhu , Bubeck , and Li [ 6 ] gave an inefficient algorithm with an optimal first-order regret guarantee , resolving the former question , but the existence of efficient first-order algorithms remained open . Contributions . We give the first optimal and efficient contextual bandit algorithm with a first-order regret guarantee , providing a resolution to the second open problem raised by Agarwal et al . [ 5 ] . Our algorithm , FastCB , builds on a recent line of research that develops efficient contextual bandit algorithms based on the computational primitive of ( online/offline ) supervised regression [ 43 , 32 , 29 , 58 ] , and is efficient in terms of queries to an online oracle for regression with the logarithmic loss . Beyond attaining first-order regret , FastCB inherits all of the benefits of recent algorithms based on regression : it is simple and practical , accommodates flexible function classes , requires no statistical assumptions beyond realizability , and enjoys strong empirical performance . Technical highlights . By invoking the framework of regression oracles , our algorithm design approach deviates sharply from prior approaches to first-order regret and necessitates the use of techniques that are novel even in the context of statistical learning . At a high-level , the design of FastCB leverages two key techniques : 1 . First-order regret for plug-in classification via logarithmic loss : We show that algorithms based on regression with least-squares , as used in prior work [ 29 , 58 , 68 , 34 , 23 ] , fail to attain first-order regret , even for the simpler problem of cost-sensitive classification in statistical learning . In spite of this apparent setback , we show that regression with the logarithmic loss does lead to first-order regret for statistical learning . This is established through a new analysis based on an information-theoretic quantity called the triangular discrimination [ 66 , 44 , 62 ] . 2 . Reweighted inverse gap weighting : Moving from statistical learning to contextual bandits , we transform predictions into distributions over actions using a scale-sensitive refinement to the inverse-gap weighting scheme used in the SquareCB algorithm [ 1 , 29 ] . Our new scheme is tailored to small losses , and we show that its error is controlled by the triangular discrimination . Summarizing , our approach leverages prediction via the logarithmic loss , allocation via reweighted inverse gap weighting , and triangular discrimination as the bridge from prediction to allocation . Empirical results . In Section 5 , we evaluate FastCB on the large-scale contextual bandit benchmark of Bietti et al . [ 13 ] and find that it typically outperforms SquareCB and other non-adaptive baselines [ 35 ] . Interestingly , we observe that most of the performance improvement can be attributed to the use of the logarithmic loss , while the reweighted allocation scheme provides modest additional benefit . These findings raise a natural question as to whether simply moving to the logarithmic loss can yield performance improvements in production contextual bandit deployments . On the regression oracle model . As a disclaimer , we caution that our algorithm is efficient in terms of an oracle for online regression , while Agarwal et al . [ 5 ] originally asked for an algorithm that is efficient in terms of a cost-sensitive classification oracle capable of solving the policy optimization problem argminπ∈Π ∑T t=1 ` t ( π ( xt ) ) . Hence , while FastCB is the first algorithm with first-order regret that is efficient in any oracle model , it does not formally solve the original open problem . Nonetheless , there are strong reasons to prefer a solution based on regression over one based on classification . First , cost-sensitive classification is intractable to implement even for simple function classes for which regression can be solved efficiently [ 29 ] . Setting this issue aside , ( online ) Algorithm 1 FastCB ( “ Fast Rates for Contextual Bandits ” ) 1 : parameters : Learning rate γ > 0 . Online regression oracle AlgKL . 2 : for t = 1 , . . . , T do 3 : Receive context xt . // Compute oracle ’ s predictions ( Eq . ( 4 ) ) . 4 : For each action a ∈ A , compute ŷt ( xt , a ) : = Alg ( t ) KL ( xt , a ; { ( xi , ai , ` i ( ai ) ) } t−1 i=1 ) . 5 : Let bt ∈ argmina∈A ŷt , a . // Reweighted inverse gap weighting . 6 : For each a 6= bt , define pt , a = ŷt ( xt , bt ) Aŷt ( xt , bt ) +γ ( ŷt ( xt , a ) −ŷt ( xt , bt ) ) . Let pt , bt = 1− ∑ a6=bt pt , a . 7 : Sample at ∼ pt and observe loss ` t ( at ) . 8 : Update AlgKL with example ( xt , at , ` t ( at ) ) . 9 : end for regression-based algorithms are typically simpler and faster than classification-based algorithms , and multiple empirical evaluations have shown that algorithms based on regression dominate those based on classification [ 32 , 13 , 35 ] . Organization . Section 2 contains our algorithm and main theorem . Section 3 describes the motivation and analysis ideas behind FastCB , beginning from new techniques for statistical learning with regression-based classifiers . Examples for the main theorem are given in Section 4 , and experimental results are given in Section 5 . Detailed discussion of related work is deferred to Appendix A . 2 Main Result : An Efficient First-Order Algorithm for Contextual Bandits . We begin by formally introducing the contextual bandit model . At each round t ∈ [ T ] , the learner observes a context xt ∈ X , selects an action at ∈ A , then observes a loss ` t ( at ) ∈ [ 0 , 1 ] for the action they selected . We assume that A : = |A| is finite and that each loss function ` t : A → [ 0 , 1 ] is drawn independently from a fixed distribution P ` t ( · | xt ) , where P ` 1 , . . . , P ` T and x1 , . . . , xT are selected by a potentially adaptive adversary . We make a standard realizability assumption [ 24 , 2 , 32 , 29 ] . Namely , we assume that the learner has access to a class of value functions F ⊂ ( X ×A → [ 0 , 1 ] ) ( e.g. , neural networks , kernels , or forests ) that models the mean of the loss distribution . Assumption 1 ( Realizability ) . There exists f ? ∈ F such that for all t , f ? ( x , a ) = E [ ` t ( a ) | xt = x ] . The aim of the learner is to minimize their regret to the optimal policy π ? ( x ) : = argmina∈A f ? ( x , a ) : RegCB ( T ) : = T∑ t=1 ` t ( at ) − T∑ t=1 ` t ( π ? ( xt ) ) . ( 1 ) For each f ∈ F , we let πf ( x ) : = argmina∈A f ( x , a ) be the induced policy . We let Π : = { πf | f ∈ F } be the induced policy class . Further notation . We adopt standard big-oh notation , and write f = Õ ( g ) to denote that f = O ( gmax { 1 , polylog ( g ) } ) . We use . only in informal statements to highlight the most salient elements of an inequality . We use a ∨ b = max { a , b } and a ∧ b = min { a , b } . 2.1 Algorithm and Main Result . FastCB builds on the SquareCB algorithm of Foster and Rakhlin [ 29 ] , which provides an efficient , minimax-optimal reduction from contextual bandits to online regression with the square loss . Compared to SquareCB and other subsequent algorithms based on online regression [ 34 , 23 ] , the first twist here is that rather than working with the square loss , we build on the computational primitive of online regression with the logarithmic loss . While this point is inconsequential for worst-case guarantees , we show that it is a fundamental distinction for first-order guarantees . Online regression oracles . In more detail , an online regression oracle , which we denote by AlgKL ( for “ Kullback-Leibler ” ) operates in the following protocol : For each time t , the algorithm receives a context-action pair ( xt , at ) , produces a prediction ŷt ∈ [ 0 , 1 ] , then receives a response yt . The algorithm ’ s prediction error is measured through the binary logarithmic/cross-entropy loss , ` log ( ŷ , y ) : = y log ( 1/ŷ ) + ( 1− y ) log ( 1/ ( 1− ŷ ) ) . ( 2 ) The algorithm ’ s goal is to ensure that the log loss regret to F is minimized for all sequences . Assumption 2 . The algorithm AlgKL guarantees that for every ( possibly adaptively chosen ) sequence x1 : T , a1 : T , y1 : T , the log loss regret is bounded by a known function RegKL ( T ) : T∑ t=1 ` log ( ŷt , yt ) − inf f∈F T∑ t=1 ` log ( f ( xt , at ) , yt ) ≤ RegKL ( T ) , ( 3 ) Online regression with the logarithmic loss ( or , sequential probability assignment ) is a fundamental and well-studied problem in online learning , and there are efficient algorithms available for many function classes of interest [ 25 , 67 , 40 , 37 , 52 , 55 , 33 , 48 ] ; see Section 4 for examples . While log loss regret is a more stringent notion of performance than square loss regret , it nonetheless has a relatively mature theory characterizing optimal rates [ 57 , 51 , 20 , 14 ] . The algorithm . FastCB ( Algorithm 1 ) is a reduction that efficiently transforms any online regression oracle satisfying Assumption 2 into a contextual bandit algorithm with an optimal first-order regret bound . At each round t , the algorithm first computes the estimated loss ŷt ( xt , a ) : = Alg ( t ) KL ( xt , a ; { ( xi , ai , ` i ( ai ) ) } t−1 i=1 ) ( 4 ) predicted by the regression oracle for each action a ( Line 4 ) ; see Appendix C.1 for a more detailed formal description of the oracle model . Next , FastCB uses these estimates to assign a probability of being played to each action a via a scale-sensitive refinement to the inverse gap weighting strategy used in SquareCB [ 1 , 29 ] , which we call reweighted inverse gap weighting ( Line 6 ) . Letting bt : = argmina∈A ŷt ( xt , a ) be the greedy action according to the predicted losses , we define pt , a : = ŷt ( xt , bt ) Aŷt ( xt , bt ) + γ ( ŷt ( xt , a ) − ŷt ( xt , bt ) ) ∀a 6= bt , and pt , bt : = 1− ∑ a6=bt pt , a , ( 5 ) where γ > 0 is a learning rate parameter . Given this distribution , FastCB simply samples at ∼ pt , then updates the oracle with the resulting tuple ( xt , at , ` t ( at ) ) . Our main theorem shows that this leads to an optimal first-order regret bound.1 Theorem 1 ( Main theorem ) . Suppose Assumptions 1 and 2 hold . Then Algorithm 1 guarantees that for all sequences with E [ ∑T t=1 ` t ( π ? ( xt ) ) ] ≤ L ? , by choosing γ = √ AL ? /3RegKL ( T ) ∨ 10A , E [ RegCB ( T ) ] ≤ 40 √ L ? ·ARegKL ( T ) + 600ARegKL ( T ) . ( 6 ) The dominant term in this regret bound scales with √ L ? whenever the oracle AlgKL attains a fast log ( T ) -type regret bound . As a simple example , whenever F is finite , we can instantiate AlgKL so that RegKL ( T ) ≤ log|F| [ 67 ] , whereby FastCB enjoys optimal [ 2 ] first-order regret : E [ RegCB ( T ) ] ≤ O ( √ L ? ·A log|F|+A log|F| ) . Beyond first-order regret , FastCB inherits all of the advantages of online regression-based algorithms : • Efficiency and simplicity . The memory and runtime used by the algorithm—on top of what is required by the regression oracle—scales only as O ( A ) per step ; implementation is trivial . • Flexibility . Working with regression as a primitive means that the algorithm easily accomodates rich , potentially nonparametric function classes , and we can instantiate Theorem 1 to get provable end-to-end regret guarantees for concrete classes of interest . For example , for linear models in Rd we can efficiently attain RegKL ( T ) ≤ O ( d log ( T ) ) [ 25 , 40 ] , which yields a first-order regret bound RegCB ( T ) . √ L ? ·Ad ; our result is new even for this simple special case . Similar guarantees are available for kernels , generalized linear models , and many other nonparametric classes . On the other hand , even for function classes where provable algorithms are not available , regression is amenable to practical heuristics ( e.g. , gradient descent ) . See Section 4 for detailed examples . 1While we assume that an upper bound on the optimal loss is known for simplicity , one can extend to the unknown case by running the algorithm in epochs , setting γ in terms of the algorithm ’ s estimated loss Lt = ∑t τ=1 ` τ ( aτ ) , and applying the doubling trick . Theorem 1 also readily extends to high probability .

Summary: This work is about contextual bandits in the low noise regime. It introduces the first efficient algorithm with a small loss bound. The latter is close to a previous algorithm SquareCB. However, the main idea is to use an online regression oracle working with a log-loss instead of a square loss. They also show that this change is fundamental as SquareCB can not achieve small loss bound. Some experiments have also been done.