Efficient First-Order Contextual Bandits: Prediction, Allocation, and Triangular Discrimination

1 Introduction . In the contextual bandit problem , a learning agent repeatedly makes decisions based on contextual information , with the goal of learning a decision-making policy that minimizes their total loss over time . This model captures simple reinforcement learning tasks in which the agent must learn to make high-quality decisions in an uncertain environment , but does not need to engage in long-term planning or credit assignment . Owing to the availability of high-quality engineered reward metrics , contextual bandit algorithms are now routinely deployed in production for online personalization systems [ 4 , 61 ] . Contextual bandits encompass both the general problem of statistical learning with function approximation ( specifically , cost-sensitive classification ) and the classical multi-armed bandit problem , yet present algorithmic challenges greater than the sum of both parts . In spite of these difficulties , extensive research effort over the past decade has resulted in efficient , general-purpose algorithms , as well as a sharp understanding of the optimal worst-case sample complexity [ 9 , 12 , 3 , 29 , 58 ] . While the algorithmic and statistical foundations for contextual bandits are beginning to take shape , we still lack an understanding of adaptive or data-dependent algorithms that can go beyond the worst case and exploit nice properties of real-world instances for better performance . This is in stark contrast to supervised statistical learning , where adaptivity has substantial theory , and where standard 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . algorithms ( e.g. , empirical risk minimization ) are known to automatically adapt to nice data [ 17 ] . For contextual bandits , adaptivity poses new challenges that seem to require algorithmic innovation , and a major research frontier is to develop algorithmic principles for adaptivity and an understanding of the fundamental limits . To highlight the lack of understanding for adaptive and data-dependent algorithms , a COLT 2017 open problem posed by Agarwal , Krishnamurthy , Langford , Luo , and Schapire [ 5 ] asks whether there exist contextual bandit algorithms that achieve a certain data-dependent first-order regret bound , which scales with the cumulative loss L ? of the best policy , rather than with the time horizon T . For multiarmed bandits , first-order regret bounds ( also known as small-loss bounds or fast rates ) typically scale as √ L ? and imply faster convergence for “ easy ” problems , interpolating between the optimal √ T rate for worst-case instances and constant/logarithmic regret for noise-free instances [ 7 , 31 ] . Agarwal et al . [ 5 ] observed that existing techniques appear to be inadequate to achieve this type of guarantee in contextual bandits . Beyond simply asking whether first-order regret can be achieved , they also asked whether it can be achieved efficiently , which is essential for real-world deployment . Subsequently , Allen-Zhu , Bubeck , and Li [ 6 ] gave an inefficient algorithm with an optimal first-order regret guarantee , resolving the former question , but the existence of efficient first-order algorithms remained open . Contributions . We give the first optimal and efficient contextual bandit algorithm with a first-order regret guarantee , providing a resolution to the second open problem raised by Agarwal et al . [ 5 ] . Our algorithm , FastCB , builds on a recent line of research that develops efficient contextual bandit algorithms based on the computational primitive of ( online/offline ) supervised regression [ 43 , 32 , 29 , 58 ] , and is efficient in terms of queries to an online oracle for regression with the logarithmic loss . Beyond attaining first-order regret , FastCB inherits all of the benefits of recent algorithms based on regression : it is simple and practical , accommodates flexible function classes , requires no statistical assumptions beyond realizability , and enjoys strong empirical performance . Technical highlights . By invoking the framework of regression oracles , our algorithm design approach deviates sharply from prior approaches to first-order regret and necessitates the use of techniques that are novel even in the context of statistical learning . At a high-level , the design of FastCB leverages two key techniques : 1 . First-order regret for plug-in classification via logarithmic loss : We show that algorithms based on regression with least-squares , as used in prior work [ 29 , 58 , 68 , 34 , 23 ] , fail to attain first-order regret , even for the simpler problem of cost-sensitive classification in statistical learning . In spite of this apparent setback , we show that regression with the logarithmic loss does lead to first-order regret for statistical learning . This is established through a new analysis based on an information-theoretic quantity called the triangular discrimination [ 66 , 44 , 62 ] . 2 . Reweighted inverse gap weighting : Moving from statistical learning to contextual bandits , we transform predictions into distributions over actions using a scale-sensitive refinement to the inverse-gap weighting scheme used in the SquareCB algorithm [ 1 , 29 ] . Our new scheme is tailored to small losses , and we show that its error is controlled by the triangular discrimination . Summarizing , our approach leverages prediction via the logarithmic loss , allocation via reweighted inverse gap weighting , and triangular discrimination as the bridge from prediction to allocation . Empirical results . In Section 5 , we evaluate FastCB on the large-scale contextual bandit benchmark of Bietti et al . [ 13 ] and find that it typically outperforms SquareCB and other non-adaptive baselines [ 35 ] . Interestingly , we observe that most of the performance improvement can be attributed to the use of the logarithmic loss , while the reweighted allocation scheme provides modest additional benefit . These findings raise a natural question as to whether simply moving to the logarithmic loss can yield performance improvements in production contextual bandit deployments . On the regression oracle model . As a disclaimer , we caution that our algorithm is efficient in terms of an oracle for online regression , while Agarwal et al . [ 5 ] originally asked for an algorithm that is efficient in terms of a cost-sensitive classification oracle capable of solving the policy optimization problem argminπ∈Π ∑T t=1 ` t ( π ( xt ) ) . Hence , while FastCB is the first algorithm with first-order regret that is efficient in any oracle model , it does not formally solve the original open problem . Nonetheless , there are strong reasons to prefer a solution based on regression over one based on classification . First , cost-sensitive classification is intractable to implement even for simple function classes for which regression can be solved efficiently [ 29 ] . Setting this issue aside , ( online ) Algorithm 1 FastCB ( “ Fast Rates for Contextual Bandits ” ) 1 : parameters : Learning rate γ > 0 . Online regression oracle AlgKL . 2 : for t = 1 , . . . , T do 3 : Receive context xt . // Compute oracle ’ s predictions ( Eq . ( 4 ) ) . 4 : For each action a ∈ A , compute ŷt ( xt , a ) : = Alg ( t ) KL ( xt , a ; { ( xi , ai , ` i ( ai ) ) } t−1 i=1 ) . 5 : Let bt ∈ argmina∈A ŷt , a . // Reweighted inverse gap weighting . 6 : For each a 6= bt , define pt , a = ŷt ( xt , bt ) Aŷt ( xt , bt ) +γ ( ŷt ( xt , a ) −ŷt ( xt , bt ) ) . Let pt , bt = 1− ∑ a6=bt pt , a . 7 : Sample at ∼ pt and observe loss ` t ( at ) . 8 : Update AlgKL with example ( xt , at , ` t ( at ) ) . 9 : end for regression-based algorithms are typically simpler and faster than classification-based algorithms , and multiple empirical evaluations have shown that algorithms based on regression dominate those based on classification [ 32 , 13 , 35 ] . Organization . Section 2 contains our algorithm and main theorem . Section 3 describes the motivation and analysis ideas behind FastCB , beginning from new techniques for statistical learning with regression-based classifiers . Examples for the main theorem are given in Section 4 , and experimental results are given in Section 5 . Detailed discussion of related work is deferred to Appendix A . 2 Main Result : An Efficient First-Order Algorithm for Contextual Bandits . We begin by formally introducing the contextual bandit model . At each round t ∈ [ T ] , the learner observes a context xt ∈ X , selects an action at ∈ A , then observes a loss ` t ( at ) ∈ [ 0 , 1 ] for the action they selected . We assume that A : = |A| is finite and that each loss function ` t : A → [ 0 , 1 ] is drawn independently from a fixed distribution P ` t ( · | xt ) , where P ` 1 , . . . , P ` T and x1 , . . . , xT are selected by a potentially adaptive adversary . We make a standard realizability assumption [ 24 , 2 , 32 , 29 ] . Namely , we assume that the learner has access to a class of value functions F ⊂ ( X ×A → [ 0 , 1 ] ) ( e.g. , neural networks , kernels , or forests ) that models the mean of the loss distribution . Assumption 1 ( Realizability ) . There exists f ? ∈ F such that for all t , f ? ( x , a ) = E [ ` t ( a ) | xt = x ] . The aim of the learner is to minimize their regret to the optimal policy π ? ( x ) : = argmina∈A f ? ( x , a ) : RegCB ( T ) : = T∑ t=1 ` t ( at ) − T∑ t=1 ` t ( π ? ( xt ) ) . ( 1 ) For each f ∈ F , we let πf ( x ) : = argmina∈A f ( x , a ) be the induced policy . We let Π : = { πf | f ∈ F } be the induced policy class . Further notation . We adopt standard big-oh notation , and write f = Õ ( g ) to denote that f = O ( gmax { 1 , polylog ( g ) } ) . We use . only in informal statements to highlight the most salient elements of an inequality . We use a ∨ b = max { a , b } and a ∧ b = min { a , b } . 2.1 Algorithm and Main Result . FastCB builds on the SquareCB algorithm of Foster and Rakhlin [ 29 ] , which provides an efficient , minimax-optimal reduction from contextual bandits to online regression with the square loss . Compared to SquareCB and other subsequent algorithms based on online regression [ 34 , 23 ] , the first twist here is that rather than working with the square loss , we build on the computational primitive of online regression with the logarithmic loss . While this point is inconsequential for worst-case guarantees , we show that it is a fundamental distinction for first-order guarantees . Online regression oracles . In more detail , an online regression oracle , which we denote by AlgKL ( for “ Kullback-Leibler ” ) operates in the following protocol : For each time t , the algorithm receives a context-action pair ( xt , at ) , produces a prediction ŷt ∈ [ 0 , 1 ] , then receives a response yt . The algorithm ’ s prediction error is measured through the binary logarithmic/cross-entropy loss , ` log ( ŷ , y ) : = y log ( 1/ŷ ) + ( 1− y ) log ( 1/ ( 1− ŷ ) ) . ( 2 ) The algorithm ’ s goal is to ensure that the log loss regret to F is minimized for all sequences . Assumption 2 . The algorithm AlgKL guarantees that for every ( possibly adaptively chosen ) sequence x1 : T , a1 : T , y1 : T , the log loss regret is bounded by a known function RegKL ( T ) : T∑ t=1 ` log ( ŷt , yt ) − inf f∈F T∑ t=1 ` log ( f ( xt , at ) , yt ) ≤ RegKL ( T ) , ( 3 ) Online regression with the logarithmic loss ( or , sequential probability assignment ) is a fundamental and well-studied problem in online learning , and there are efficient algorithms available for many function classes of interest [ 25 , 67 , 40 , 37 , 52 , 55 , 33 , 48 ] ; see Section 4 for examples . While log loss regret is a more stringent notion of performance than square loss regret , it nonetheless has a relatively mature theory characterizing optimal rates [ 57 , 51 , 20 , 14 ] . The algorithm . FastCB ( Algorithm 1 ) is a reduction that efficiently transforms any online regression oracle satisfying Assumption 2 into a contextual bandit algorithm with an optimal first-order regret bound . At each round t , the algorithm first computes the estimated loss ŷt ( xt , a ) : = Alg ( t ) KL ( xt , a ; { ( xi , ai , ` i ( ai ) ) } t−1 i=1 ) ( 4 ) predicted by the regression oracle for each action a ( Line 4 ) ; see Appendix C.1 for a more detailed formal description of the oracle model . Next , FastCB uses these estimates to assign a probability of being played to each action a via a scale-sensitive refinement to the inverse gap weighting strategy used in SquareCB [ 1 , 29 ] , which we call reweighted inverse gap weighting ( Line 6 ) . Letting bt : = argmina∈A ŷt ( xt , a ) be the greedy action according to the predicted losses , we define pt , a : = ŷt ( xt , bt ) Aŷt ( xt , bt ) + γ ( ŷt ( xt , a ) − ŷt ( xt , bt ) ) ∀a 6= bt , and pt , bt : = 1− ∑ a6=bt pt , a , ( 5 ) where γ > 0 is a learning rate parameter . Given this distribution , FastCB simply samples at ∼ pt , then updates the oracle with the resulting tuple ( xt , at , ` t ( at ) ) . Our main theorem shows that this leads to an optimal first-order regret bound.1 Theorem 1 ( Main theorem ) . Suppose Assumptions 1 and 2 hold . Then Algorithm 1 guarantees that for all sequences with E [ ∑T t=1 ` t ( π ? ( xt ) ) ] ≤ L ? , by choosing γ = √ AL ? /3RegKL ( T ) ∨ 10A , E [ RegCB ( T ) ] ≤ 40 √ L ? ·ARegKL ( T ) + 600ARegKL ( T ) . ( 6 ) The dominant term in this regret bound scales with √ L ? whenever the oracle AlgKL attains a fast log ( T ) -type regret bound . As a simple example , whenever F is finite , we can instantiate AlgKL so that RegKL ( T ) ≤ log|F| [ 67 ] , whereby FastCB enjoys optimal [ 2 ] first-order regret : E [ RegCB ( T ) ] ≤ O ( √ L ? ·A log|F|+A log|F| ) . Beyond first-order regret , FastCB inherits all of the advantages of online regression-based algorithms : • Efficiency and simplicity . The memory and runtime used by the algorithm—on top of what is required by the regression oracle—scales only as O ( A ) per step ; implementation is trivial . • Flexibility . Working with regression as a primitive means that the algorithm easily accomodates rich , potentially nonparametric function classes , and we can instantiate Theorem 1 to get provable end-to-end regret guarantees for concrete classes of interest . For example , for linear models in Rd we can efficiently attain RegKL ( T ) ≤ O ( d log ( T ) ) [ 25 , 40 ] , which yields a first-order regret bound RegCB ( T ) . √ L ? ·Ad ; our result is new even for this simple special case . Similar guarantees are available for kernels , generalized linear models , and many other nonparametric classes . On the other hand , even for function classes where provable algorithms are not available , regression is amenable to practical heuristics ( e.g. , gradient descent ) . See Section 4 for detailed examples . 1While we assume that an upper bound on the optimal loss is known for simplicity , one can extend to the unknown case by running the algorithm in epochs , setting γ in terms of the algorithm ’ s estimated loss Lt = ∑t τ=1 ` τ ( aτ ) , and applying the doubling trick . Theorem 1 also readily extends to high probability .

Summary: The authors establish first-order regret guarantees for contextual bandits, that is, the $\sqrt{T}$ dependence on the horizon is replaced by the total loss incurred by the optimal policy. This requires the algorithm to adapt to "low noise". The proposed algorithm, called FastCB, is a variant of SquareCB by Foster et al, which is based on a reduction to online regression. These kind of algorithms have two components: an online "regression oracle" that learns to predict rewards, and an allocation rule that selects actions based on these predictions. The main innovation w.r.t. SquareCB is replacing the loss of the regression oracle, from square to logarithmic. The allocation rule is also modified to exploit this change. The authors derive first-order regret bound for FastCB. The change of loss is further motivated by analyzing the off-line cost-sensitive classification setting, which could be of independent interest. The authors show that a plug-in classifier optimizing the squared loss cannot achieve first-order guarantees in general, while the latter is possible by using the logarithmic loss. The argument is two steps: first the excess risk is bounded by the triangular discrimination between true and predicted loss. Then they show that, by minimizing the logarithmic loss, the algorithm also makes the triangular discrimination small. The authors conjecture that triangular discrimination could be a useful tool for studying plug-in classification and contextual bandits in general. Extensive experiments show that FastCB improves over SquareCB. Ablation shows that the gain is mostly due to the change of loss.