Title: An Improved Algorithm for Adversarial Linear Contextual Bandits via Reduction
Venue: NeurIPS 2025
Accepted: yes
Keywords: Online learning

contextual bandits

linear bandits
Abstract: We present an efficient algorithm for linear contextual bandits with adversarial losses and stochastic action sets. Our approach reduces this setting to misspecification-robust adversarial linear bandits with fixed action sets. Without knowledge of the context distribution or access to
a context simulator, the algorithm achieves $\tilde{O}(\min\{d^2\sqrt{T}, \sqrt{d^3T\log K}\})$ regret and runs in $\mathrm{poly}(d,C,T)$ time, where $d$ is the feature dimension, $C$ is an upper bound on the number of linear constraints defining the action set in each round, $K$ is an upper bound on the number of actions in each round, and $T$ is number of rounds.  This resolves the open question by Liu et al. (2023) on whether one can obtain $\mathrm{poly}(d)\sqrt{T}$ regret in polynomial time independent of the number of actions.  For the important class of combinatorial bandits with adversarial losses and stochastic action sets where the action sets can be described by a polynomial number of linear constraints, our algorithm is the first to achieve $\mathrm{poly}(d)\sqrt{T}$ regret in polynomial time, while no prior algorithm achieves even $o(T)$ regret in polynomial time to our knowledge. When a simulator is available, the regret bound can be improved to $\tilde{O}(d\sqrt{L^\star})$, where $L^\star$ is the cumulative loss of the best policy.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the linear contextual bandit problem, where contexts (action sets) are generated stochastically and linear losses are selected adversarially. The main contribution is an efficient algorithm that achieves a regret bound of $\\tilde O(d^2\\sqrt{T})$ for the dimensionality $d$ of action sets and time horizon $T$. The algorithm runs in polynomial time in $d$, $T$, and the number of constraints that specify the action sets, thereby allowing for efficient learning in large action spaces. Moreover, given access to the context simulator, the bound is improved to $\\tilde O(d\\sqrt{L^*})$, where $L^*$ is the cumulative loss of the best policy. The main technical idea is to reduce the problem to the adversarial linear bandit problem with misspecification. Combing this with a uniform convergence argument and the complexity analysis techniques known in linear programming leads to the desired result.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths\n1. This paper provides the first efficient $\\mathrm{poly}(d)\\sqrt{T}$-regret algorithm for the adversarial linear contextual bandit problem that can handle large action spaces, resolving the open question raised by Liu et al. (2023).\n2. The main reduction idea appears novel and interesting, which may inspire future research in this area.\n\n### Weaknesses\n1. The details of the misspecification-robust adversarial linear bandit algorithm are deferred to the appendix, and little intuition regarding the proof of Theorem 2 is provided in the main text. This makes it difficult to assess the significance of the adversarial linear bandit algorithm, particularly for readers unfamiliar with the misspecification-robust setting, such as myself.\n2. I do not consider this a major weakness, but the algorithm relies on somewhat costly operations such as the ellipsoid method. Additionally, the complexity also depends on the number of constraints, which may be large in some cases. The authors are aware of these points and have discussed them in the paper.\n\nQUESTIONS:\n1. In equation (1), how to detail with possibly multiple minimizers?\n2. In the equation after line 121, what is the rightmost expectation taken about?\n3. I could not understand the rightmost expression of $\\hat \\Omega$ after line 125. Why does it contain neither $\\pi$ nor $\\Pi$?\n4. In line 194, what is the specific amount of the misspecification $\\varepsilon$? Does it come from the uniform convergence bound?\n\nMinor comment:\n- In line 121, $y_t$ should be $a_t$ for the consistency with the subsequent equation.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 2, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-10"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper focuses on the problem of adversarial linear bandits with stochastic action set (as context). Specifically, without the knowledge of the context distribution, the authors provide an algorithm achieving $O(d^2\\sqrt{T})$ regret with runtime polynomial in the dimension $d$ and the horzion $T$ (independent of the number of action $K$ in each action set). Moreover, with the knowledge of the context distribution, the authors further show that an improved first-order bound $O(d\\sqrt{L_0})$ is achievable, where $L_0$ is the expected loss of the best policy in hindsight. The algorithm is motivated by the recent advance in reducing contextual linear bandits to non-contextual bandits by Hanna et al., 2023. However, the reduction introduced in this paper is slightly different from the one in Hanna et al., 2023 in the sense that Hanna et al., 2023 only considers the stochastic loss case.\n\nSTRENGTHS AND WEAKNESSES:\nStrength:\n- The problem is well-motivated and this paper provides the first algorithm that achieves $O(\\sqrt{T})$ regret with runtime independent of the number of action without the knowledge of the context distribution. The paper is also well-written in general.\n- The proposed algorithm is intuitive and easy to follow. Specifically, this paper proposes a variant of contextual-to-non-contexutal reduction from Hanna et al., 2023 to handle the adversarial linear loss, which is interesting to me. In addition, to achieve the efficiency improvements, the authors also proposed an algorithm for non-contextual linear bandits with known misspecification, which utilizes the nice property of this reduction.\n- With the knowledge of the context distribution, the algorithm does not need to approximate the non-contextual action set and achieves a better $d\\sqrt{L_0}$ regret.\n\nWeaknesses:\n- As the authors mentioned, the $d$ dependency in the regret bound when the context distribution is unknown is still suboptimal.\n- Maybe the authors can add more explanations to Algorithm 2 in the main text. Currently, it seems that the authors just list Algorithm 2 and its guarantee without introducing more about the intuition and the procedure.\n\nIn general, I do not find more major concerns.\n\nQUESTIONS:\n- Based on my understanding, the reason why first-order regret bound is not obtained for the unknown context distribution setting is that the action set approximation error can not be related to the loss of the algorithm. I wonder whether there are some methods to address this?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-26"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe work investigates the problem of adversarial linear contextual bandits, where in each round a set $\\mathcal{A}_t$ of $K$ actions is drawn i.i.d. from an unknown but fixed distribution over $\\mathbb{R}^d$. The randomized learner picks $a_t \\in \\mathcal{A}_t$ and observes a noisy loss whose expectation is $\\langle a_t,\\theta_t \\rangle \\in [-1,1]$, where $\\theta_t\\in\\mathbb{R}^d$ is a loss vector chosen by an oblivious adversary. The regret is defined with respect to the optimal policy, which is known to be linear. The authors prove regret bounds of the order of $d^2\\sqrt{T}$ using an algorithm whose running time is polynomial in $d$, $T$, and the number of linear constraints sufficient to define the convex hull of each action set. This is the best regret among all known efficient algorithms with full bandit feedback, no access to simulators, and no dependence on $K$. The lack of dependence on $K$ is useful for applications to certain instances of combinatorial bandits such as shortest paths. When a simulator for the distribution is available, the regret can be improved to order of $d\\sqrt{L^\\star}$ (up to log factors), where where $L^\\star$ is the cumulative loss of the best policy.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths: The result is obtained by a novel and technically involved extension to an adversarial setting of a reduction previously proposed for the stochastic setting. As the reduction is to adversarial linear bandits with fixed action set and misspecification, the authors also design a new algorithm for this setting with the required tolerance and runnning time. The lack of the dependence on $K$ allows to obtain the first efficient algorithm with $\\sqrt{T}$ regret for combinatorial bandits with random action sets.\n\nWeaknesses: The paper is not particularly easy to understand unless one has a good prior knowledge of the setting and its basic properties (the quality of the writing could also be improved). The abstract suggests that the proposed algorithm allows to solve any combinatorial bandit problem in polytime, whereas this seems true only for those action sets whose convex hull can be described by a polynomial number of linear constraints. Despite the rather impressive technical effort, the overall significance of the contribution remains a bit unclear. The algorithm is neither simple nor practical.\n\nA rather typical NeurIPS contribution that privileges technical prowess over conceptual novelty. It is nevertheless a solid contribution that advances the state of the art. I am therefore supporting acceptance.\n\nQUESTIONS:\n1. The optimal regret $d\\sqrt{T}$ is currently achieved either inefficiently or with a polynomial number of calls to a simulator. The question whether the optimal regret could be achieved in poly time (even allowing a dependence on $K$) and without using a simulator seems rather fundamental and is not directly addressed by your work. Could your techniques provide any insight on this question?\n\n2. Is your algorithm the first one working in the convex hull of the action set? What did prevent previous approaches to depend on $C$ rather than $K$?\n\n3. There is a large body of literature proposing algorithms that simultaneosly achieve optimal (or near-optimal) stochastic and adversarial regret bounds for bandits (best of both worlds). Could your techniques be applied to obtain results in this direction?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper studies the adversarial linear contextual bandits problem with stochastic action sets.\nIn this problem, in each round, an adversary chooses a loss vector $\\theta_t \\in R^d$ \nand a random action set (here also referred to as \"context\")\n$A_t \\subseteq R^d$ is chosen via sampling from some fixed (but unknown) distribution.\nThe algorithm can then choose some vector $a_t \\in A_t$, incurring\nthe loss $\\ell_t(a_t) = <a_t, \\theta_t>$.\nThe goal is to compete with the set of mappings from\ncontext (aka action sets) to actions;\n\nThe paper obtains an algorithm with\n$\\tilde{O}(d^2\\sqrt{T})$ regret\nthat runs in time $poly(d, T, C)$ time where\n$C$ is the number of linear constraints defining $A_t$.\nImportantly, the running time does not depend on the number of actions,\nwhich was a major limitation of the prior work.\nAs a special case, this gives the first polynomial-time\nalgorithm with sublinear regret for combinatorial bandits with adversarial losses and stochastic action sets.\nThis can further be improved to $\\tilde{O}(d\\sqrt{L^*})$ when\ngiven access to a simulator for sampling $A_t$.\n\nThe paper obtains the result by providing a reduction to \nthe linear bandits with misspecification (where the action set is fixed).\nThe problem is misspecified in that the feedback received\nby the learner is not unbiased; they receive a feedback\nthat is at most $\\epsilon$ away from the inner product of $\\theta_t$\nand the chosen action.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- The paper addresses a computational limitation of existing work.\n  As such, it fits in nicely with a larger (and growing) body of literature which have attempted to do this\n  for many statistical problems.\n- The approach is based on a reduction to a simpler problem, which in my opinion is nice.\n\nWeaknesses:\n- The regret does not match that of the existing works which are not computationally efficient, though\n  in my opinion this is fine.\n- The paper has quiet a few typos (e.g., see below) and could benefit from proofreading.\n\nQUESTIONS:\nQuestions:\n- In the definition of linear bandits, does the learner\n  incur the loss they see as feedback or do they incur $<y_t, \\theta_t>$?\n- The definition of regret in line 84 seems wrong; there should be an expectation over $A_t$?\n  Prior work (e.g., see Liu et al.) also take expectation.\n  Without expectations, assuming $A_t$ comes from a distribution that is not discrete,\n  there are likely going to be different $A_t$ values across all $t$.\n- Line 90: \"learner decides\": I think you mean adversary decides?\n- The reduction seems fairly generic. Are their similar problems in which the authors believe\n  such approaches would work? Alternatively, are there similar examples known in the literature the authors are aware of?\n  If so it would be good to discuss them in more detail.\n- It would be good if you provide a more detailed comparison of your reduction with that of Hanna et al.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-01"}