Title: Regret Bounds for Adversarial Contextual Bandits with General Function Approximation and Delayed Feedback
Venue: NeurIPS 2025
Accepted: yes
Keywords: Regret

Bandits

Contextual Bandits

Delayed Feedback

Function Approximation
Abstract: We present regret minimization algorithms for the contextual multi-armed bandit (CMAB) problem over $K$ actions in the presence of delayed feedback, a scenario where loss observations arrive with delays chosen by an adversary. 

As a preliminary result, assuming direct access to a finite policy class $\Pi$ we establish an optimal expected regret bound of $ O (\sqrt{KT \log |\Pi|} + \sqrt{D \log |\Pi|)} $ where $D$ is the sum of delays. 
For our main contribution, we study the general function approximation setting over a (possibly infinite) contextual loss function class $ \mathcal{F} $ with access to an online least-square regression oracle $\mathcal{O}$ over $\mathcal{F}$. In this setting, we achieve an expected regret bound of $O(\sqrt{KTR_T(\mathcal{O})} + \sqrt{ d_{\max} D \beta})$ assuming FIFO order, where $d_{\max}$ is the maximal delay, $R_T(\mathcal{O})$ is an upper bound on the oracle's regret and $\beta$ is a stability parameter associated with the oracle. 

We complement this general result by presenting a novel stability analysis of a Hedge-based version of Vovk's aggregating forecaster as an oracle implementation for least-square regression over a finite function class $\mathcal{F}$ and show that its stability parameter $\beta$ is bounded by $\log |\mathcal{F}|$, resulting in an expected regret bound of $O(\sqrt{KT \log |\mathcal{F}|} + \sqrt{d_{\max} D \log |\mathcal{F}|})$ which is a $\sqrt{d_{\max}}$ factor away from the lower bound of $\Omega(\sqrt{KT \log |\mathcal{F}|} + \sqrt{D \log |\mathcal{F}|})$ that we also present.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper studies a variation of the well-known problem of contextual multi-armed bandits in which each action taken by the agent incurs in a loss that can only be observed with a certain delay $d_t$, possibly adversarially chosen, with respect to the time when the action itself was taken. The work focuses on two specific settings and for each provides algorithms and derives bounds on their expected regret.\nThe first one (Policy Class Learning_ setting) assumes that the benchmark optimal policy belongs to a known class of cardinality $N$. In this case, the algorithm proposed is an adapted version of EXP4, which incurs in an expected regret of $O(\\sqrt{KT\\log N} + \\sqrt{D\\log N})$. \nIn the second setting (_Online Function Approximation_ setting), instead, the loss function $f_{\\star}$ is assumed to belong to a class $\\mathcal{F}$, which the agent has access to.  The goal is therefore to compete against the benchmark given by $\\pi_{\\star}(x)=argmin_{a\\in A}f_{\\star}(x, a)$. The algorithm relies on an oracle to build an estimate the loss function, which incurs in a regret of $\\mathcal{R}_T(\\mathcal{O})$, and the total expected regret of the algorithm is $\\sqrt{KT\\mathcal{R}_T(\\mathcal{O})}$. Lastly, they provide an example of an oracle based on Hedge algorithm, further specifying in this case both upper and lower bounds.\n\nSTRENGTHS AND WEAKNESSES:\nThe paper is technically sound, and it analyses a problem with theoretical relevance, which is part of a broad literature of interest to the community, providing new results and insights on the topic. Furthermore, the setting and the needed assumptions are clearly stated, making it very clear to understand.  \n\nThe main weaknesses originate from the theoretical assumptions required to derive the bounds, which could be restrictive. Specifically, for both settings, complete knowledge of the classes $\\Pi$ and $\\mathcal{F}$, respectively, is assumed.\n\nMoreover, if I am not mistaken, there is no formal definition of the quantity $\\mathcal{R}_T(\\mathcal{O}_{sq}^{\\mathcal{F}})$ in the main paper. I think the paper would benefit clarity-wise from a precise definition.\n\nQUESTIONS:\nI have the following questions for the authors.\n\n1. On the line of what I wrote in the previous section, I wonder if there are ways to relax the assumption of knowledge of the classes even partially, as for instance assuming $\\Pi$ to be infinite.\n\n2. I am curious about the idea of introducing the oracle as part of the DA-FA algorithm. I understand the algorithm still works using other subroutines other than the one detailed in Algorithm 2, and that this phrasing allows the result to be more general, but are you aware of other oracle algorithms possibly achieving the same regret rate or preferable with respect to other criteria?\n\nLIMITATIONS:\nThe authors have addressed all the limitations.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper provides a comprehensive analysis of adversarial Contextual Multi-Armed Bandits (CMAB) in scenarios with delayed feedback, tackling both finite policy class and general function approximation frameworks. The authors highlight a key discrepancy between this formulation and its non-delayed counterpart: standard relatively weak assumptions on a regression oracle are insufficient in the presence of delays, introducing instead a stability condition as a necessary criterion for achieving sublinear regret.\n\nUnder this stability criterion, they successfully establish an optimal regret bound in the finite policy setting and a near-optimal regret bound in the general function approximation scenario. In the latter case, a lower bound is provided. Specifically, the authors employ a Hedge-based version of Vovk’s aggregating forecaster, demonstrating its satisfaction of the stability condition, and thereby justifying its efficacy as a practical oracle. The aggregating forecaster matches the lower bound up to a factor of $\\sqrt{d_{\\max}}$ in the delayed term.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- The authors have pinpointed a fundamental difference in the requirements of a regression oracle in case of delayed feedback, and demonstrated the sufficiently of a stability condition on the regression oracle to achieve sublunar regret. \n- The contributions lay the foundation for the analysis of more complex settings, such as learning in MDPs with function approximation, and\n- The paper is well written, and contributions are clearly delineated.\n- The structure of the paper is such that ideas are built up gradually, and consequently renders the paper very easy to follow.\n- Complementing results with a lower bound has narrowed down the remaining  slack to a relatively small subset of possibilities.\n\nWeaknesses:\n- A few of the assumptions are standard, but arguably strong already (e.g. realisability), but this compounded with the FIFO assumption does limit the applicability of results.\n- The discussion on the necessity of the FIFO assumption does lack a bit of nuance, and I do think that weakening of this criteria may still lead to a setting where sublinear regret is possible (for example, if the learner has other means to weakly attribute ordering of feedback). \n- The main regret bound is still off by a factor of $\\sqrt{d_{\\max}}$, which isn’t discussed in very great detail. Given the contributions, this is a minor consideration.\n\nQUESTIONS:\n1. Stability over the delay length appears pivotal in the oracle’s analysis. Is the global stability assumption on the oracle (i.e., stability over all intervals of length T) fundamentally necessary for sublinear regret? Could a weaker assumption, such as max inf-norm stability restricted to intervals of length $\\leq d_{\\max}$ suffice instead?\n2. To what would you attribute the slack between the upper and lower bounds on the regret for the aggregating forecaster? Is it related to the comment above? A quirk of the aggregating forecaster in particular? Is it more likely slack in the lower bound, or something else entirely?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper studies the contextual bandit problem\nwith delayed feedback; i.e., the algorithm sees the rewards only after some delay\nwhich is determined by an adversary.\nIn the finite policy case, they show a regret bound of\n$O(\\sqrt{KT \\log|\\Pi|} + \\sqrt{D \\log|\\Pi|})$ where $D$ is total delay.\nThey further show that this bound is optimal.\n\n\nFor the general function approximation setting, with a regression oracle, \nthey use an approach inspired by the SquareCB algorithm of Foster and Rakhlin to obtain a bound of\n$O(\\sqrt{KT R_{T}(O)} + \\sqrt{d_{\\max}D \\beta})$\nwhere $R_T(O)$ is the regret of the oracle and $\\beta$ is an \"oracle stability\" parameter\nand $d_{\\max}$ is the maximum delay.\nThis result is under the realizability assumption as well as the assumptions that\nthe delays do not change the order in which feedback is seen (i.e., FIFO).\nFor finite function classes, they show that $\\beta$ can be bounded with\n$\\log|F|$ using a specific oracle they provide,\nobtaining a bound of \n$O(\\sqrt{KT \\log|F|}(O) + \\sqrt{d_{\\max}D \\log|F|})$\nFor this special case, they also show a nearly matching lower bound which does not\nhave the $d_{\\max}$ term.\nIn order to implement the oracle, the authors use Vovk's aggregation algorithm and prove\nthat it satisfies the stability requirements in their results.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- The paper studies (in my opinion) an interesting problem.\n  Delayed feedback is interesting both from a practical perspective as in many cases\n  one cannot observe the consequences of one's actions (e.g., submitting papers to a conference)\n  and from a theoretical perspective as it is a simple and nice modification of the original problem.\n- The combination of upper and lower bounds for the settings they study\n  makes this a solid contribution and the paper is fairly well-written.\n\nWeaknesses:\n- I think most of the technical depth of the paper is from prior work and\n  perhaps the marginal technical contribution is not necessarily very high.\n  In particular, the algorithm itself is heavily based on SquareCB and\n  the use of Vovk's aggregating algorithm for the oracle is also based on prior work.\n\nQUESTIONS:\nQuestions:\n- Is there any intuition as to why removing the FIFO assumption may make the problem challenging?\n  The explanations in line 257-262 focus on your algorithm rather than the problem statement itself.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies contextual multi-armed bandits (CMAB) under adversarial delayed feedback. The authors consider two settings: (1) policy class learning with direct access to a finite policy class, and (2) general function approximation via online least-squares regression oracle under a stability assumption. The authors prove matching lower bounds, show that the stability condition is essential, and demonstrate that a simple Hedge-based forecaster satisfies it, positioning their framework as a significant but still partially restrictive step toward practical delayed-feedback bandit learning.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths  \n\n- **Practically motivated**: Delayed feedback is the norm in applications like ad placement and clinical trials.  \n- **Tight theory for policy classes**: Upper and lower bounds coincide, fully characterizing delay complexity.  \n- **Novel technical ideas** : Stability analysis of Hedge introduces a fresh lens for delayed bandits.  Lower-bound proof uses an inventive optimal-transport argument on Hamming space.  \n- **General framework**: Extends from finite policy classes to general function approximation via oracle access, which is more practical.\n\n---\n\n## Weaknesses  \n- **Strong assumptions**\n  - *β-stability* (Assumption 4.4) excludes many common online regressors.  \n  - FIFO arrival order (Assumption 4.5) is rarely met in real networks where packets can be reordered..  \n- **Presentation hurdles**  \n  - Key intuitions behind the loss estimator and stability lemma are buried in proofs.  \n  - Relationship between $d_{\\epsilon}$ (policy classes) and stability parameter β lacks high-level explanation.  \n- **Missing empirical evidence & comparisons**  \n  - No experiments to illustrate constants or runtime.  \n  - Limited discussion of connections to delayed linear bandits or privacy-aware bandits.\n\nQUESTIONS:\n1. Can you characterize the β-stability parameter for common oracle implementations beyond Hedge? Does ridge regression with time-varying regularization satisfy β-stability?\n2. How does your approach compare to the \"doubling trick\" baseline that simply restarts the algorithm every $d_{\\max}$ rounds?*\n\nLIMITATIONS:\nyes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 2, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}