Title: Eluder dimension: localise it!
Venue: NeurIPS 2025
Accepted: yes
Keywords: eluder dimension

bandits

reinforcement learning
Abstract: We establish a lower bound on the eluder dimension in generalised linear model classes, showing that standard eluder dimension-based analysis cannot lead to first-order regret bounds. To address this, we introduce a localisation method for the eluder dimension; our analysis immediately recovers and improves on classic results for Bernoulli bandits, and allows for the first genuine first-order bounds for finite-horizon reinforcement learning tasks with bounded cumulative returns.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposed a localized $\\ell_1$-eluder dimension that can handle the limitation of the standard eluder dimension in the regret bound of the decision-making problem. The authors first give a lower bound of a small-cost bound by using global eluder dimension in generalised bandits to show that the drawback of global eluder dimension can not match the best known upper bound. Based on this motivation,the authors provided UCB-based algorithms for both Bernoulli bandits and online RL problems and established the adaptive, first-order regret bounds based on their localization approach.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. The paper provided solid theoretical contributions and fundamental insights. It is good to first provide Theorem 4.2 as evidence to show the disadvantages of the previous method and then introduce localised eluder dimension to show how localization helps it, which is also conceptually meaningful.\n2. The writing is good. And it is good to give some function examples to Assumption 2.1. And the authors give some sufficient comparisons with previous related work, such as section 4.1.\n3. The paper studied both bandits and RL problems under this method, which shows the efficiency of the method and its potential to be used in more problems, such as RLHF.\n\nWeaknesses:\nThe paper lacks experimental results. And it is unclear how to use the localised eluder dimension in practice. I understand the main contribution is on the theoretical side, but some empirical results would show the practical benefits of localisation.\n\nQUESTIONS:\n1. The set of functions $\\mathcal{F}$ is bounded and positive. Can it extend to negative or unbounded?\n2. Could you give some remarks on the last term in the result of Theorem 4.1, $\\mathrm{card}${$ t \\leq n : f_t \\notin \\mathcal{F}' $}? Is it normal in previous related work?\n3. In line 172, the sentence seems incomplete. $\\mathcal{F}^{\\prime}(r)=$ {$\\mu(\\langle\\cdot, \\theta\\rangle): \\theta \\in \\Theta^{\\prime}(r)$}  for what?\n4. The lower bound in Theorem 4.2 is for the Eluder dimension. Is it possible to provide the lower bound for the regret bound? Or any evidence to show how good your regret upper bound is?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 2, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-21"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper investigates the limitations of the standard eluder dimension in achieving first-order regret bounds for GLMs in online decision-making. The authors show that exisiting eluder-based approaches inherently suffer from dependence on a global worst-case complexity term \\kappa, which cancels out potential gains in small-cost settings. To address this, they propose a localized \\ell_1-eluder dimension that captures the complexity near the optimal predictor. They use this refined measure to derive genuinely first-order, instance-adaptive regret bounds in both stochastic bandits and finite-horizon RL, instantiated via new algorithms (\\ell-UCB and \\ell-GOLF). Theoretical results show improved guarantees over prior work without requiring warm-up phases or likelihood assumptions.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\nThe paper identifies and rigorously proves a key limitation in prior work using global eluder dimensions and proposes a novel and solid fix.\n\nThe paper achieves the first true first-order bounds in episodic MDP under bounded cost, improving over existing works which suffer from \\kappa-dependent regret.\n\nThe work applies to both stochastic bandit and online RL settings,with insights that may extend to other model classes and loss structures.\n\nWeaknesses:\n\nThe paper is entirely theoretical. While this is acceptable given the depth of the contribution, even a small synthetic experiment would help illustrate the practical gains from localization.\n\nThe regret bounds rely on specific assumptions about the loss function (e.g. triangle condition), realizability and self-concordance. These may not hold in broader settings thus it would be helpful to know how sensitive the results are to violations of these assumptions.\n\nOptimization over localized confidence sets can be intractable in general, and the paper only briefly references convex relaxations without discussing implementation feasibility.\n\nQUESTIONS:\nHow restrictive is triangle condition in practice? Are there other loss functions beyond log and Poisson loss that satisfy it?\n\nCould your framework be extended to misspecified setting?\n\nDo you envision the localized loss-based analysis extending to more general function classes, like RKHS or NNs?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-25"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe argument in the paper (both the bandit and RL case) proceeds as:\n\n1. There is a pathological information constant $\\kappa$ that nullifies the benefit of small cost regret bounds in prior work.\n\n2. But actually, we should be able to do with a much smaller constant so the above is not a \"real effect,\" but an artifact of the worst-case nature of the previous analysis.\n\n3. To make this tighter, the authors replace with the local constant $\\kappa$ but then need to pay an additional term for mis-approximating the actual \\kappa constant.. and we are guaranteed that this is generally bounded for good algorithms that eventually hang out in a ball around the true \\eta.\n\nSTRENGTHS AND WEAKNESSES:\nI thank the authors for the interesting paper and I enjoyed reading it.\n\n\n### Strengths\n\n**Significance and Originality** The authors identify an important problem in the prior theoretical RL literature and come up with a nice way to tighten the bound and make it genuinely instance-dependent.\n\n**Quality** The mathematical presentation is technically sound.\n\n\n### Weaknesses\n\n**Literature** The authors overall do a good job of situating the paper. However, they do not comprehensively cover the related area of instance-dependent guarantees in RL that use experiment design-based approaches. Some relevant papers to compare with:\n\nhttps://arxiv.org/abs/2207.02575\nhttps://arxiv.org/abs/2406.06856\nhttps://arxiv.org/abs/2311.05638 \n\nI would be curious to understand how the bound the authors obtain relate to the upper and lower bounds in this line of work, since it seems that the experiment-design style of analysis does not suffer from the limitations that the authors aim to address.\n\n**Writing** While the technical results are nice, some improvements in the writing could definitely help the central insights shine through more; currently, the insights are hidden inside a lot of exposition. For instance, a couple things that would be good to do:\n\n- Have a concrete and easy-to-understand motivating example in the beginning which shows why the information gain factor easily blows up. And make more explicit what sorts of instances this effect is predominant in.\n\n- Highlight the additional technical challenges for the RL case as compared to the bandit case. And highlight this result more as the strongest result of the paper. It could be good to state this inside a theorem environment even if the statement is informal.\n\nQUESTIONS:\nSee weaknesses above\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper aims at overcoming a key limitation in standard regret analysis in standard regret analysis for generalized linear models, where traditional methods fail to achieve first-order regret bounds due to worst-case dependencies. The authors incorporate a new concept called the localized ℓ₁-eluder dimension, enabling sharper, instance-dependent regret bounds. The proposed method improves upon classical results for Bernoulli bandits and extends to reinforcement learning, delivering the first true first-order regret bounds in finite-horizon RL. The findings highlight how localization techniques can unlock more adaptive algorithms, paving the way for further advances in learning theory.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1.\tThe paper makes significant theoretical contributions, with rigorous proofs supporting the proposed method in manuscript and appendix.\n2.\tThe approach is well-suited for online RL tasks, eliminating the need for impractical exploration mechanisms required by prior methods.\n\nWeaknesses:\n1.\tWhile the theoretical analysis is thorough, the paper lacks sufficient experimental validation (e.g., RL benchmarks or real-world datasets).\n2.\tThe computational efficiency of the method, especially in high-dimensional state-action spaces, remains unexplored.\n3.\tThe writing could be improved for clarity and style. For instance, the title uses inconsistent capitalization (e.g., \"Localise It!\")\n\nQUESTIONS:\n1.\tWhile the localized eluder dimension eliminates k-dependence, Proposition 4.4 shows rogue steps still depend on the parameter k. Could k still significantly impact performance in extreme cases (e.g., very large k)?\n2.\tHow should key hyperparameters like localization radius r be chosen? Are there adaptive methods to set r?\n3.\tWhile 2D bandit experiments demonstrate advantages, would similar improvements hold for higher-dimensional RL tasks?\n4.\tHave you considered comparisons with non-eluder dimension methods?\n5.\tCould the authors provide some experimental results on RL benchmarks? Should the authors make the computational complexity of the proposed method?\n\nLIMITATIONS:\nHere the two suggestions for the authors: (i) The theoretical analysis currently focuses on bounded-cost scenarios. The authors should discuss whether/how the method could generalize to unbounded reward settings. (ii) Computational complexity of the localized eluder dimension calculation isn't analyzed. Some discussion of practical scalability would strengthen the work.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-05"}