Title: Infrequent Exploration in Linear Bandits
Venue: NeurIPS 2025
Accepted: yes
Keywords: linear bandits

greedy selection
Abstract: We study the problem of infrequent exploration in linear bandits, addressing a significant yet overlooked gap between fully adaptive exploratory methods (e.g., UCB and Thompson Sampling), which explore potentially at every time step, and purely greedy approaches, which require stringent diversity assumptions to succeed. Continuous exploration can be impractical or unethical in safety-critical or costly domains, while purely greedy strategies typically fail without adequate contextual diversity. To bridge these extremes, we introduce a simple and practical framework, INFEX, explicitly designed for infrequent exploration. INFEX executes a base exploratory policy according to a given schedule while predominantly choosing greedy actions in between. Despite its simplicity, our theoretical analysis demonstrates that INFEX achieves instance-dependent regret matching standard provably efficient algorithms, provided the exploration frequency exceeds a logarithmic threshold. Additionally, INFEX is a general, modular framework that allows seamless integration of any fully adaptive exploration method, enabling wide applicability and ease of adoption. By restricting intensive exploratory computations to infrequent intervals, our approach can also enhance computational efficiency. Empirical evaluations confirm our theoretical findings, showing state-of-the-art regret performance and runtime improvements over existing methods.


Peer reviews:
comments: SUMMARY:
This paper provides a generic algorithmic template for infrequent exploration in stochastic linear bandits. It provides a simple meta algorithm that attains near-optimal regret, with a near-optimal regret bound. The algorithm can at most tolerate $w(\frac{\log t}{t})$ frequence exploration, in the asymptotic regime when $d$ is treated as a constant.

STRENGTHS AND WEAKNESSES:
**Strengths:** the result is novel and technically elegant. It uses a simple meta algorithm by reduction to black-box linear bandit algorithms. The regret bound is also generic and reflects trade-offs between factors.

**Weakness:** the statement of the main theorem should clarify that $f(t)=w(\log t)$ for any $t$, otherwise it would lead to a pathological interpretation.

It would be beneficial to have a more detailed, quantitative discussion on the trade-off between regret and exploration, mainly on the effect of $G(T)$. For example, one question in my mind is that when (how large $T$ grows in $d$) frequent exploration is for free, for different choices of $f(t)$ from the table.

QUESTIONS:
See Strengths And Weaknesses.

LIMITATIONS:
Yes.
TITLE: Official Review
CLARITY: 2
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-06-08

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the problem that lies in the intersection between fully adaptive algorithms that explore at every step and greedy algorithms that do not explore. In particular, the authors are interested in whether infrequent exploration can achieve near-optimal regret while reducing exploration costs and computational burden. To address this, they propose a new general algorithm, INFEX, that allows a base algorithm as an input and decides a schedule for which the algorithm explores or commits. The authors show regret bounds for their algorithm and show that these bounds match with the bounds of fully adaptive algorithms asymptotically. The authors also show numerical experiments that compare the empirical performance of INFEX with fully adaptive and greedy algorithms.\n\nSTRENGTHS AND WEAKNESSES:\nStrength: \n- In terms of clarity, the writing is very clear with sections and subsection titles, also the authors stated contributions very clearly. \n- In terms of quality, the proposed framework is quite general, in the sense that we can plug in a lot of existing algorithms. \n- The numerical experiments and visualizations are clear, and the authors compare their algorithms with various benchmarks. \n\nWeaknesses: \n- In terms of significance, the motivation could be discussed further, especially in terms of practical use. The authors mentioned this in their last open question, \"Can infrequent exploration methods also demonstrate practical advantages beyond theoretical considerations?\", but I'm not sure if the authors fully answered it in this paper.\n\nQUESTIONS:\n- In Theorem 1, you provide a regret bound that holds \"with high probability\", which is a bit unusual since high probability statements often occur in best-arm identification instead of regret. Is this a probability 1 statement? If it is with probability 1-delta, where can I see the dependence of delta in the statement of Theorem 1?\n- Also in Theorem 1, can you describe intuitively what the constants a,b, and c are, and in which scenarios they would be large?\n- Can you explain intuitively why inefficient exploration for the instance you constructed in the numerical experiment has a benefit? I'm a bit surprised that for this instance, INFEX even beats LinUCB which is fully adaptive. I would imagine infrequent exploration gives benefits mostly computationally, but not necessarily a better regret bound. \n- Also, it would be good to understand why no exploration, e.g. LinTS, performs so poorly here. \n- Does INFEX work in settings with budget constraints? I was thinking whether you could frame some objectives for which the number of exploration is limited due to e.g. resources or time, so we have to do infrequent exploration here. Could the authors provide some intuition on whether their algorithm will or will not work in this case, or they are different settings?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes INFEX, a general framework for linear contextual bandits that performs infrequent exploration. It aims to bridge the gap between: \n- \"Fully\" exploratory algorithms like UCB or Thompson Sampling, which explore continuously but may be ethically problematic in practice; and \n- greedy algorithms, which completely avoids exploration but require strong assumptions (e.g., persistent feature diversity) to achieve good regret.\n\nTheir algo, INFEX, maintains a predefined exploration schedule, running a base exploratory algorithm (e.g., LinUCB) at these rounds, and acting greedily in other rounds. The main result shows that INFEX achieves instance-dependent regret bounds matching standard algorithms, as long as the exploration schedule is superlog.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n- Clarity and simplicity: The algorithm is intuitive, easy to implement, and compatible with many existing base algorithms.\n- Clean results: The paper proves that logarithmically infrequent exploration suffices to maintain optimal performance in the linear bandit setting. \n- Practical motivation: The framework is well-motivated for domains like healthcare or safety-critical systems, where constant exploration may be infeasible.\n- Good writing: Overall, the paper is easier to read and better organized than many others.\n\n**Weaknesses.** The authors repeatedly state that previous works require “strong distributional assumptions,” particularly for greedy policies. However, it is not clearly explained what these assumptions are, why they are necessary for greedy algorithms, and why INFEX avoids them. A more explicit comparison (ideally with counterexamples or illustrations) would greatly strengthen the paper. \n\nIn particular, what are the “bad cases” that INFEX handles well? It would be illuminating to show specific examples where greedy fails but INFEX succeeds — this would clarify the paper’s practical impact.\n\nQUESTIONS:\nsee above\n\nLIMITATIONS:\nsee above", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the theoretical regret guarantees of infrequent exploration in linear bandits. Specifically, this work proposes INFEX, which interleaves greedy actions and the use of an adaptive exploratory policy, and establishes that INFEX achieves a polylogarithmic instance-dependent regret bound under the condition that number of exploration steps is at least $\\omega(\\log t)$. As a result, this work fills the gap between the fully exploratory methods and the purely greedy method in linear bandits. Moreover, the paper complements the regret analysis with numerical simulations, showing that INFEX can attain strong empirical regrets while achieving less computation time compared to the two base linear bandit algorithms LinUCB and LinTS.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- The proposed INFEX is a generic and conceptually simple infrequent exploration method that nicely bridges the gap between the various existing adaptive exploratory bandit algorithms and the purely greedy policy. This offers interesting new insights into the linear stochastic bandit problems.\n- The proposed INFEX achieves an instance-dependent regret bound for any linear bandit algorithm with polylogarithmic regret (under some mild exploration condition). The results accordingly gives a new instance-dependent regret bound for LinTS, which can be of independent interest to the bandit community.\n- One empirical benefit of using INFEX is that under some base bandit algorithms (such as LinTS and LinUCB), the infrequent exploration mechanism can reduce the overall computation time required while still achieving similar scaling of regret.\n- Overall, the writing is very clear, and the proof sketch nicely highlights the important steps of the regret analysis, which makes the paper easier to read.\n\nWeaknesses\n\n- While I can appreciate the contribution regarding the algorithm and the established regret bounds, I do find INFEX somewhat not that easy to use in practice. Specifically, as shown in Theorem 1 and Table 1, the choice of exploration schedule can significantly affect the $G_{const}$, which depends on multiple problem-dependent and algorithm-dependent parameters. These parameters are typically unknown and hence make it difficult to optimize the trade-off between $G_{const}$ and the other regret terms. Therefore, in practice, it is hard to choose a proper exploration schedule and the resulting $f(t)$ a priori.\nOne quite possible scenario is that infrequent exploration can incur a much higher regret by using infrequent exploration despite that the computation time can be somewhat reduced. Without addressing the above issues with the choice of exploration schedule, I find the algorithm not as promising as claimed despite that theoretically this is an interesting contribution.\n\n- While I can see that this work mainly focuses on instance-dependent regret bounds, I wonder if the infrequent exploration can possibly hurt the minimax regret bound. For example, suppose we use LinUCB and consider the degenerate case where each arm $i$ is a one-hot vector with the $i$-th element being one and $\\theta$ is $K$-dimensional with the $i$-th entry being the mean reward of arm $i$, then intuitively, if the resulting exploration budget is only as small as $f(t)=\\log^m t$ with $m>1$ (which still satisfies the condition in Theorem 1), then the minimax regret can be linear in $T$. With that said, I believe that this work can be strengthened if a discussion on the resulting minimax regrets can be provided as well. \n\n- Another concern is that this paper assumes that the arm set is finite and fixed, but the more common setting in the (generalized) linear bandit literature is that the arm set can be arbitrary (e.g., OFUL [Abbasi-Yadkori et al., 2011] and LinTS [Abeille and Lazaric, 2017]) or time-varying (e.g., LinUCB [Li et al., 2010] and UCB-GLM [Li et al., 2017]). It is not immediately clear to me whether the regret bounds in Theorems 1-2 can be directly extended to the more standard linear bandit settings.\n\nQUESTIONS:\n1. This paper assumes that the arm set is finite and fixed, but the more common setting in the (generalized) linear bandit literature is that the arm set can be arbitrary or time-varying. Can the authors comment on whether these assumptions are indeed needed or the results can be readily extended to the more standard linear bandit settings?\n\n2. While I can see that this work mainly focuses on instance-dependent regret bounds, I wonder if the infrequent exploration can possibly hurt the minimax regret bound. Can the authors comment on this?\n\n3. From the numerical results in Figure 1, it appears that INFEX typically has higher standard deviation in regret as the exploration budget gets smaller. This can be an undesired property in practice as the regrets in different trials can vary quite significantly.\n\nLIMITATIONS:\nThe work does not describe any potential limitations in the main paper.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-05"}