Title: Non-Stationary Lipschitz Bandits
Venue: NeurIPS 2025
Accepted: yes
Keywords: non-stationary bandits

Lipschitz bandits

tracking most significant shifts
Abstract: We study the problem of non-stationary Lipschitz bandits, where the number of actions is infinite and the reward function, satisfying a Lipschitz assumption, can change arbitrarily over time. We design an algorithm that adaptively tracks the recently introduced notion of significant shifts, defined by large deviations of the cumulative reward function. To detect such reward changes, our algorithm leverages a hierarchical discretization of the action space. Without requiring any prior knowledge of the non-stationarity, our algorithm
achieves a minimax-optimal dynamic regret bound of  $\mathcal{\widetilde{O}}(\tilde{L}^{1/3}T^{2/3})$, where $\tilde{L}$ is the number of significant shifts and $T$ the horizon. This result provides the first optimal guarantee in this setting.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies non-stationary Lipschitz bandits and proves the first minimax optimal dynamic regret bounds in terms of a new non-stationary metric, \"significant shifts\", which follows a similar line of results in the non-stationary K-armed bandits.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. The first minimax optimal regret bound for non-stationary Lipschitz bandits with both proven lower and upper bounds.\n2. New techniques involving multi-depth binning strategy for Lipschitz bandits.\n\nWeaknesses:\n1. Lack of experimental validation of the MDBE algorithm.\n2. I think Section 3 is quite heavy in notation and can be better served with a diagram/picture and example to illustrate the process of choosing a depth/arm.\n3. Much of the analysis seems structured similarly to the previous works on learning significant shifts. In particular, the work of Suk and Kpotufe [2023] also uses a multi-depth adaptive discretization idea to address the problem of learning the right discretization level, although it is for a finite-armed problem. The authors could better highlight what are the key differences or technical challenges addressed over the Lipschitz contextual bandit setting.\n\nQUESTIONS:\nQuestions/Suggestions:\n1. Why is there a log(T) factor in Definition 1? Is this because of concentration bounds or does it optimize the log dependence in the regret bound?\n2. In the pseudocode of Section B, why reset the master set at each block instead of carrying over information about eliminated regions from previous block to current block?\n3. Again, I believe a diagram or illustrative example would be helpful to explain the bin selection mechanism and replay scheduling.\n4. The dependence on the Lipschitz constant in the regret bound and other generalities such as Holder smooth bandits are missing, some discussion on these extensions would make the paper more thorough.\n5. What is the computational complexity of the algorithm and how does it compare to other algorithms for the non-stationary setting?\n\nTypos:\n1.  \"almost\" in Line 979\n2. Line 596: \"discretion\"\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-24"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis manuscript introduces the first algorithm with optimal regret guarantees for continuum-armed bandits\nunder time-varying, Lipschitz-continuous reward functions. The key idea is to focus on significant shifts,\nwhich meaningfully affect regret. The proposed algorithm, Multi-Depth Bin Elimination (MDBE), adaptively\nexplores a dyadic partition of the action space and uses randomized replays to detect shifts and prune\nsuboptimal regions. Without knowing the number or location of changes, MDBE achieves minimax-optimal\nregret. However, there are several aspects that should be addressed. My main concern lies\nin the comparison with prior work and the justification for adopting the new metric. The detail comments\nare listed below.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n\t1.\tAddresses a previously unstudied intersection of settings:\nThe paper considers non-stationary bandits with a continuous (Lipschitz) action space, a combination that had not been analyzed in depth before. While both components—non-stationarity and Lipschitz structure—are individually well studied, their interaction introduces new technical challenges that are not addressed by existing work.\n\t2.\tTechnically non-trivial adaptation of known methods:\nAlthough one might imagine combining known ideas—such as time-scale decomposition (as in Wang or Besbes–Zeevi) with Kleinberg-style discretization—this paper shows that a naïve hybrid is insufficient. The core difficulty lies in adaptively detecting global, cumulative changes over a continuous space without knowing when or how these changes occur. The algorithm’s hierarchical, replay-based design and multi-scale gap estimation address this challenge in a subtle and well-justified way.\n\t3.\tTheoretical guarantees match minimax lower bounds:\nThe MDBE algorithm achieves a regret bound of \\widetilde{O}(\\tilde{L}_T^{1/3} T^{2/3}), matching the minimax lower bound up to logarithmic factors. Crucially, this is done without requiring prior knowledge of the number or locations of significant shifts. The framework is also robust to small or inconsequential fluctuations, making the bound potentially sharper than those based on classical variation metrics.\n\nWeaknesses\nSummary of the Paper: “Non-Stationary Lipschitz Bandits”\n\nThis paper investigates the problem of non-stationary bandits with a continuous action space and Lipschitz reward functions that can change arbitrarily over time. Unlike traditional non-stationary bandit approaches, which often assume a discrete action set and require prior knowledge of the non-stationarity level, this work addresses the more challenging setting of an infinite arm space under Lipschitz continuity without any such assumptions.\n\nThe authors propose a novel algorithm, MDBE (Multi-Depth Bin Elimination), which adaptively tracks a refined notion of non-stationarity called significant shifts—large aggregate changes in regret that are statistically meaningful. MDBE uses a hierarchical discretization of the action space and a replay mechanism to detect these significant shifts at different resolutions efficiently.\n\nThe main result is that MDBE achieves minimax-optimal dynamic regret of order \\widetilde{O}(\\tilde{L}^{1/3} T^{2/3}), where \\tilde{L} is the number of significant shifts and T is the time horizon. This regret bound matches known lower bounds and does not rely on any prior knowledge of the number or timing of shifts. The paper provides theoretical analysis, a detailed algorithmic design, and high-probability guarantees.\n\n⸻\n\nStrengths:\n\t1.\tNovelty and Scope:\nThe paper pioneers the study of non-stationary bandits in a continuous action setting under Lipschitz continuity—a previously unexplored combination—and introduces a theoretically grounded notion of significant shifts to this context.\n\t2.\tStrong Theoretical Guarantees:\nThe algorithm achieves minimax-optimal regret bounds without knowing the degree of non-stationarity in advance. This is a substantial theoretical contribution that strengthens the relevance of the proposed method.\n\t3.\tSophisticated Algorithm Design:\nMDBE incorporates a multi-resolution replay mechanism and hierarchical bin elimination, reflecting a careful treatment of exploration vs. exploitation in a non-stationary continuous setting. The sampling and bin eviction strategies are technically sound and rigorously justified.\n\n⸻\n\nWeaknesses:\n\t1.\tLack of Empirical Validation:\nThe paper is entirely theoretical and does not provide any simulations or experiments to demonstrate how the algorithm performs in practice. This limits its immediate impact for practitioners.\n\t2.\tComplexity of the Algorithm:\nMDBE’s design is intricate, involving nested partitions, probabilistic replays, and importance-weighted estimators. While theoretically elegant, the practicality and implementation complexity may hinder adoption.\n\t3.\tComputational Concerns Unaddressed:\nThe authors acknowledge that computational efficiency is not considered. Given the algorithm’s multi-scale structure, it is unclear whether MDBE is computationally feasible for large-scale or real-time applications.\n\nQUESTIONS:\n1. In the related work part, when the authors are comparing [Suk and Kim 2025], it is mentioned that\n[Suk and Kim 2025] assumes no regularity across arms thus it is not applicable to this manuscript.\nThis statement seems too vague for me. I suggest the authors make more efforts on elaborating the key\ndifference between this paper and prior works.\n2. The notation may always be confusing. The variable x in this manuscript is denoted as the arm/action.\nHowever, in the literature of bandit, x is denoted as the context and a is denoted as the arm/action.\nI would suggest the authors make some efforts to change it, though it will not take any effect on my\nscoring.\n3. In definition 2, it is confusing whether [s1, s2] ⊆ [τi−1, τi\n] depends with x or it exists for all x.\n4. In the same time, I would like to ask the authors to compare the difference in the definition of significant\nshit with [Suk and Kim 2025], which is not cited in Definition 2.\n5. In the meantime, I also find\n6. For the Section 2.2, though mathematically the comparison makes sense, I would also want to see a\nconcrete example to show using LeT metric is statistically and significantly more efficient. This would\nstrengthen the motivation for adopting LeT and also shows the bounds in Section 4 can be tighter than\nother metrics.\n7. In the proof of Proposition 3, the notation ft−1 should also indicate the condition xt ∈ B. Thus, ft−1,B\nis a better representation.\n\nLIMITATIONS:\nAdequate.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the problem of non-stationary Lipschitz bandits, which generalizes both non-stationary bandits and Lipschitz bandits by allowing the reward function to evolve over time while satisfying a Lipschitz condition. The key contribution is MDBE (Multi-Depth Bin Elimination), a novel algorithm that adaptively adjusts the discretization level. The paper establishes the first optimal dynamic regret guarantees for non-stationary bandits with continuous action spaces under Lipschitz continuity.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\nThis paper is very well-written, with a clear motivation for studying non-stationary Lipschitz bandits as a natural generalization of both non-stationary and Lipschitz bandits with potential real-world applications. The proposed algorithm, MDBE, is novel and thoughtfully designed to address the key challenges of adaptivity and continuous action spaces under unknown non-stationarity. The theoretical results are strong. Moreover, the paper does a good job positioning its contribution relative to prior work, especially in how it extends the notion of significant shifts and adapts replay-based elimination techniques to the continuous setting.\n\nWeaknesses:\n\nThe paper introduces a three-way trade-off in the introduction, but it's unclear what the three ways are.\n\nSome technical terms, such as “dyadic tree”, may be unfamiliar to a broader audience and could benefit from a more intuitive or visual explanation.\n\nNotations in some key places, such as d in Proposition 1 or δ in Definition 4, could be clarified within the theorem statement to improve readability.\n\nWhile the paper is theoretically focused, a small simulation or experiment could help validate the regret bounds and provide intuition about the algorithm’s behavior in practice.\n\nQUESTIONS:\nWhat is the potential for broader applicability to other non-stationary settings of the adaptive discretization strategy? For example, do non-stationary smooth bandits have similar challenges in discretization? If yes, would the strategy proposed in this work be applicable to that problem?\n\nYou clearly summarize how your algorithmic contributions relate to prior work. However, the novelty of the proof techniques is less explicitly highlighted. A brief discussion of what is technically new in the analysis would help clarify the theoretical contribution.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}

comments: SUMMARY:
This paper studies a fundamental variant of the multi-armed bandit problem in which the mean reward function is Lipschitz in space $[0,1]$) and non-stationary in time. The paper considers the notion of significant changes — a refined measure of temporal non-stationarity that generalizes earlier notions like total variation or number of abrupt changes. The authors develop an algorithm that adaptively discretizes the arm space. They achieve a minimax optimal dynamic regret of $\widetilde{L}^{1/3} T^{2/3}$ where $\widetilde{L}$ is the number of significant shifts.

STRENGTHS AND WEAKNESSES:
**Strengths**
- Fundamental problem: The setting of spatially Lipschitz and temporally non-stationary bandits is both natural and under-explored. 
- Novel technical contributions: A discretization strategy based on dyadic trees to efficiently monitor changes across scales; elimination and replay mechanisms tailored to the continuous setting.
- Theoretical guarantees: The $\widetilde{L}^{1/3} T^{2/3}$ regret is minimax optimal (in T and $\tilde L$, at least)

**Weakness:** The results are stated only for Lipschitz constant L=1. The work would be more complete if they could incorporate L and compare the dependence with stationary Lipschitz bandits.

minor comments:
- The phrase "locally stationary" phases (line 52) is unclear. Please provide a more precise definition or intuition.
- The “three-way” trade-off mentioned under forced exploration (Section 3.2) appears to be more of a two-way trade-off: frequency of restarts and discretization granularity. Please clarify the third dimension.

QUESTIONS:
Do the result hold in higher dimensions?

LIMITATIONS:
see above
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 2
DATE: 2025-07-03

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper considers a fundamental online learning problem of non-stationary Lipschitz bandits. In particular, the notion of non-stationarity is new, different from previously studied ones of $L_T$ and $V_T$, both of which could be very large even if the best arm is unchanged. This new notion is called significant shifts introduced recently in other bandit problems, and the authors then propose a Multi-Depth Bin Elimination algorithm to achieve optimal regret for Lipschitz bandits under significant shifts.\n\nSTRENGTHS AND WEAKNESSES:\n### Strength\n- This paper tackles a theoretically fundamental problem in bandits with infinite continuum actions and new notion of non-stationarity. \n- The results are strong with optimal rates.\n\n### Weakness\n- The presentation of algorithm is not very clear: there are two algorithms for separate submodules but actually not an overall algorithm presented in block algorithm environment. \n- The explained intuition of proving Theorem also seems incomplete. It would be nice to have a more high-level proof framework connecting dots together before going into details. Some schematic illustration would also be very helpful. As in current presentation, it is hard to evaluate confidently the correctness of proof. \n- It appears that the main algorithm design is a combination of binning/elimination of seminal works and the replay mechanism of recent work  Suk and Kpotufe [2022] (the same authors who also proposed significant shifts). As a result, it is not very evident how much of the theoretical contribution is conditioning on these two.\n\n### Minor\n- A few typos such as [ and ] in line 127, line 182.\n\nQUESTIONS:\n- The difference of significant shifts is still quite abstract. Can the authors provide some more concrete examples to help understanding? And it is more ideal if such example has strong practical motivation/relevance.\n- Can the authors provide a more exact characterization of $c_0$ as mentioned in the footnote? How does it affect the algorithm performance?\n- Can the authors elaborate more on the unique challenges presented in the analysis of this problem, in particular, compared to previous work by Suk and coauthors?\n- Despite no numerical implementation, can the authors at least analyze the computation complexity? This would give a better sense of the feasibility of proposed approach.\n\nLIMITATIONS:\n- No numerical experiments as acknowledged by the authors that this is pure theoretical work.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-04"}