Title: Constrained Feedback Learning for Non-Stationary Multi-Armed Bandits
Venue: NeurIPS 2025
Accepted: yes
Keywords: Non-stationary bandits

constrained feedback

query budget

query-reward tradeoff
Abstract: Non-stationary multi-armed bandits (nsMAB) enable agents to adapt to changing environments by incorporating mechanisms to detect and respond to shifts in reward distributions, making them well-suited for dynamic settings. However, existing approaches typically assume that reward feedback is available at every round—an assumption that overlooks many real-world scenarios where feedback is limited. In this paper, we take a significant step forward by introducing a new model of *constrained feedback in non-stationary multi-armed bandits* (ConFee-nsMAB), where the availability of reward feedback is restricted. We propose the first prior-free algorithm—that is, one that does not require prior knowledge of the degree of non-stationarity—that achieves near-optimal dynamic regret in this setting. Specifically, our algorithm attains a dynamic regret of $\tilde {\mathcal{O}}({K^{1/3} V_T^{1/3} T }/{ B^{1/3}})$, where $T$ is the number of rounds, $K$ is the number of arms, $B$ is the query budget, and $V_T$ is the variation budget capturing the degree of non-stationarity.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies non-stationary multi-armed bandits (NSMAB) under the constrained feedback setting, where the reward feedback is limited. Given a query budget $B$,  the total number of reward feedback queries available to the agent over the time horizon $T$ is limited by a reward query budget $B$. This problem is named the constrained feedback in non-stationary multi-armed bandits (CONFEE-NSMAB)\nThere are two main focuses: \"How does the query budget constraint affect the problem difficulty?\" and \"Can we construct an algorithm that works well in CONFEE-NSMAB without knowing the degree of non-stationarity?\".\nThe authors introduced a nearly-optimal algorithm that does not require any knowledge of the degree of non-stationarity.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n- They showed an algorithm that is nearly-optimal for CONFEE-NSMAB. Both upper and lower bounds of regret is provided.\n- Proof sketches are provided, which makes it more convincing.\n\n\nWeakness\n- There is no experiment, which makes it hard to evaluate the advantage of the proposed algorithm.\n- The authors did not provide any distribution-dependent regret.\n- If we do not know anything about the non-stationarity, what is the point of \"detecting\" environmental changes? This part is unclear.\n\nQUESTIONS:\n- If we do not know anything about the non-stationarity, what is the point of \"detecting\" environmental changes? \n- Is deriving a distribution-dependent regret bound too hard for a non-stationary bandit?\n- Even when we know $V_{T}$, this algorithm can be applied to the non-stationary bandit. Isn't it necessary to compare the proposed algorithm with existing algorithms for cases when we know $V_{T}$? Conversely, I think it is valuable to compare the proposed algorithm with existing algorithms with $V_{T}=T$.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-23"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors consider the non-stationary multi-armed bandit problem with an observation budget of $B$ total observations. They design an algorithm which guarantees a dynamic regret bound of $\\widetilde O \\left( K^{1/3} V_T^{1/3} T/B^{1/3} \\right)$ where $V_T$ is the total variation, or the degree of non-stationarity of the environment, and moreover their algorithm does not require prior knowledge of $V_T$. The authors also prove a matching lower bound showing that their algorithm is optimal up to logarithmic factors. The suggested algorithm exhibits a balance between tracking the non-stationarity of the environment while maintaining a low amount of queries in order to not exceed the total budget.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n* The problem of non-stationary MAB with limited observations is interesting and well-motivated, and this seems to be the first work which combines the two variants.\n* The authors support the optimality of their results by exhibiting a matching lower bound.\n* The analysis seems nontrivial, and the authors clearly explain the need of a delicate balance between tracking the changes in the environment while simultaneously using a small number of queries.\n\nWeaknesses:\n\n* There seems to be a discrepancy between the two versions of the statement of Theorem 4.1 - one as presented in the main submission and another in the attached supplementary material. The two versions differ in the regime under which the regret bound holds - in the main submission the authors claim their bound holds as long as $B = \\Omega(T^{2/3})$, and in the supplementary version they claim that it holds as long as $B = \\omega(T^{3/4})$. I am not sure which version is the intended final version. If it the second version, the authors need to be absolutely clear about it as it implies that the main statement as presented in the submission is technically unsound. Even assuming the correct regime is when $B = \\Omega(T^{2/3})$, it still seems to me that this assumption may not be necessary, as the bound itself seems to yield nontrivial dependence on $T$ whenever $B = \\omega(1)$ with respect to $T$. I ask that the authors address this issue, as their response could highly affect my overall rating.\n\nQUESTIONS:\nI refer the authors to my previous point under the \"Weaknesses\" section. To summarize, my question is which of the two versions of Theorem 4.1 is the correct version, and whatever the answer, why is such an assumption necessary, that is, where is it used in the proof.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-25"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses the problem of non-stationary multi-armed bandits under a constrained feedback budget. The authors introduce a new framework, CONFEE-NSMAB, where the constrained feedback is in the form of the total number of times it can query for reward. The authors propose an algorithm for this setting and analyze its regret. The authors also show a lower bound for the problem.\n\nSTRENGTHS AND WEAKNESSES:\nThe paper is generally well-written and easy to follow. The method is motivated well and described clearly. \n\nThe paper studies an important problem that have real-world relevance. It would be helpful if the authors can comment on how non-stationary bandits are applied in the real world v.s. considering it as a full RL problem.\n\nI am concerned about the modeling assumption of a finite variation budget V_T. It would be helpful if the authors can comment on its effect using a concrete real-world example. The algorithm uses a query budget ratio b = ceil(2T/B). It would be helpful if the authors can provide more intuition on the selection of this value. Additionally, more detailed comments on the computational complexity would strengthen the paper. Otherwise the algorithm shows novelty, especially the query allocation mechanism. The technical proofs appear to be sound. \n\nWhile the paper is theory in nature, some comments on the empirical performance would be useful.\n\nQUESTIONS:\nAside from the questions mentioned in the comments above, can the authors comment on (1) the computational complexity of the proposed algorithm, (2) the connection between NSMAB and restless multi-armed bandits [1], and (3) if their algorithm can handle them?\n\n[1] Liu, Haoyang, Keqin Liu, and Qing Zhao. \"Learning in a changing world: Restless multi-armed bandit with unknown dynamics.\" IEEE Transactions on Information Theory 59.3 (2012): 1902-1916.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies non-stationary multi-armed bandits when the learner has a limited querying budget. The model is inspired by the well-known paper by Besbes et al. 2014, where the goal is to maintain a regret guarantee that is parameterized by how much the instance is “non-stationary”. The tweak with respect to this well-studied model is that the learner can only observe the current loss it experiences at most $B_T$ times. Note, in a stationary environment, there is no reason to delay issuing such queries (as the distributions do not change), while in the adversarial setting — where the benchmark is the best fixed action — a valid strategy consists in spreading the queries uniformly.\n\nSTRENGTHS AND WEAKNESSES:\nProviding non-trivial regret guarantees against the best dynamic strategy in online learning is a hard and interesting problem. Moreover, the study of “label-efficient” (i.e., that issues a sublinear number of queries) algorithms is also well motivated and has been studied extensively (see, for instance Minimizing Regret with Label Efficient Prediction, Cesa-Bianchi, Lugosi, and Stoltz, COLT 04).\n\nThe authors provide matching upper and lower bounds, the algorithm is clearly presented, and the technical contribution is non-trivial. At a high level, the approach consists of running in parallel multiple algorithms, each one is tailored to a specific non-stationarity in the reward sequences. There is an extra difficulty given by the fact that the algorithm has to partition the query budget between these parallel algorithms.\n\nQUESTIONS:\nIf we were to drop the requirement that $V_T$ is unknown, would the algorithm by Besbes et al, combined with uniform querying (i.e., at each time step the learner issues a query with probability $B_T/T$) provide the same result? If this is the case, maybe it is worth mentioning, so that it is clear that the difficulty lies in spreading the querying budget between the parallel algorithms.\n\nLIMITATIONS:\nnothing to add here", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-01"}