Title: Achieving $\tilde{\mathcal{O}}(1/N)$ Optimality Gap in Restless Bandits through Gaussian Approximation
Venue: NeurIPS 2025
Accepted: yes
Keywords: Markov Decision Processes

Restless Bandits

Gaussian Approximation

Stochastic Optimization
Abstract: We study the finite-horizon Restless Multi-Armed Bandit (RMAB) problem with $N$ homogeneous arms. Prior work has shown that when an RMAB satisfies a non-degeneracy condition, Linear-Programming-based (LP-based) policies derived from the fluid approximation, which captures the mean dynamics of the system, achieve an exponentially small optimality gap. However, it is common for RMABs to be degenerate, in which case LP-based policies can result in a $\Theta(1/\sqrt{N})$ optimality gap per arm. In this paper, we propose a novel Stochastic-Programming-based (SP-based) policy that, under a uniqueness assumption, achieves an $\tilde{\mathcal{O}}(1/N)$ optimality gap for degenerate RMABs. Our approach is based on the construction of a Gaussian stochastic system that captures not only the mean but also the variance of the RMAB dynamics, resulting in a more accurate approximation than the fluid approximation. We then solve a stochastic program for this system to obtain our policy. This is the first result to establish an $\tilde{\mathcal{O}}(1/N)$ optimality gap for degenerate RMABs.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies an RMAB problem with $N$ homogeneous arms. Prior results based on LP relaxation may result in a larger $\\Theta(1/\\sqrt{N})$ optimality gap when the RMAB problem is degenerate. In contrast, this paper proposes a new Gaussian approximation and stochastic-programming (SP) based policy that can achieve a smaller $\\Theta(1/N)$ optimality gap, which only requires the relaxed solution to be unique. A small 2-state problem with 2 time-steps is used to show the performance advantage of the SP-based policy.\n\nSTRENGTHS AND WEAKNESSES:\nStrength: \n\n1. The idea of using Gaussian approximation to refine the LP-relaxation seems interesting. \n\n2. The result on the $O(1/N)$ optimality gap is a strong. \n\nWeakness:\n\n1. It is unclear how to solve the SP-based policy efficiently.\n\nQUESTIONS:\n1. My main concern is how the SP-based policy can be solved efficiently. The example in the paper involves only 2 per-agent states and 2 time-steps, which is quite small and less practical. With only two per-agent states, the occupancy measure at each time-step can be parameterized by 1 value. Then, backward induction over 2 time-steps can be written as an (expectation) optimization problem over just one variable (in Line 207). This optimization problem still needs to be solved numerically, but is not that difficult. For a general problem with $K$ states, the occupancy measure will form a $K-1$-dimensional hyperplane. With multiple steps, the backward induction will lead to a highly-complex optimization problem in $K-1$ dimension. The paper does not seem to provide a way to solve this general backward induction under Gaussian approximation. \n\nFurther, note that one of the main advantages of LP relaxation (and other related approaches such as Whittle index) is the ability to decompose the problem, which is crucial for overcoming the curse of dimensionality. Specifically, for an $N$-arm problem, solving the whole RMAB involves a backward induction over $O(N^K)$ global states (if we again use the occupancy measure to represent a global state). In contrast, solving the decomposed problem only involves backward induction over $K$ per-agent states, which is much easier. It is unclear how the Gaussian approximation and SP-based policy can benefit from similar decomposition. If such a decomposition is not possible, then even approximating the general backward induction described above will incur complexity exponential in $K$. If that is the case, the Gaussian approximation will have little advantage over the standard backward induction for the original problem in terms of complexity reduction.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper, the authors have studied the finite horizon Restless Multi-armed Bandit (RMAB) problem under the homogeneous arm setting. Existing works on RMAB show that Linear-Programming (LP) based policies derived from the fluid approximation can result in an exponentially small optimality gap, however, under the assumption that the RMAB satisfies a non-degeneracy condition. However, in reality, RMABs are often degenerate, and in that case, LP-based policies can result in $\\Theta(1/\\sqrt{N})$ optimality gap per arm. The authors in this paper have proposed a new Stochastic Programming (SP) based policy that can achieve an $O(1/N)$ optimality gap for degenerate RMABs (for the first time) under the uniqueness assumption. The authors claim that the construction of a Gaussian stochastic system can capture the mean and variance of the RMAB dynamics, and it is more accurate than the fluid approximation.\n\nSTRENGTHS AND WEAKNESSES:\nThe paper is well-written and technically solid. However, more intuitions need to be provided to understand the impact of the result better. More insight is needed on the construction of policy class, as the main results of the paper are based on this class construction. The simulation section can be enriched with simulations on more scenarios.\n\nQUESTIONS:\n1.\tWhat is the motivation behind considering a Gaussian stochastic system as an approximation of the original system? What other kinds of distribution can be chosen to offer a good approximation? Does a Gaussian stochastic system offer the best approximation?\n2.\tWhy is there a need of a $1/\\sqrt{N}$ factor multiplying $Z_h$ in equation (12)? \n3.\tThe meaning of the policy class described in (13) is not clear. What is the physical interpretation behind this class construction?\n I wonder whether it would include a large number of feasible policies. The main result in the paper (Theorem 4.1) heavily depends on this class of policies. However, the significance of the result can be better understood if we get more insight about the class and its cardinality. \n4.      Please explain what is meant by the round operator in Algorithm 1. \n5.\tTheorem 4.1 holds with rounding. Is it possible to extend this theorem to the case without rounding too? \n6.\tTheorem 4.3 assumes the existence of a positive constant $\\epsilon$ independent of $N$. How can we guarantee the existence of such an $\\epsilon$?\n\nLIMITATIONS:\nYes. However, in the conclusion section, limitations of the proposed approach can be explicitly mentioned for more clarity.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

comments: SUMMARY:
Authors study the restless multi-armed bandit problem in the finite horizon setting. They give a new policy and improved theoretical optimality gap in the degenerate setting, based on Gaussian approximation applied to the standard fluid approximation. They support their findings with empirical results.

STRENGTHS AND WEAKNESSES:
Strengths:
* Key new result in RMAB literature, improving the optimality gap while relaxing the non-degeneracy assumption
* Achieved through a new technique based on Gaussian approximation, building on established methods based on fluid approximation
* Detailed illustrative example effectively demonstrates the mathematical intuition

Weaknesses:
* assumes homogenous arms (i.e., all arms have the same MDP probas and rewards)
* lacking comments on the scalability of the approach with respect to the size of the state space and the length of the horizon
* additional motivation for the degenerate setting would improve the significance -- are there convincing real-world examples?

QUESTIONS:
Since the policy is based on the fluid approximation, it would be helpful to give some idea of how much the SG policy can deviate from the base fluid approximation and still retain optimality or feasibility of computation time – what is the factor at play there?

Can the gaussian approximation technique be extended to infinite horizon settings?

Can you comment on how you might relax the homogeneity assumption and still maintain the gap improvement?

Comment on computational complexity of the approach – how it scales with |S| and H? 

Can this approach be extended to the multi-action/WCMDP setting?

LIMITATIONS:
n/a
TITLE: Official Review
CLARITY: 3
IMPACT: 4
ORIGINALITY: 4
DATE: 2025-07-03

comments: SUMMARY:
This paper studies a finite-horizon N-armed restless bandit problem with homogenous arms (identical state space and transition kernel across arms). The authors propose a Gaussian approximation-based approach that tracks the mean as well as variance of the RMAB process, resulting in a more accurate representation of the limiting process compared to standard fluid limit-based approaches. The resulting stochastic programming-based policy is shown to improve sub-optimality gap per arm by a factor of $1/\sqrt{N}$ over LP-based policies (resulting from fluid approximation) in the case of degenerate RMABs.

STRENGTHS AND WEAKNESSES:
Proofs were not carefully checked but the intuition provided is sound and generally easy to follow. The general strengths of this paper in my opinion are:

1. Well-written and instructive to follow.
2. That a higher order approximation leads to a significant improvement (breaking the $1/\sqrt{N}$ barrier) is nice to see.

Not a weakness, but a question/suggestion: The proposed Gaussian approximation bears resemblance to "Diffusion Approximation" techniques common in Operations Research and Stats literature. It would be good if the authors can comment on this and make connections to similar techniques that might have appeared under different names. I am happy to revise my score upwards pending resolution of this point. In particular, I want to be sure that this (or a very similar) problem isn't already addressed in adjacent literature under different technical nomenclature.

Also curious to hear the authors' thoughts on whether the $\mathcal{O}\left( 1/N \right)$ gap can be further reduced (under the same assumptions).

QUESTIONS:
None, please see the previous section.

LIMITATIONS:
Yes
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-03