SlowGuess/NeurIPS_2023_OCR huggingface dataset:
An example. We provide a concrete example illustrating the above arguments. Suppose the state space for each arm is  $\mathbb{S} = \{0,1,\dots ,7\}$ . We label action 1 as the preferred action for states  $0,1,2,3$  and action 0 for the other states. For an arm in state  $s$ , applying the preferred action moves the arm to state  $(s + 1)$  mod 8 with probability  $p_{s,\mathrm{R}}$ , and applying the other action moves the arm to state  $(s - 1)^+$  with probability  $p_{s,\mathrm{L}}$ ; the arm stays at state  $s$  otherwise. One unit of reward is generated when the arm goes from state 7 to state 0. We assume that the budget is  $N / 2$  and set  $\{p_{s,\mathrm{R}},p_{s,\mathrm{L}}\}$  such that the optimal solution of (LP) is  $y^{*}(s,1) = 1 / 8$  for  $s = 0,1,2,3$  and  $y^{*}(s,0) = 1 / 8$  for  $s = 4,5,6,7$ . That is, the optimal state distribution is uniform  $(\mathbb{S})$ , and the optimal single-armed policy  $\bar{\pi}^*$  always takes the preferred action so as to traverse state 7 as often as possible.
To see why this MDP makes the corresponding  $N$ -armed system tricky to control, consider the situation where  $p_{s,\mathrm{L}} >> p_{s,\mathrm{R}}$  and most arms are in state 0. Those arms prefer actions 1 to move towards state 7. However, there are only  $N/2$  units of budget. If we break ties uniformly at random, each arm is not pulled with probability  $1/2$  in each time slot and is likely to return to 0 before they leave  $\{0,1,2,3\}$ . This phenomenon can be seen from Figure 7b in Appendix G.
A similar phenomenon can be observed for LP-Priority policies. Here we consider an LP-Priority in [HF17, BS20, GGY22] that breaks ties based on the Lagrangian-optimal index. In Figure 2, we generate and visualize a sample path under LP-Priority from time 250 to 289, and contrast it with the sample path if the system switches to  $\mathrm{FTVA}(\bar{\pi}^{*})$  from time 250 onwards. We can see that under LP-Priority, although the arms have a strong tendency to move up from state 4 to state 5, they move back when they reach state 5 and thus get stuck at state 4. In contrast, when switching to  $\mathrm{FTVA}(\bar{\pi}^{*})$ , the arrows gradually change to the correct direction, which helps the arms to escape from state 4 and converge to the uniform distribution over the state space. Intuitively, under  $\mathrm{FTVA}(\bar{\pi}^{*})$ , an increasing number of arms
Figure 3: Comparing policies based on virtual states and real states
couple their real states with virtual states over time, which allows these arms to consistently apply the preferred actions afterward. In Appendix G.4, we include more visualizations of the policies.
In Figure 3, we compare the average reward of  $\mathrm{FTVA}(\bar{\pi}^{*})$  with those of the random tie-breaking policy and the LP-Priority policy discussed above. We can see that  $\mathrm{FTVA}(\bar{\pi}^{*})$  is near-optimal, while the other two policies have nearly zero rewards. Further details are provided in Appendix G. Note that while a different tie-breaking rule may solve this particular example with real states, currently there is no known rule that works in general.
4 Theoretical result on optimality gap
In this section, we present our main theoretical result, an upper bound on the conversion loss  $V_{1}^{\bar{\pi}} - V_{N}^{\mathrm{FTVA}(\bar{\pi})}$  for any given single-armed policy  $\bar{\pi}$ . Setting  $\bar{\pi}$  to be an optimal single-armed policy  $\bar{\pi}^{*}$  then leads to an upper bound on the optimality gap of our N-armed policy  $\mathrm{FTVA}(\bar{\pi}^{*})$ . Our result holds under the Synchronization Assumption (SA), which we formally introduce below.
4.1 Synchronization Assumption
SA is imposed on a given single-armed policy  $\bar{\pi}$ . To describe SA, we first define a two-armed system called the leader-and-follower system, which consists of a leader arm and a follower arm. Each arm is associated with the MDP  $(\mathbb{S},\mathbb{A},P,r)$ . At each time step  $t\geq 1$ , the leader arm is in state  $\widehat{S} (t)$  and uses the policy  $\bar{\pi}$  to choose an action  $\widehat{A} (t)$  based on  $\widehat{S} (t)$ ; the follower arm is in state  $S(t)$ , and it takes the action  $A(t) = \widehat{A} (t)$  regardless of  $S(t)$ . The state transitions of the two arms are coupled as follows. If  $S(t) = \widehat{S} (t)$ , then  $S(t + 1) = \widehat{S} (t + 1)$ . If  $S(t)\neq \widehat{S} (t)$ , then  $S(t + 1)$  and  $\widehat{S} (t + 1)$  are sampled independently from  $P(S(t),A(t),\cdot)$  and  $P(\widehat{S} (t),\widehat{A} (t),\cdot)$ , respectively. Note that once the states of the two arms become identical, they stay identical indefinitely.
Given the initial states and actions  $(S(0),A(0),\widehat{S} (0),\widehat{A} (0)) = (s,a,\widehat{s},\widehat{a})\in \mathbb{S}\times \mathbb{A}\times \mathbb{S}\times \mathbb{A}$  , we define the synchronization time as the first time the two states become identical:
$$
\tau^ {\text {s y n c}} (s, a, \widehat {s}, \widehat {a}) \triangleq \min  \{t \geq 0: S (t) = \widehat {S} (t) \}. \tag {9}
$$
Assumption 1 (Synchronization Assumption (SA) for a policy  $\bar{\pi}$ ). We say that a single-armed policy  $\bar{\pi}$  satisfies the Synchronization Assumption (SA) if for any initial states and actions  $(s, a, \widehat{s}, \widehat{a}) \in \mathbb{S} \times \mathbb{A} \times \mathbb{S} \times \mathbb{A}$ , the synchronization time  $\tau^{\mathrm{sync}}(s, a, \widehat{s}, \widehat{a})$  is a stopping time and satisfies
$$
\mathbb {E} \left[ \tau^ {\text {s y n c}} (s, a, \widehat {s}, \widehat {a}) \right] <   \infty . \tag {10}
$$
We view SA as an appealing alternative to UGAP for two reasons. First, there are some instances that satisfy SA but not UGAP. Two such examples are given in Figures 1 and 3. [GGY20] provides more
examples that violate UGAP, all of which can be verified to satisfy SA. Second, SA is easier to verify than UGAP. It has been acknowledged in many prior papers that UGAP lacks intuitive sufficient conditions and can only be checked numerically. In contrast, SA can be efficiently verified in several ways, which is discussed in Appendix C.
4.2 Bounds on conversion loss and optimality gap
We are now ready to state our main theorem.
Theorem 1. Consider an  $N$ -armed RB problem  $(N, \mathbb{S}^N, \mathbb{A}^N, P, r, \alpha N)$  under the single-armed unichain assumption. Let  $\bar{\pi}$  be any single-armed policy satisfying SA. For any  $N \geq 1$ , the conversion loss of FTVA satisfies the upper bound
$$
V _ {1} ^ {\bar {\pi}} - V _ {N} ^ {\mathrm {F T V A} (\bar {\pi})} \leq \frac {r _ {\max } \bar {\tau} _ {\max } ^ {\mathrm {s y n c}}}{\sqrt {N}}, \tag {11}
$$
where  $r_{\max} \triangleq \max_{s \in \mathbb{S}, a \in \mathbb{A}} |r(s, a)|$  and  $\overline{\tau}_{\max}^{\mathrm{sync}} \triangleq \max_{(s, a, \widehat{s}, \widehat{a}) \in \mathbb{S} \times \mathbb{A} \times \mathbb{S} \times \mathbb{A}} \mathbb{E}\left[\tau^{\mathrm{sync}}(s, a, \widehat{s}, \widehat{a})\right]$ .
Consequently, given any optimal single-armed policy  $\bar{\pi}^*$  satisfying SA, for all any  $N\geq 1$  the optimality gap of FTVA  $(\bar{\pi}^{*})$  is upper bounded as
$$
V _ {N} ^ {*} - V _ {N} ^ {\mathrm {F T V A} \left(\bar {\pi} ^ {*}\right)} \leq V _ {1} ^ {\bar {\pi} ^ {*}} - V _ {N} ^ {\mathrm {F T V A} \left(\bar {\pi} ^ {*}\right)} \leq \frac {r _ {\max } \bar {\tau} _ {\max } ^ {\text {s y n c}}}{\sqrt {N}}. \tag {12}
$$
Proof sketch. The proof of Theorem 1 is given in Appendix E. Here we sketch the main ideas of the proof, whose key step involves bounding the conversion loss  $V_{1}^{\bar{\pi}} - V_{N}^{\mathrm{FTVA}(\bar{\pi})}$  using a fundamental tool from queueing theory, the Little's Law [Kle75]. Specifically, we start with the upper bound
$$
\begin{array}{l} V _ {1} ^ {\bar {\pi}} - V _ {N} ^ {\mathrm {F T V A} (\bar {\pi})} = \frac {1}{N} \mathbb {E} \left[ \sum_ {i = 1} ^ {N} r \big (\widehat {S} _ {i} (\infty), \widehat {A} _ {i} (\infty) \big) - \sum_ {i = 1} ^ {N} r \big (S _ {i} (\infty), A _ {i} (\infty) \big) \right] \\ \leq \frac {2 r _ {\max }}{N} \mathbb {E} \left[ \sum_ {i = 1} ^ {N} \mathbb {1} \left\{\left(\widehat {S} _ {i} (\infty), \widehat {A} _ {i} (\infty)\right) \neq \left(S _ {i} (\infty), A _ {i} (\infty)\right) \right\} \right], \tag {13} \\ \end{array}
$$
which holds since the virtual process  $\left(\widehat{S}_i(t),\widehat{A}_i(t)\right)$  of each arm  $i$  follows the single-armed policy  $\bar{\pi}$ . We say an arm  $i$  is a bad arm at time  $t$  if  $\left(\widehat{S}_i(t),\widehat{A}_i(t)\right)\neq \left(S_i(t),A_i(t)\right)$ , and a good arm otherwise. Then  $\mathbb{E}\left[\sum_{i = 1}^{N}\mathbb{1}\Big\{\big(\widehat{S}_i(\infty),\widehat{A}_i(\infty)\big)\neq \big(S_i(\infty),A_i(\infty)\big)\Big\} \right] = \mathbb{E}\big[\# \text{bad arms}\big]$  in steady state.
By Little's Law, we have the following relationship:
$\mathbb{E}\left[\# \text{ bad arms}\right] = \left(\text{rate of generating bad arms}\right) \times \mathbb{E}\left[\text{time duration of a bad arm}\right]$ .
It suffices to bound the two terms on the right hand side. Note that the virtual actions  $\widehat{A}_i(t)$ 's are i.i.d. with mean  $\mathbb{E}[\widehat{A}_i(t)] = \alpha$ ; a standard concentration inequality shows that at most  $\left|\sum_{i=1}^{N} \widehat{A}_i(t) - \alpha N\right| \approx O(\sqrt{N})$  bad arms are generated per time slot. On the other hand, each bad arm stays bad until its real state becomes identical to its virtual state, which occurs in  $O(1)$  time by virtue of SA.
5 Continuous-time restless bandits
In this section, we consider the continuous-time setting. The setup, policy, and theoretical results for this setting parallel those for the discrete-time setting, except that we no longer require SA. Detailed proofs are provided in Appendix F.
Problem setup. The continuous-time restless bandit (CTRB) problem is similar to its discrete-time counterpart (cf. Section 2), except that each single-armed MDP runs in continuous time. In particular, an  $N$ -armed CTRB is given by a tuple  $(N,\mathbb{S}^N,\mathbb{A}^N,G,r,\alpha N)$ , where  $\mathbb{S}$  is the finite state space and  $\mathbb{A} = \{0,1\}$  is the action space of a single arm. In continuous time, each arms dynamics is governed by the transition rates (rather than probabilities)  $G = \{G(s,a,s')\}_{s,s' \in \mathbb{S}, a \in \mathbb{A}}$ , where  $G(s,a,s')$  is the rate of transitioning from state  $s$  to state  $s' \neq s$  upon taking action  $a$ . We again assume that the
transition kernel  $G$  of each arm is unichain. Given the states and actions of all arms, the transitions of different arms are independent from each other. Similarly,  $r(s,a)$  is the instantaneous rate of accumulating reward while taking action  $a$  in state  $s$ . The budget constraint now requires that at any moment of time, the total number of arms taking the active action 1 is equal to  $\alpha N$ .
The objective is again maximizing the long-run average reward, that is,
$$
\underset {\text {p o l i c y} \pi} {\operatorname {m a x i m i z e}} V _ {N} ^ {\pi} \triangleq \lim  _ {T \rightarrow \infty} \frac {1}{T} \int_ {0} ^ {T} \frac {1}{N} \sum_ {i = 1} ^ {N} \mathbb {E} \left[ r \left(S _ {i} ^ {\pi} (t), A _ {i} ^ {\pi} (t)\right)\right] d t \tag {14}
$$
$$
\text {s u b j e c t} \quad \sum_ {i = 1} ^ {N} A _ {i} ^ {\pi} (t) = \alpha N \quad \forall t \geq 0. \tag {15}
$$
Let  $V_{N}^{*} = \sup_{\pi} V_{N}^{\pi}$  denote the optimal value of the above optimization problem.
Single-armed problem. The single-armed problem for CTRB is defined analogously as its discrete-time counterpart (3)-(4). This single-armed problem can again be written as a linear program, where the decision variable  $y(s,a)$  represents the steady-state probability of being in state  $s$  taking action  $a$ :
$$
\underset {\{y (s, a) \} _ {s \in \mathbb {S}, a \in \mathbb {A}}} {\text {m a x i m i z e}} \sum_ {s \in \mathbb {S}, a \in \mathbb {A}} r (s, a) y (s, a) \tag {LP-CT}
$$
$$
\text {s u b j e c t} \quad \sum_ {s \in \mathbb {S}} y (s, 1) = \alpha \tag {16}
$$
$$
\sum_ {s ^ {\prime} \in \mathbb {S}, a \in \mathbb {A}} y \left(s ^ {\prime}, a\right) G \left(s ^ {\prime}, a, s\right) = 0 \quad \forall s \in \mathbb {S} \tag {17}
$$
$$
\sum_ {s \in \mathbb {S}, a \in \mathbb {A}} y (s, a) = 1; \quad y (s, a) \geq 0 \forall s \in \mathbb {S}, a \in \mathbb {A}. \tag {18}
$$
In (17), we use the convention that  $G(s, a, s) = -G(s, a) \triangleq -\sum_{s' \neq s} G(s, a, s')$ . Again, the optimal value of (LP-CT) upper bounds the optimal value for the  $N$ -armed problem, i.e.,  $V_1^{\mathrm{rel}} = V_1^{\bar{\pi}^*} \geq V_N^*$ , where  $\bar{\pi}^*$  is any optimal single-armed policy and can be computed using the same formula in (8). Note that in continuous time, the optimal policy  $\bar{\pi}^*$  is carried out through uniformization [Put05].
Our policy. Our framework for the CTRB, Follow-the-Virtual-Advice-CT (FTVA-CT), is presented in Algorithm 2. FTVA-CT works in a similar fashion as its discrete-time counterpart. It takes a single-armed policy  $\bar{\pi}$  as input, and each arm  $i$  independently simulates a virtual single-armed process  $(\widehat{S}_i(t),\widehat{A}_i(t))$  following  $\bar{\pi}$ . FTVA-CT then chooses the real actions  $A_{i}(t)$ 's to match the virtual actions  $\widehat{A}_i(t)$ 's to the extent allowed by the budget constraint  $\sum_{i = 1}^{N}A_{i}(t) = \alpha N$ .
To run FTVA-CT in continuous time for the  $N$ -armed problem, we use the following uniformization to set discrete decision epochs  $\{t_k\}_{k = 0,1,\ldots}$  for updating the actions and virtual processes. We uniformize at rate  $2Ng_{\mathrm{max}}$  with  $g_{\mathrm{max}} = \max_{s\in \mathbb{S},a\in \mathbb{A}}G(s,a)$ , which is an upper bound on the total transition rate of the real and virtual states in an  $N$ -armed system. We generate a decision epoch either when a real state transitions, or when an independent exponential timer with rate  $2Ng_{\mathrm{max}} - \sum_{i = 1}^{N}G(S_i(t),A_i(t))$  ticks.
The definitions of conversion loss, optimality gap, and asymptotic optimality are the same as the discrete-time setting.
Conversion loss and optimality gap. For a given single-armed policy  $\bar{\pi}$ , we consider a continuous-time version of the leader-and-follower system (cf. Section 4.1). For technical reasons, the initial actions are specified differently from the discrete-time setting. Specifically, we assume that the initial action  $\widehat{A}(0)$  of the leader arm is chosen by  $\bar{\pi}$  based on  $\widehat{S}(0)$ , and the follower's initial action  $A(0)$  equals  $\widehat{A}(0)$ . As before, the follower arm always takes the same action as the leader arm regardless of its own state. Given initial states  $(S(0),\widehat{S}(0)) = (s,\widehat{s})$ , the synchronization time is defined as
$$
\tau^ {\text {s y n c}} (s, \widehat {s}) \triangleq \inf  \{t \geq 0: S (t) = \widehat {S} (t) \}. \tag {19}
$$
We no longer need to impose the Synchronization Assumption, since the unichain assumption automatically implies  $\mathbb{E}\left[\tau^{\mathrm{sync}}(s,\widehat{s})\right] < \infty$  in continuous time—see Lemma 5 in Appendix F.
Algorithm 2 Follow-the-Virtual-Advice-CT (FTVA-CT)
Input:  $(N,\mathbb{S}^N,\mathbb{A}^N,G,r,\alpha N)$  , initial states  $S(0)$  , single-armed policy  $\bar{\pi}$  , max transition rate  $g_{\mathrm{max}}$    
Initialize: Virtual states  $\widehat{S} (0)$  are  $N$  i.i.d. samples following the stationary distribution of  $\bar{\pi}$  .  $t_0 = 0$    
1: for  $k = 0,1,2,\ldots$  do   
2: Independently sample  $\widehat{A}_i(t_k)\gets \bar{\pi} (\cdot |\widehat{S}_i(t_k))$  for each arm  $i\in [N]\triangleright$  Generate virtual actions   
3: if  $\sum_{i = 1}^{N}\widehat{A}_{i}(t_{k})\geq \alpha N$  then Select a set A of  $\alpha N$  arms to activate   
4:  $\mathcal{A}\gets$  a set of  $\alpha N$  arms chosen from  $\{i\colon \widehat{A}_i(t_k) = 1\}$  (any tie-breaking)   
5: else   
6:  $\mathcal{B}\gets$  a set of  $\alpha N - \sum_{i = 1}^{N}\widehat{A}_{i}(t_{k})$  arms chosen from  $\{i\colon \widehat{A}_i(t_k) = 0\}$  (any tie-breaking)   
7:  $\mathcal{A}\gets \{i\colon \widehat{A}_i(t_k) = 1\} \cup \mathcal{B}$    
8: Apply  $A_{i}(t_{k}) = 1$  for each arm  $i\in \mathcal{A}$  , and apply  $A_{i}(t_{k}) = 0$  for each arm  $i\notin \mathcal{A}$    
9: The section below progresses the system to the next decision epoch and updates states   
10:  $g_{k}^{\mathrm{real}}\gets \sum_{i = 1}^{N}G(S_{i}(t_{k}),A_{i}(t_{k}))$    
11:  $\widetilde{t}_{k + 1}\gets t_k +$  a sample from the distribution Exp(2Ngmax-gk real)   
12: if there is an arm  $i^{*}$  whose real state transitions before  $\widetilde{t}_{k + 1}$  then   
13:  $(t_{k + 1},S_{i^*}(t_{k + 1}))\gets$  (transition time, new real state)   
14: if  $\widehat{S}_{i^*}(t_k) = S_{i^*}(t_k)$  and  $\widehat{A}_{i^*}(t_k) = A_{i^*}(t_k)$  then Couple virtual and real states   
15:  $\widehat{S}_{i^*}(t_{k + 1})\gets S_{i^*}(t_{k + 1})$    
16: else No transitions or transitions of virtual states of uncoupled arms   
17:  $t_{k + 1}\gets \widetilde{t}_{k + 1}$    
18:  $g_{k}^{\mathrm{virtual}}\gets \sum_{i = 1}^{N}G(\widehat{S}_{i}(t_{k}),\widehat{A}_{i}(t_{k}))\mathbb{1}\Bigl {\{}\widehat{S}_{i}(t_{k})\neq S_{i}(t_{k})\mathrm{~or~}\widehat{A}_{i}(t_{k})\neq A_{i}(t_{k})\Bigr \}$    
19: With probability  $1 - \frac{g_k^{\mathrm{virtual}}}{2Ng_{\mathrm{max}} - g_k^{\mathrm{real}}}$  continue   
20: Sample a pair  $(i^{*},s^{\prime})\in [N]\times \mathbb{S}$  with probability  $\frac{G(\widehat{S}_i*(t_k),\widehat{A}_i*(t_k),s')\cdot\mathbb{1}\bigl\{\widehat{S}_i*(t_k)\neq s'\bigr\}}{2Ng_{\mathrm{max}} - g_k^{\mathrm{real}}}$    
21: if  $\widehat{S}_{i^*}(t_k)\neq S_{i^*}(t_k)$  or  $\widehat{A}_{i^*}(t_k)\neq A_{i^*}(t_k)$  then  $\widehat{S}_{i^*}(t_{k + 1})\gets s'$    
22: Other real and virtual states stay unchanged
Theorem 2. Consider an  $N$ -armed CTRB  $(N, \mathbb{S}^N, \mathbb{A}^N, G, r, \alpha N)$  under the single-armed unichain assumption. For any single-armed policy  $\bar{\pi}$ , the conversion loss of FTVA-CT is upper bounded as
$$
V _ {1} ^ {\bar {\pi}} - V _ {N} ^ {\mathrm {F T V A} - \mathrm {C T} (\bar {\pi})} \leq \frac {r _ {\max } \left(1 + 2 g _ {\max } \bar {\tau} _ {\max } ^ {\text {s y n c}}\right)}{\sqrt {N}}, \quad \forall N \geq 1, \tag {20}
$$
where  $r_{\max} = \max_{s\in \mathbb{S},a\in \mathbb{A}}|r(s,a)|$ $\overline{\tau}_{\mathrm{max}}^{\mathrm{sync}} = \max_{s\in \mathbb{S},\hat{s}\in \mathbb{S}}\mathbb{E}\left[\tau^{\mathrm{sync}}(s,\widehat{s})\right]$
Consequently, for any optimal single-armed policy  $\bar{\pi}^*$ , the optimality gap of FTVA-CT  $(\bar{\pi}^*)$  satisfies
$$
V _ {N} ^ {*} - V _ {N} ^ {\mathrm {F T V A - C T} \left(\bar {\pi} ^ {*}\right)} \leq V ^ {\bar {\pi} ^ {*}} - V _ {N} ^ {\mathrm {F T V A - C T} \left(\bar {\pi} ^ {*}\right)} \leq \frac {r _ {\max } \left(1 + 2 g _ {\max } \bar {\tau} _ {\max } ^ {\text {s y n c}}\right)}{\sqrt {N}}, \quad \forall N \geq 1. \tag {21}
$$
Theorem 2 establishes an  $O(1 / \sqrt{N})$  optimality gap without requiring UGAP or any additional assumptions beyond the standard unichain condition.
In this paper, we study the average-reward restless bandit problem. We propose a simulation-based framework called Follow-the-Virtual-Advice that converts a single-armed optimal policy into a policy in the original  $N$ -arm system with an  $O(1/\sqrt{N})$  optimality gap. In the discrete-time setting, our results hold under the Synchronization Assumption (SA), a mild and easy-to-verify assumption that covers some problem instances that do not satisfy UGAP. In the continuous-time setting, our results do not require any additional assumptions beyond the standard unichain condition. In both settings, our work is the first to achieve asymptotic optimality without assuming UGAP. It will be interesting in future work to explore the possibility of achieving optimality gaps smaller than  $O(1/\sqrt{N})$  without relying on UGAP or the non-degenerate condition.
Acknowledgement. Y. Hong and W. Wang are supported in part by NSF grants CNS-200773 and ECCS-2145713. Q. Xie is supported in part by NSF grant CNS-1955997 and a J.P. Morgan Faculty Research Award. Y. Chen is supported in part by NSF grants CCF-1704828 and CCF-2233152. This project started when Q. Xie, Y. Chen, and W. Wang were attending the Data-Driven Decision Processes program of the Simons Institute for the Theory of Computing.
[AADJ13] Konstantin Avrachenkov, Urtzi Ayesta, Josu Doncel, and Peter Jacko. Congestion control of tcp flows in internet routers by means of index policy. Computer Networks, 57(17):3463-3478, 2013.
[AB19] Konstantin Avrachenkov and Vivek S. Borkar. A learning algorithm for the whittle index policy for scheduling web crawlers. In Proc. Ann. Allerton Conf. Communication, Control and Computing, pages 1001-1006, Monticello, IL, 2019.
[ABG09] T. W. Archibald, D. P. Black, and K. D. Glazebrook. Indexability and index heuristics for a simple class of inventory routing problems. Oper. Res., 57(2):314-326, 2009.
[ALT19] Samuli Aalto, Pasi Lassila, and Ianire Taboada. Whittle index approach to opportunistic scheduling with partial channel information. Perform. Eval., 136:102052, 2019.
[BS20] David B. Brown and James E. Smith. Index policies and performance bounds for dynamic selection problems. Management Science, 66(7):3029-3050, 2020.
[GGW11] John C. Gittins, K. D. Glazebrook, and Richard Weber. Multi-armed bandit allocation indices. John Wiley & Sons, 2011.
[GGY20] Nicolas Gast, Bruno Gaujal, and Chen Yan. Exponential convergence rate for the asymptotic optimality of whittle index policy. arXiv:2012.09064 [cs.PF], 2020.
[GGY22] Nicolas Gast, Bruno Gaujal, and Chen Yan. LP-based policies for restless bandits: Necessary and sufficient conditions for (exponentially fast) asymptotic optimality. arXiv:2106.10067 [math.OC], 2022.
[Git79] John C. Gittins. Bandit processes and dynamic allocation indices. Journal of the Royal Statistical Society: Series B (Methodological), 41(2):148-164, 1979.
[GJ74] John C. Gittins and D. M. Jones. A dynamic allocation index for the sequential design of experiments. In J. Gani, editor, Progress in Statistics, pages 241-266. North-Holland, Amsterdam, 1974.
[GMA05] K. D. Glazebrook, H. M. Mitchell, and P. S. Ansell. Index policies for the maintenance of a collection of machines by a set of repairmen. European Journal of Operational Research, 165(1):267-284, 2005.
[GNJT23] Abheek Ghosh, Dheeraj Nagaraj, Manish Jain, and Milind Tambe. Indexability is not enough for whittle: Improved, near-optimal algorithms for restless bandits. In Proc. Int. Conf. Autonomous Agents and Multiagent Syst., page 1294–1302, Richland, SC, 2023.
[HF17] Weici Hu and Peter Frazier. An asymptotically optimal index policy for finite-horizon restless bandits. arXiv:1707.00205 [math.OC], 2017.
[HXW23] Yige Hong, Qiaomin Xie, and Weina Wang. Near-optimal stochastic bin-packing in large service systems with time-varying item sizes. Proc. ACM Meas. Anal. Comput. Syst., 7(3), 2023.
[Kle75] Leonard Kleinrock. Queueing Systems. John Wiley & Son, 1975.
[LS20] Tor Lattimore and Csaba Szepesvári. Bandit Algorithms. Cambridge University Press, 2020.
[PT99] Christos H. Papadimitriou and John N. Tsitsiklis. The complexity of optimal queuing network control. Math. Oper. Res., 24(2):293-305, 1999.