Title: Restless Bandits with Average Reward: Breaking the Uniform Global Attractor Assumption
Venue: NeurIPS 2023
Accepted: yes
Keywords: restless bandits

average reward MDP

simulation-based method

asymptotic optimality
Abstract: We study the infinite-horizon restless bandit problem with the average reward criterion, in both discrete-time and continuous-time settings.
A fundamental goal is to efficiently compute policies that achieve a diminishing optimality gap as the number of arms, $N$, grows large. 
Existing results on asymptotic optimality all rely on the uniform global attractor property (UGAP), a complex and challenging-to-verify assumption. 
In this paper, we propose a general, simulation-based framework, Follow-the-Virtual-Advice, that converts any single-armed policy into a policy for the original $N$-armed problem. 
This is done by simulating the single-armed policy on each arm and carefully steering the real state towards the simulated state. 
Our framework can be instantiated to produce a policy with an $O(1/\sqrt{N})$ optimality gap. 
In the discrete-time setting, our result holds under a simpler synchronization assumption, which covers some problem instances that violate UGAP. 
More notably, in the continuous-time setting, we do not require \emph{any} additional assumptions beyond the standard unichain condition. 
In both settings, our work is the first asymptotic optimality result that does not require UGAP.

Restless Bandits with Average Reward: Breaking the Uniform Global Attractor Assumption

Abstract

We study the infinite-horizon restless bandit problem with the average reward criterion, in both discrete-time and continuous-time settings. A fundamental goal is to efficiently compute policies that achieve a diminishing optimality gap as the number of arms, N , grows large. Existing results on asymptotic optimality all rely on the uniform global attractor property (UGAP), a complex and challenging-to-verify assumption. In this paper, we propose a general, simulation-based framework, Follow-the-Virtual-Advice , that converts any single-armed policy into a policy for the original N -armed problem. This is done by simulating the single-armed policy on each arm and carefully steering the real state towards the simulated state. Our framework can be instantiated to produce a policy with an O (1 / √ N ) optimality gap. In the discrete-time setting, our result holds under a simpler synchronization assumption, which covers some problem instances that violate UGAP. More notably, in the continuous-time setting, we do not require any additional assumptions beyond the standard unichain condition. In both settings, our work is the first asymptotic optimality result that does not require UGAP.

1 Introduction

The restless bandit (RB) problem is a dynamic decision-making problem that involves a number of Markov decision processes (MDPs) coupled by a constraint. Each MDP, referred to as an arm, has a binary action space, { passive, active } . At every decision epoch, the decision maker is constrained to select a fixed number of arms to activate, with the goal of maximizing the expected reward accrued. The RB problem finds applications across a spectrum of domains, including wireless communication [ALT19], congestion control [AADJ13], queueing models [ABG09], crawling web content [AB19], machine maintenance [GMA05], clinical trials [VBW15], to name a few.

In this paper, we focus on infinite-horizon RBs with the average-reward criterion. Since the exact optimal policy is PSPACE-hard to compute [PT99], it is of great theoretical and practical interest to focus on policies that approximately achieve the optimal value and compute such policies in an efficient matter. The optimality gap of a policy is defined as the difference between its average reward per arm and that of an optimal policy. In a typical asymptotic regime where the number of arms, N , grows large, we say that a policy is asymptotically optimal if its optimality gap is o (1) as N →∞ .

Prior work and the uniform global attractor property assumption. Prior work has studied the celebrated Whittle index policy [Whi88] and LP-Priority policies [Ver16] and established sufficient conditions for their asymptotic optimality [WW90, Ver16, GGY20, GGY22]. One key assumption underpinning all prior work is the uniform global attractor property (UGAP)-also known as globally asymptotic stability-which pertains to the mean-field/fluid limit of the restless bandit system in the

asymptotic limit N →∞ . UGAP stipulates that the system's state distribution in the mean-field limit converges to the optimal state distribution attaining the maximum reward, from any initial distribution. It has been well recognized that UGAP is a highly technical assumption and challenging to verify: the primary way to test UGAP is numerical simulations; see [GGY20] for a detailed discussion.

More recent work studies the rate at which the optimality gap converges to zero. The work [GGY20] and [GGY22] prove a striking O (exp( -cN )) optimality gap for the Whittle index policy and LPPriority policies, respectively, where c is a constant. In addition to UGAP, these results require a non-singularity or non-degenerate condition. We are not aware of any rate of convergence result without assuming UGAP or non-singularity/degeneracy. See Table 1 for a summary.

Therefore, prior work on average-reward restlest bandit leaves two fundamental questions open:

Is it possible to achieve asymptotic optimality without UGAP?

Is is possible to establish a non-trivial convergence rate for the optimality gap in the absence of the non-singular/non-degenerate assumption (and UGAP)?

Our contributions. We consider both the discrete-time and continuous-time settings of the average-reward restless bandit problem. We propose a general, simulation-based framework, Follow-the-Virtual-Advice ( FTVA ) and its continuous-time variant FTVA-CT , which convert any single-armed policy into a policy for the original N -armed problem, with a vanishing performance loss. Our framework can be instantiated to produce a policy with an O (1 / √ N ) optimality gap, under the conditions summarized in Table 1, which we elaborate on later.

Under our framework, computing an asymptotically optimal policy is efficient since it reduces to deriving an optimal single-armed policy, whose complexity is independent of N . The resultant policy can be implemented with a linear-inN computational cost and some of its subroutines can be implemented in a distributed fashion over the arms (see Appendix B for more details).

Our results can also be extended to RBs with heterogeneous arms. See Appendix H for details.

We now elaborate on the conditions in Table 1. In the discrete-time setting, our result holds under a condition called the Synchronization Assumption (SA), in addition to the standard unichain assumption required by all prior work. The SA condition, which is imposed on the MDP associated with a single arm, admits several intuitive sufficient conditions. While it is unclear whether SA subsumes UGAP, we show that there exist problem instances that violate UGAP but satisfy SA. Figure 1 shows one such instance (constructed by [GGY20], described in Appendix G for ease of reference), in which the Whittle Index and LP-Priority policies coincide and have a nondiminishing optimality gap, whereas our policy, named as FTVA ( ¯ π ∗ ), is asymptotically optimal. In addition, our O (1 / √ N ) bound on the optimality gap does not require a non-singularity/non-degeneracy assumption.

More notably, in the continuous-time setting, we completely eliminate the UGAP assumption . We show that our policy FTVA-CT ( ¯ π ∗ ) achieves an O (1 / √ N ) optimality gap under only the standard unichain assumption (which is required by all prior work).

In both settings, our results are the first asymptotic optimality results that do not require UGAP. We point out that UGAP is considered necessary for LP-Priority policies [Ver16, GGY22]. Our policy, FTVA ( ¯ π ∗ ), is not a priority policy. As such, we hope our results open up new directions for developing new classes of RB algorithms that achieve asymptotic optimality without relying on UGAP.

Intuitions. Our algorithm and many existing approaches solve an LP relaxation of the original problem. The solution of the LP induces a policy and gives an 'ideal' distribution for the system state. Existing approaches directly apply the LP policy to the current system state, and then perform a simple rounding of the resulting actions so as to satisfy the budget constraint. When the current system state is far from the ideal distribution, the actual actions after rounding may deviate substantially from the LP solution and thus, in the absence of UGAP, would fail to drive the system to the optimum.

Our FTVA framework, in contrast, prioritizes constraint satisfaction. We apply the LP solution to the state of a simulated system, which is constructed carefully so that the resulting actions satisfy the constraint after a minimal amount of rounding. Consequently, starting from any initial state, our policy steers the system towards the ideal distribution and hence approximates the optimal value.

This method is inspired by the approach in [HXW23], which also involves a simulated system, but for a stochastic bin-packing problem.

Other related work. Another condition extensively discussed in the RB literature is the indexability condition [Whi88], which is necessary for the Whittle index policy to be well-defined, but not required by LP-Priority policies [Ver16]. However, indexability alone does not guarantee the asymptotic optimality of Whittle index.

So far we have discussed prior work on infinite-horizon RBs with average reward. For the related setting of finite-horizon total reward RBs, a line of recent work has established an O (1 / √ N ) optimality gap [HF17, ZCJW19, BS20, GNJT23], and an O (exp( -cN )) gap assuming non-degeneracy [ZF21, GGY22]. For the infinite-horizon discounted reward setting, [ZF22, GNJT23] propose policies with O (1 / √ N ) optimality gap without assuming indexability and UGAP. While these results are not directly comparable to ours, it is of future interest to see if our simulation-based framework can be applied to their settings. For a more detailed discussion of prior work, see Appendix A.

Paper Organization. While our continuous-time result is stronger, the discrete-time setting is more accessible. Therefore, we first discuss the discrete-time setting, which includes the problem statement in Section 2, our proposed framework, Follow-the-Virtual-Advice , in Section 3, and our results on the optimality gap in Section 4. Results for the continuous-time setting are presented in Section 5. We conclude the paper in Section 6. Proofs and additional discussion are given in the appendices.

2 Problem Setup

Consider the infinite-horizon, discrete-time restless bandit problem with N arms indexed by [ N ] ≜ { 1 , 2 , . . . , N } . Each arm is associated with an MDP described by the tuple ( S , A , P, r ) . Here S is the state space, assumed to be finite; A = { 0 , 1 } is the action set, and we say the arm is activated or pulled when action 1 is applied; P : S × A × S → [0 , 1] is the transition kernel, where P ( s, a, s ′ ) is the probability of transitioning from state s to state s ′ upon taking action a ; r = { r ( s, a ) } s ∈ S ,a ∈ A is the reward function, where r ( s, a ) is the reward for taking action a in state s . Throughout the paper, we assume that the transition kernel P is unichain [Put05]; that is, under any Markov policy for this single-armed MDP ( S , A , P, r ) , the induced Markov chain has a single recurrent class. The unichain assumption is standard in most existing work on restless bandits [WW90, GGY20, GGY22, Ver16]. We will discuss relaxing the unichain assumption in Appendix D.

In the above setting, we are subject to a budget constraint that exactly αN arms must be activated in each time step, where α ∈ (0 , 1) is a given constant and αN is assumed to be an integer for simplicity. This N -armed RB problem can be represented by the tuple ( N, S N , A N , P, r, αN ) .

A policy π for the N -armed problem specifies the action for each of the N arms based on the history of states and actions. Under policy π , let S π i ( t ) ∈ S denote the state of the i th arm at time t , and we

call S π ( t ) ≜ ( S π i ( t )) i ∈ [ N ] ∈ S N the system state . Similarly, let A π i ( t ) ∈ A denote the action applied to the i th arm at time t , and let A π ( t ) ≜ ( A π i ( t )) i ∈ [ N ] ∈ A N denote the joint action vector.

The controller's goal is to find a policy that maximizes the long-run average of the total expected reward from all N arms, subject to the budget constraint, assuming full knowledge of the model:

Under the unichain assumption, the value V π N of any policy π is independent of the initial state. Let V ∗ N ≜ sup π V π N denote the optimal value. The optimality gap of π is defined as V ∗ N -V π N . We say a policy π is asymptotically optimal if its optimality gap converges to 0 as N →∞ .

Classical theory guarantees that for a finite-state Markov decision process like an RB, there exists an optimal policy that is Markovian and stationary [Put05]. Nevertheless, the policies we propose are not Markovian policies; rather, they have internal states. Under such a policy π , the system state S π ( t ) together with the internal state form a Markov chain, and the action A π ( t ) depends on both the system and internal states. We design a policy such that this Markov chain has a stationary distribution. Let S π ( ∞ ) and A π ( ∞ ) denote the random elements that follow the stationary distributions of S π ( t ) and A π ( t ) , respectively. Then the average reward of π is equal to V π N = 1 N ∑ N i =1 E [ r ( S π i ( ∞ ) , A π i ( ∞ ))] . In later sections, when the context is clear, we drop the superscript π from S π i and A π i .

3 Follow-the-Virtual-Advice : A simulation-based framework

In this section, we present our framework, Follow-the-Virtual-Advice ( FTVA ). We first describe a single-armed problem , which involves an 'average arm' from the original N -armed problem. We then use the optimal single-armed policy to construct the proposed policy FTVA ( ¯ π ∗ ).

3.1 Single-armed problem

The single-armed problem involves the MDP ( S , A , P, r ) associated with a single arm (say arm 1 without loss of generality), where the budget is satisfied on average . Formally, consider the problem:

The constraint (4) stipulates that the average rate of applying the active action must equal α . Various equivalent forms of this single-armed problem have been considered in prior work [WW90, GGY20, GGY22, Ver16].

By virtue of the unichain assumption, the single-armed problem can be equivalently rewritten as the following linear program, where each decision variable y ( s, a ) represents the steady-state probability that the arm is in state s taking action a :

Algorithm 1 Follow-the-Virtual-Advice ( FTVA ): A simulation-based framework

Here (5) corresponds to the relaxed budget constraint, (6) is the flow balance equation, and (6)-(7) guarantee that y ( s, a ) 's are valid steady-state probabilities.

By standard results for average reward MDPs [Put05], an optimal solution { y ∗ ( s, a ) } s ∈ S ,a ∈ A to (LP) induces an optimal policy ¯ π ∗ for the single-armed problem via the following formula:

Let V rel 1 = V ¯ π ∗ 1 be the optimal value of (LP) and the single-armed problem.

(LP) can be viewed as a relaxation of the N -armed problem. To see this, take any N -armed policy π and set y ( s, a ) to be the fraction of arms in state s taking action a in steady state under π , i.e., y ( s, a ) = E [ 1 N ∑ N i =1 1 { S π i ( ∞ )= s,A π i ( ∞ )= a } ] . Whevener π satisfies the budget constraint (2), { y ( s, a ) } satisfies the relaxed constraint (5) and the consistency constraints (6)-(7). Therefore, the optimal value of (LP) is an upper bound of the optimal value of the N -armed problem: V rel 1 ≥ V ∗ N .

3.2 Constructing the N -armed policy

We now present Follow-the-Virtual-Advice , a simulation-based framework for the N -armed problem. FTVA takes as input any single-armed policy ¯ π that satisfies the relaxed budget constraint (4) and converts it into a N -armed policy, denoted by FTVA ( ¯ π ). Of particular interest is when ¯ π is an optimal single-armed policy, which leads to our result on the optimality gap. Below we introduce the general framework of FTVA without imposing any restriction on the input policy ¯ π.

The proposed policy FTVA ( ¯ π ) has two main components:

Virtual single-armed processes. Each arm i simulates a virtual single-armed process, whose state is denoted as ̂ S i ( t ) , with action ̂ A i ( t ) chosen according to ¯ π . To make the distinction conspicuous, we sometimes refer to the state S i ( t ) and action A i ( t ) in the original N -armed problem as the real state/action. The virtual processes associated with different arms are independent.

Follow the virtual actions. At each time step t , we choose the real actions A i ( t ) 's to best match the virtual actions ̂ A i ( t ) 's, to the extent allowed by the budget constraint ∑ N i =1 A i ( t ) = αN .

FTVA is presented in detail in Algorithm 1. Note that we use an appropriate coupling in Algorithm 1 to ensure that the virtual processes ( ̂ S i ( t ) , ̂ A i ( t )) 's are independent and each follows the Markov chain induced by the single-armed policy ¯ π . FTVA is designed to steer the real states to be close to the virtual states, thereby ensuring a small conversion loss V ¯ π 1 -V FTVA (¯ π ) N . Here recall that V ¯ π 1 is the

0

1

2

3

4

5

6

7

state

250

254

258

262

266

270

274

278

282

286

289

time slot

0

1

2

3

4

5

6

7

state

0

1/8

2/8

3/8

4/8

5/8

6/8

7/8

1

average reward achieved by the input policy ¯ π in the single-armed problem, and that V FTVA (¯ π ) N is the average reward per arm achieved by policy FTVA ( ¯ π ) in the N -armed problem.

3.3 Discussion on FTVA and the role of virtual processes

In this subsection, we provide insights into the mechanism of FTVA and explain the crucial role of the virtual processes. In particular, we contrast FTVA with the alternative approach of directly using the real states to choose actions, e.g., by applying the single-armed policy ¯ π ∗ to each arm's real state. We note that existing policies are essentially real-state-based, so the insights here can also explain why UGAP is necessary for existing policies to have asymptotic optimality.

We first observe that the above two approaches are equivalent in the absence of the budget constraint. In particular, even if the initial virtual state and real state of an arm i are different, they will synchronize (i.e., become identical) in finite time by chance under mild assumptions (see Section 4.1). After this event, if there were no constraint, each arm i will consistently follow the virtual actions, i.e., A i ( t ) = ̂ A i ( t ) = ¯ π ( ̂ S i ( t ) ) for all t , and the virtual states will remain identical to real states.

In the presence of the budget constraint, the arms may not remain synchronized, so the virtual processes become crucial: they guarantee that the virtual actions ̂ A i ( t ) = ¯ π ( ̂ S i ( t ) ) nearly satisfy budget constraint, which allows the real system to approximately follow the virtual actions to remain synchronized. To see this, note that regardless of the current real states, the N virtual states ̂ S 1 , . . . , ̂ S N independently follow the single-armed policy ¯ π ∗ , so, in the long run, each ( ̂ S i ( ∞ ) , ̂ A i ( ∞ )) is distributed per y ∗ ( · , · ) , the optimal solution to (LP). For large N , the sum ∑ N i =1 ̂ A i ( ∞ ) concentrates around its expectation N ∑ s ∈ S y ∗ ( s, 1) = αN and thus tightly satisfy the budget constraint. In contrast, the actions generated by applying ¯ π to the real states are likely to significantly violate the constraint, especially when the empirical distribution of the current real states deviates from y ∗ ( · , · ) .

An example. We provide a concrete example illustrating the above arguments. Suppose the state space for each arm is S = { 0 , 1 , . . . , 7 } . We label action 1 as the preferred action for states 0 , 1 , 2 , 3 , and action 0 for the other states. For an arm in state s , applying the preferred action moves the arm to state ( s +1) mod 8 with probability p s, R, and applying the other action moves the arm to state ( s -1) + with probability p s, L; the arm stays at state s otherwise. One unit of reward is generated when the arm goes from state 7 to state 0 . We assume that the budget is N/ 2 and set { p s, R , p s, L } such that the optimal solution of (LP) is y ∗ ( s, 1) = 1 / 8 for s = 0 , 1 , 2 , 3 and y ∗ ( s, 0) = 1 / 8 for s = 4 , 5 , 6 , 7 . That is, the optimal state distribution is uniform ( S ) , and the optimal single-armed policy ¯ π ∗ always takes the preferred action so as to traverse state 7 as often as possible.

To see why this MDP makes the corresponding N -armed system tricky to control, consider the situation where p s, L >> p s, R and most arms are in state 0 . Those arms prefer actions 1 to move towards state 7 . However, there are only N/ 2 units of budget. If we break ties uniformly at random, each arm is not pulled with probability 1 / 2 in each time slot and is likely to return to 0 before they leave { 0 , 1 , 2 , 3 } . This phenomenon can be seen from Figure 7b in Appendix G.

A similar phenomenon can be observed for LP-Priority policies. Here we consider an LP-Priority in [HF17, BS20, GGY22] that breaks ties based on the Lagrangian-optimal index. In Figure 2, we generate and visualize a sample path under LP-Priority from time 250 to 289 , and contrast it with the sample path if the system switches to FTVA (¯ π ∗ ) from time 250 onwards. We can see that under LP-Priority, although the arms have a strong tendency to move up from state 4 to state 5 , they move back when they reach state 5 and thus get stuck at state 4 . In contrast, when switching to FTVA (¯ π ∗ ) , the arrows gradually change to the correct direction, which helps the arms to escape from state 4 and converge to the uniform distribution over the state space. Intuitively, under FTVA (¯ π ∗ ) , an increasing number of arms

couple their real states with virtual states over time, which allows these arms to consistently apply the preferred actions afterward. In Appendix G.4, we include more visualizations of the policies.

In Figure 3, we compare the average reward of FTVA (¯ π ∗ ) with those of the random tie-breaking policy and the LP-Priority policy discussed above. We can see that FTVA (¯ π ∗ ) is near-optimal, while the other two policies have nearly zero rewards. Further details are provided in Appendix G. Note that while a different tie-breaking rule may solve this particular example with real states, currently there is no known rule that works in general.

4 Theoretical result on optimality gap

In this section, we present our main theoretical result, an upper bound on the conversion loss V ¯ π 1 -V FTVA (¯ π ) N for any given single-armed policy ¯ π . Setting ¯ π to be an optimal single-armed policy ¯ π ∗ then leads to an upper bound on the optimality gap of our N-armed policy FTVA (¯ π ∗ ) . Our result holds under the Synchronization Assumption (SA), which we formally introduce below.

4.1 Synchronization Assumption

SA is imposed on a given single-armed policy ¯ π . To describe SA, we first define a two-armed system called the leader-and-follower system, which consists of a leader arm and a follower arm. Each arm is associated with the MDP ( S , A , P, r ) . At each time step t ≥ 1 , the leader arm is in state ̂ S ( t ) and uses the policy ¯ π to chooses an action ̂ A ( t ) based on ̂ S ( t ) ; the follower arm is in state S ( t ) , and it takes the action A ( t ) = ̂ A ( t ) regardless of S ( t ) . The state transitions of the two arms are coupled as follows. If S ( t ) = ̂ S ( t ) , then S ( t +1) = ̂ S ( t +1) . If S ( t ) = ̂ S ( t ) , then S ( t +1) and ̂ S ( t +1) are sampled independently from P ( S ( t ) , A ( t ) , · ) and P ( ̂ S ( t ) , ̂ A ( t ) , · ) , respectively. Note that once the states of the two arms become identical, they stay identical indefinitely.

̸

Given the initial states and actions ( S (0) , A (0) , ̂ S (0) , ̂ A (0)) = ( s, a, ̂ s, ̂ a ) ∈ S × A × S × A , we define the synchronization time as the first time the two states become identical:

Assumption 1 (Synchronization Assumption (SA) for a policy ¯ π ) . We say that a single-armed policy ¯ π satisfies the Synchronization Assumption (SA) if for any initial states and actions ( s, a, ̂ s, ̂ a ) ∈ S × A × S × A , the synchronization time τ sync ( s, a, ̂ s, ̂ a ) is a stopping time and satisfies

We view SA as an appealing alternative to UGAP for two reasons. First, there are some instances that satisfy SA but not UGAP. Two such examples are given in Figures 1 and 3. [GGY20] provides more

examples that violate UGAP, all of which can be verified to satisfy SA. Second, SA is easier to verify than UGAP. It has been acknowledged in many prior papers that UGAP lacks intuitive sufficient conditions and can only be checked numerically. In contrast, SA can be efficiently verified in several ways, which is discussed in Appendix C.

4.2 Bounds on conversion loss and optimality gap

We are now ready to state our main theorem.

Theorem 1. Consider an N -armed RB problem ( N, S N , A N , P, r, αN ) under the single-armed unichain assumption. Let ¯ π be any single-armed policy satisfying SA. For any N ≥ 1 , the conversion loss of FTVA satisfies the upper bound

where r max ≜ max s ∈ S ,a ∈ A | r ( s, a ) | and τ sync max ≜ max ( s,a, ̂ s, ̂ a ) ∈ S × A × S × A E [ τ sync ( s, a, ̂ s, ̂ a )] .

Consequently, given any optimal single-armed policy ¯ π ∗ satisfying SA, for all any N ≥ 1 the optimality gap of FTVA (¯ π ∗ ) is upper bounded as

Proof sketch. The proof of Theorem 1 is given in Appendix E. Here we sketch the main ideas of the proof, whose key step involves bounding the conversion loss V ¯ π 1 -V FTVA (¯ π ) N using a fundamental tool from queueing theory, the Little's Law [Kle75]. Specifically, we start with the upper bound

̸

which holds since the virtual process ( ̂ S i ( t ) , ̂ A i ( t ) ) of each arm i follows the single-armed policy ¯ π . We say an arm i is a bad arm at time t if ( ̂ S i ( t ) , ̂ A i ( t ) ) = ( S i ( t ) , A i ( t ) ) , and a good arm otherwise. Then E [ ∑ N i =1 ✶ { ( ̂ S i ( ∞ ) , ̂ A i ( ∞ ) ) = ( S i ( ∞ ) , A i ( ∞ ) ) }] = E [ # bad arms ] in steady state. By Little's Law, we have the following relationship:

̸

E [ # bad arms ] = (rate of generating bad arms) × E [ time duration of a bad arm ] .

It suffices to bound the two terms on the right hand side. Note that the virtual actions ̂ A i ( t ) 's are i.i.d. with mean E [ ̂ A i ( t )] = α ; a standard concentration inequality shows that at most ∣ ∣ ∑ N i =1 ̂ A i ( t ) -αN ∣ ∣ ≈ O ( √ N ) bad arms are generated per time slot. On the other hand, each bad arm stays bad until its real state becomes identical to its virtual state, which occurs in O (1) time by virtue of SA.

5 Continuous-time restless bandits

In this section, we consider the continuous-time setting. The setup, policy, and theoretical results for this setting parallel those for the discrete-time setting, except that we no longer require SA. Detailed proofs are provided in Appendix F.

̸

Problem setup. The continuous-time restless bandit (CTRB) problem is similar to its discrete-time counterpart (cf. Section 2), except that each single-armed MDP runs in continuous time. In particular, an N -armed CTRB is given by a tuple ( N, S N , A N , G, r, αN ) , where S is the finite state space and A = { 0 , 1 } is the action space of a single arm. In continuous time, each arms dynamics is governed by the transition rates (rather than probabilities) G = { G ( s, a, s ′ ) } s,s ′ ∈ S ,a ∈ A , where G ( s, a, s ′ ) is the rate of transitioning from state s to state s ′ = s upon taking action a . We again assume that the

̸

transition kernel G of each arm is unichain. Given the states and actions of all arms, the transitions of different arms are independent from each other. Similarly, r ( s, a ) is the instantaneous rate of accumulating reward while taking action a in state s . The budget constraint now requires that at any moment of time, the total number of arms taking the active action 1 is equal to αN .

The objective is again maximizing the long-run average reward, that is,

Let V ∗ N = sup π V π N denote the optimal value of the above optimization problem.

Single-armed problem. The single-armed problem for CTRB is defined analogously as its discretetime counterpart (3)-(4). This single-armed problem can again be written as a linear program, where the decision variable y ( s, a ) represents the steady-state probability of being in state s taking action a :

̸

In (17), we use the convention that G ( s, a, s ) = -G ( s, a ) ≜ -∑ s ′ = s G ( s, a, s ′ ) . Again, the optimal value of (LP-CT) upper bounds the optimal value for the N -armed problem, i.e., V rel 1 = V ¯ π ∗ 1 ≥ V ∗ N , where ¯ π ∗ is any optimal single-armed policy and can be computed using the same formula in (8). Note that in continuous time, the optimal policy ¯ π ∗ is carried out through uniformization [Put05].

Our policy. Our framework for the CTRB, Follow-the-Virtual-Advice-CT ( FTVA-CT ), is presented in Algorithm 2. FTVA-CT works in a similar fashion as its discrete-time counterpart. It takes a single-armed policy ¯ π as input, and each arm i independently simulates a virtual single-armed process ( ̂ S i ( t ) , ̂ A i ( t )) following ¯ π . FTVA-CT then chooses the real actions A i ( t ) 's to match the virtual actions ̂ A i ( t ) 's to the extent allowed by the budget constraint ∑ N i =1 A i ( t ) = αN .

To run FTVA-CT in continuous time for the N -armed problem, we use the following uniformization to set discrete decision epochs { t k } k =0 , 1 ,... for updating the actions and virtual processes. We uniformize at rate 2 Ng max with g max = max s ∈ S ,a ∈ A G ( s, a ) , which is an upper bound on the total transition rate of the real and virtual states in an N -armed system. We generate a decision epoch either when a real state transitions, or when an independent exponential timer with rate 2 Ng max -∑ N i =1 G ( S i ( t ) , A i ( t )) ticks.

The definitions of conversion loss, optimality gap, and asymptotic optimality are the same as the discrete-time setting.

Conversion loss and optimality gap. For a given single-armed policy ¯ π , we consider a continuoustime version of the leader-and-follower system (cf. Section 4.1). For technical reasons, the initial actions are specified differently from the discrete-time setting. Specifically, we assume that the initial action ̂ A (0) of the leader arm is chosen by ¯ π based on ̂ S (0) , and the follower's initial action A (0) equals ̂ A (0) . As before, the follower arm always takes the same action as the leader arm regardless of its own state. Given initial states ( S (0) , ̂ S (0)) = ( s, ̂ s ) , the synchronization time is defined as

We no longer need to impose the Synchronization Assumption, since the unichain assumption automatically implies E [ τ sync ( s, ̂ s )] < ∞ in continuous time-see Lemma 5 in Appendix F.

Algorithm 2 Follow-the-Virtual-Advice-CT ( FTVA-CT )

Input: ( N, S N , A N , G, r, αN ) , initial states S (0) , single-armed policy ¯ π , max transition rate g max Initialize: Virtual states ̂ S (0) are N i.i.d. samples following the stationary distribution of ¯ π ; t 0 = 0

1: for k = 0 , 1 , 2 , . . . do

2: Independently sample ̂ A i ( t k ) ← ¯ π ( ·| ̂ S i ( t k )) for each arm i ∈ [ N ] ▷ Generate virtual actions

3:

Select a set of

arms to activate

αN

▷

αN

A

then if

∑

N

i

=1

A

̂

i

(

t

k

)

4: A ← a set of αN arms chosen from { i : ̂ A i ( t k ) = 1 } (any tie-breaking)

5:

6:

N

i

=1

else

a set of

αN

A

-

∑

8: Apply A i ( t k ) = 1 for each arm i ∈ A , and apply A i ( t k ) = 0 for each arm i / ∈ A

9: ▷ The section below progresses the system to the next decision epoch and updates states

11:

k

a sample from the distribution Exp

+

12:

13:

14:

15:

16:

(2

Ng

-

whose real state transitions before

))

+1

t

there is an arm

(

k

+1

i

t

, S

(

t

∗

k

if

S

i

̂

∗

S

˜

+1

k

t

(transition time, new real state)

and

k

i

k

∗

(

t

) =

A

(

t

)

(

̂

A

+1

i

)

(

t

k

i

̂

∗

(

) =

t

k

+1

i

∗

+1

S

i

∗

)

(

t

k

)

S

i

∗

t

k

else ▷ No transitions or transitions of virtual states of uncoupled arms

̸

̸

̸

real

̸

1

-

,

continue

22: Other real and virtual states stay unchanged

Theorem 2. Consider an N -armed CTRB ( N, S N , A N , G, r, αN ) under the single-armed unichain assumption. For any single-armed policy ¯ π , the conversion loss of FTVA-CT is upper bounded as

where r max = max s ∈ S ,a ∈ A | r ( s, a ) | , τ sync max = max s ∈ S , ̂ s ∈ S E [ τ sync ( s, ̂ s )] .

Consequently, for any optimal single-armed policy ¯ π ∗ , the optimality gap of FTVA-CT (¯ π ∗ ) satisfies

Theorem 2 establishes an O (1 / √ N ) optimality gap without requiring UGAP or any additional assumptions beyond the standard unichain condition.

6 Conclusion

In this paper, we study the average-reward restless bandit problem. We propose a simulation-based framework called Follow-the-Virtual-Advice that converts a single-armed optimal policy into a policy in the original N -arm system with an O (1 / √ N ) optimality gap. In the discrete-time setting, our results hold under the Synchronization Assumption (SA), a mild and easy-to-verify assumption that covers some problem instances that do not satisfy UGAP. In the continuous-time setting, our results do not require any additional assumptions beyond the standard unichain condition. In both settings, our work is the first to achieve asymptotic optimality without assuming UGAP. It will be interesting in future work to explore the possibility of achieving optimality gaps smaller than O (1 / √ N ) without relying on UGAP or the non-degenerate condition.

∗

then real

k

g

)

then

▷

Couple virtual and real states

t

˜

k

if

g

2

virtual

k

-

i

̂

(

t

g

k

)

arms chosen from

{

i

:

A

i

̂

(

t

k

}

) = 0

(any tie-breaking)

̸

Acknowledgement. Y. Hong and W. Wang are supported in part by NSF grants CNS-200773 and ECCS-2145713. Q. Xie is supported in part by NSF grant CNS-1955997 and a J.P. Morgan Faculty Research Award. Y. Chen is supported in part by NSF grants CCF-1704828 and CCF-2233152. This project started when Q. Xie, Y. Chen, and W. Wang were attending the Data-Driven Decision Processes program of the Simons Institute for the Theory of Computing.

References

[AADJ13] Konstantin Avrachenkov, Urtzi Ayesta, Josu Doncel, and Peter Jacko. Congestion control of tcp flows in internet routers by means of index policy. Computer Networks , 57(17):3463-3478, 2013.

[AB19] Konstantin Avrachenkov and Vivek S. Borkar. A learning algorithm for the whittle index policy for scheduling web crawlers. In Proc. Ann. Allerton Conf. Communication, Control and Computing , pages 1001-1006, Monticello, IL, 2019.

[ABG09] T. W. Archibald, D. P. Black, and K. D. Glazebrook. Indexability and index heuristics for a simple class of inventory routing problems. Oper. Res. , 57(2):314-326, 2009.

[ALT19] Samuli Aalto, Pasi Lassila, and Ianire Taboada. Whittle index approach to opportunistic scheduling with partial channel information. Perform. Eval. , 136:102052, 2019.

[BS20] David B. Brown and James E. Smith. Index policies and performance bounds for dynamic selection problems. Management Science , 66(7):3029-3050, 2020.

[GGW11] John C. Gittins, K. D. Glazebrook, and Richard Weber. Multi-armed bandit allocation indices . John Wiley & Sons, 2011.

[GGY20] Nicolas Gast, Bruno Gaujal, and Chen Yan. Exponential convergence rate for the asymptotic optimality of whittle index policy. arXiv:2012.09064 [cs.PF] , 2020.

[GGY22] Nicolas Gast, Bruno Gaujal, and Chen Yan. LP-based policies for restless bandits: Necessary and sufficient conditions for (exponentially fast) asymptotic optimality. arXiv:2106.10067 [math.OC] , 2022.

[Git79] John C. Gittins. Bandit processes and dynamic allocation indices. Journal of the Royal Statistical Society: Series B (Methodological) , 41(2):148-164, 1979.

[GJ74] John C. Gittins and D. M. Jones. A dynamic allocation index for the sequential design of experiments. In J. Gani, editor, Progress in Statistics , pages 241-266. North-Holland, Amsterdam, 1974.

[GMA05] K. D. Glazebrook, H. M. Mitchell, and P. S. Ansell. Index policies for the maintenance of a collection of machines by a set of repairmen. European Journal of Operational Research , 165(1):267-284, 2005.

[GNJT23] Abheek Ghosh, Dheeraj Nagaraj, Manish Jain, and Milind Tambe. Indexability is not enough for whittle: Improved, near-optimal algorithms for restless bandits. In Proc. Int. Conf. Autonomous Agents and Multiagent Syst. , page 1294-1302, Richland, SC, 2023.

[HF17] Weici Hu and Peter Frazier. An asymptotically optimal index policy for finite-horizon restless bandits. arXiv:1707.00205 [math.OC] , 2017.

[HXW23] Yige Hong, Qiaomin Xie, and Weina Wang. Near-optimal stochastic bin-packing in large service systems with time-varying item sizes. Proc. ACM Meas. Anal. Comput. Syst. , 7(3), 2023.

[Kle75] Leonard Kleinrock. Queueing Systems . John Wiley & Son, 1975.

[LS20] Tor Lattimore and Csaba Szepesvári. Bandit Algorithms . Cambridge University Press, 2020.

[PT99] Christos H. Papadimitriou and John N. Tsitsiklis. The complexity of optimal queuing network control. Math. Oper. Res. , 24(2):293-305, 1999.

[Put05] Martin L Puterman. Markov decision processes: Discrete stochastic dynamic programming . John Wiley & Sons, 2005.

[VBW15] Sofía S. Villar, Jack Bowden, and James Wason. Multi-armed Bandit Models for the Optimal Design of Clinical Trials: Benefits and Challenges. Statistical Science , 30(2):199 - 215, 2015.

[Ver16] I. M. Verloop. Asymptotically optimal priority policies for indexable and nonindexable restless bandits. Ann. Appl. Probab. , 26(4):1947-1995, 2016.

[Whi88] Peter Whittle. Restless bandits: activity allocation in a changing world. J. Appl. Probab. , 25:287 - 298, 1988.

[WW90] Richard R. Weber and Gideon Weiss. On an index policy for restless bandits. J. Appl. Probab. , 27(3):637-648, 1990.

[ZCJW19] Gabriel Zayas-Cabán, Stefanus Jasin, and Guihua Wang. An asymptotically optimal heuristic for general nonstationary finite-horizon restless multi-armed, multi-action bandits. Advances in Applied Probability , 51:745-772, 2019.

[ZF21] Xiangyu Zhang and Peter I. Frazier. Restless bandits with many arms: Beating the central limit theorem. arXiv:2107.11911 [math.OC] , 2021.

[ZF22] Xiangyu Zhang and Peter I. Frazier. Near-optimality for infinite-horizon restless bandits with many arms. arXiv:2203.15853 [cs.LG] , 2022.

Appendices

A More related work

In this section, we provide additional discussion on related work.

Relationship between the infinite-horizon average-reward setting and the finite-horizon total-reward/infinite-horizon discounted-reward setting. While infinite-horizon average-reward RBs, finite-horizon total-reward RBs, and infinite-horizon discounted-reward RBs are three different settings, one may think that the approaches in the other two settings apply to the infinite-horizon average-reward setting by taking a large enough horizon in the finite-horizon setting or letting the discount factor to go to 1 in the infinite-horizon discounted-reward setting. Here we briefly discuss the fundamental difference between the infinite-horizon average-reward setting and the other two settings for RB problems. For simplicity, we will refer to the other two settings as the finite-horizon/discounted-reward setting.

From the algorithm design perspective, the finite-horizon/discounted-reward setting focuses on optimizing the transient performance, while the infinite-horizon average reward setting focuses on optimizing the steady-state performance. This distinction has a big impact on the algorithm design and the complexity. To see this, we note that both the finite-horizon/discounted-reward setting [HF17, ZCJW19, BS20, GNJT23, ZF21, ZF22, GGY22] and the infinite-horizon averagereward setting [Whi88, WW90, Ver16, GGY20, GGY22] utilize certain forms of linear programming (LP) relaxations to design algorithms. However, the LPs utilized differ substantially. In the finitehorizon/discounted-reward setting, the LP optimizes the future trajectory for a predetermined number of time steps. In contrast, in the infinite-horizon average-reward setting, the LP solves for the optimal state-action frequency in steady state, which can be seen as a fixed point rather than a trajectory. As a result, the number of variables and constraints of the LP for the finite-horizon/discounted-reward setting scales with the number of time steps, which is not the case for the infinite-horizon averagereward setting. On the other hand, because the LP relaxation for the infinite-horizon average-reward setting does not take into account the transient behavior, it provides less information, making the algorithm design more tricky. Consequently, the policies in prior work require the strong assumption of UGAP to achieve asymptotic optimality.

From the analysis perspective, the infinite-horizon average-reward setting requires a more careful analysis of the long-term effect of actions. To see this, note that the optimality gap bounds in the finite-horizon/discounted-reward setting have at least a quadratic dependency on the (effective) horizon [see, e.g. ZF22, GGY22]. Consequently, when translated to the bounds on the optimality gap of average reward per time slot, those bounds diverge to infinity as the time horizon goes to infinity.

Relationship between the RB problem and other bandit problems. While the exact optimality of the RB problem is in general intractable, there is a special case that has been solved optimally. Specifically, consider the case when an arm stays in the same state when it is not pulled, and only one arm is pulled at a time. The optimal policy for this special case is the celebrated Gittins index policy [GJ74, Git79]. A more recent reference on this topic is [GGW11].

The RB problem falls within the broader class of bandit problems, for which different formulations exist including stochastic bandits, adversarial bandits, and Bayesian bandits. The common theme in these formulations is to find a reward-maximizing strategy of pulling arms in the presence of uncertainty in the arms' rewards; see the book [LS20] for a comprehensive overview. Among these formulations, closely related to RBs is the Bayesian bandit problem, where Bayesian posteriors are used to model knowledge of unknown reward distributions. The Bayesian posterior can be seen as a state with known transition probabilities, hence the Bayesian bandit problem can be analyzed by applying tools from RBs. Examples demonstrating this connection can be found in [BS20].

B Discussion on the computational costs of FTVA

FTVA defined in Algorithm 1 can be computed and implemented efficiently.

First, computing FTVA requires solving (LP) once. Given that (LP) does not depend on N , the computational cost here is a constant.

Next, we examine the computational cost of implementing FTVA .

Each arm simulates a virtual process. Because the computation required for each simulation does not scale with N , the overall computational cost here is linear in N .

In each time step, FTVA selects arms from a subset of arms to determine the real actions. The computational cost for this selection process scales linearly with N .

Combining these components, we can see that the computational cost of implementing FTVA is linear in N .

Additionally, we note that the simulation of the virtual processes is independent across the arms, so they can be implemented in a fully distributed manner. However, when taking real actions, the policy needs to know the virtual actions from all arms, so this step cannot be implemented distributedly.

C Sufficient conditions for Synchronization Assumption (SA)

Recall that for the discrete-time setting, our bound on the conversion loss in Theorem 1 holds when the input single-armed policy ¯ π satisfies the Synchronization Assumption (SA). SA stipulates that the two arms in the leader-and-follower system (cf. Section 4.1) under the policy ¯ π will reach the same state in finite expected time. All other existing work requires the UGAP assumption, which pertains to the global behavior of a non-linear dynamic system and has no known sufficient conditions that are easily checkable. In comparison, SA is a reachability assumption imposed on a finite state Markov chain and thus substantially simpler.

To provide a more intuitive understanding of SA, in this section we present several sufficient conditions for SA to hold. These conditions involve the recurrent classes in the Markov chains induced by certain single-armed policies as well as the existence of self-loops or cycles in the corresponding transition diagrams. Such conditions can be verified, often in a straightforward manner, by inspecting the transition probabilities. As self-loops and cycles are common in many classes of MDPs, these conditions showcase that SA is a relatively mild assumption. We emphasize that the sufficient conditions presented in this section are not exhaustive. They serve as illustrative examples of when SA holds and offer insights on the nature of synchronization.

The discussion in this section pertains to the discrete-time setting. We reiterate that our results for the continuous-time setting do not require SA.

C.1 Preliminaries

Before presenting the sufficient conditions, we first prove a few preliminary facts. The first one is a basic property of unichain MDPs.

̸

Proposition 1. Consider two arbitrary Markovian policies ¯ π and ¯ π ′ for the unichain MDP ( S , A , P, r ) . Let the recurrent classes of the two policies ¯ π and ¯ π ′ be S and S ′ , respectively. Then S ∩ S ′ = ∅ .

Proof. We prove this proposition by contradiction. Suppose S ∩ S ′ = ∅ . We define a new policy ¯ π ′′ as

̸

Then S and S ′ are two distinct recurrent classes under ¯ π ′′ , because by definition of ¯ π ′′ , an arm with the initial state in S remains in S , so it cannot reach S ′ ; similarly an arm cannot reach S from S ′ . The existence of two recurrent classes under ¯ π ′′ contradicts the unichain condition. Therefore, we must have S ∩ S ′ = ∅ .

The next proposition provides a convenient way for verifying SA and is used for establishing other sufficient conditions.

Proposition 2. Consider the leader-and-follower system under the policy ¯ π . If there exists some t < ∞ such that for all initial states and actions ( s, a, ̂ s, ̂ a ) ∈ S × A × S × A ,

then Synchronization Assumption holds for the policy ¯ π .

Proof. We claim that for any ( s, a, ̂ s, ̂ a ) ∈ S × A × S × A ,

where the max s ′ ,a ′ , ̂ s ′ , ̂ a ′ is taken over S × A × S × A . We prove this claim by induciton on k . The base case of k = 1 is given. Suppose we have proved (22) for a certain k , then

̸

where in the second equality we have used the Markov property of the system, and in the last inequality we apply the induction hypothesis. We have proved (22).

The bound on the expectation E [ τ sync ( s, a, ̂ s, ̂ a )] follows by summing the tail bound (22) over k :

We have established SA and thereby finished the proof.

Remark 1 . Proposition 2 implies that SA only requires that the Markov chain of the leader-andfollower system, ( S ( t ) , ̂ S ( t )) , can reach the subset of states { ( s, ̂ s ) ∈ S 2 : s = ̂ s } from any initial state. As a result, SA can be efficiently verified using path-finding graph algorithms on the state transition diagram of the Markov chain.

C.2 Sufficient conditions based on self-loops

The unichain condition and Proposition 1 guarantee that there are some states that the leader arm and the follower arm will both visit for a positive fraction of times. 2 If we can further show that the two arms have a positive probability to visit one of those states at the same time , then Proposition 2 can be applied to verify SA. A natural sufficient condition is the existence of self-loops, which guarantees that with a positive probability, the arm reaching those states first will wait for the other arm to come to the same state.

Proposition 3. Consider the single-armed problem with the unichain MDP ( S , A , P, r ) and budget α ∈ (0 , 1) . Let ¯ π be a single-armed policy with recurrent class S . If P ( s, 0 , s ) > 0 and P ( s, 1 , s ) > 0 for all s ∈ S , then Synchronization Assumption holds for the policy ¯ π .

Proposition 3 assumes that all states in the recurrent class of the policy ¯ π have self-loops. We can actually require fewer self-loops if we can characterize the recurrent states of the follower arm. In particular, let ¯ π 1 denote the all-one policy , the policy that applies action 1 in all states. Observe that if the leader arm applies action 1 for a sufficiently long time, the follower arm will also apply action 1 and effectively follows ¯ π 1 . The recurrent class of the policy ¯ π 1 must intersect the recurrent class of ¯ π , so it suffices to have self-loops in the intersection of these two recurrent classes. The idea is formalized in Proposition 4.

leader

leader

2

2

3

b

follower

follower

S

a

S

S

1

2

3

2

S

S1

S

S1

Proposition 4. Consider the single-armed problem with the unichain MDP ( S , A , P, r ) and budget α ∈ (0 , 1) . Let ¯ π be a single-armed policy with recurrent class S . Let the recurrent class of the all-one policy ¯ π 1 be denoted as S 1 . If the following conditions hold:

There exists s a ∈ S such that ¯ π (1 | s a ) > 0 and P ( s a , 1 , s a ) > 0 ;

There exists s b ∈ S ∩ S 1 such that P ( s b , 0 , s b ) > 0 and P ( s b , 1 , s b ) > 0 ,

then Synchronization Assumption holds for the policy ¯ π .

Proposition 5 below gives another sufficient condition, which only requires one self-loop. Note that this condition does not subsume the one in Proposition 4.

Proposition 5. Consider the single-armed problem with the unichain MDP ( S , A , P, r ) and budget α ∈ (0 , 1) . Let ¯ π be a single-armed policy with recurrent class S . Let the recurrent class of the all-one policy ¯ π 1 be denoted as S 1 . If there exists s ∗ ∈ S ∩ S 1 such that ¯ π (1 | s ∗ ) > 0 and P ( s ∗ , 1 , s ∗ ) > 0 , then Synchronization Assumption holds for the policy ¯ π .

We remark that Propositions 4 and 5 have analogous versions that are stated in terms of the all-zero policy ¯ π 0 , the policy that applies action 0 in all states. We omit the details.

Now we prove the above propositions. Note that Proposition 3 is strictly weaker than Proposition 4: because the single-armed policy ¯ π satisfies the budget constraint with α ∈ (0 , 1) , there must be a state s a ∈ S such that ¯ π (1 | s a ) > 0 . Therefore, we only need to prove Propositions 4 and 5.

Proof of Proposition 4. Given any initial states and actions ( S (0) , A (0) , ̂ S (0) , ̂ A (0)) = ( s, a, ̂ s, ̂ a ) ∈ S × A × S × A , we construct a sequence of positive probability events that leads to S ( t ) = ̂ S ( t ) :

The leader arm reaches the state s a ∈ S after t 1 time slots;

The leader arm stays in state s a and keeps applying action 1 for t 2 times slots; meanwhile, the follower arm also applies action 1 and reaches the state s b ∈ S ∩ S 1 ;

The leader arm reaches the state s b in another t 3 time slots; meanwhile, the follower arm stays in the state s b , so the two arms synchronize.

The transitions of the two arms during the above sequence of events are illustrated in Figure 4. We argue that there exists suitably large t 1 , t 2 , t 3 such that the above three events happen with a positive probability. The first event can happen for a suitably large t 1 because the leader arm follows the policy ¯ π , and s a is in the recurrent class of ¯ π . In the second event, the leader arm can stay in state s a and keep applying action 1 because ¯ π (1 | s a ) > 0 and P ( s a , 1 , s a ) > 0 . The follower arm can reach s b after a suitably large t 2 time slots because it keeps applying action 1 and s b is in the recurrent class of the all-one policy ¯ π 1 . In the third event, the leader arm can reach s b after a suitably large t 3 time slots because s b is also in the recurrent class of ¯ π . Meanwhile, the follower arm can stay in s b because P ( s b , 0 , s b ) > 0 and P ( s b , 1 , s b ) > 0 . Therefore, we have proved that for any ( s, a, ̂ s, ̂ a ) ,

By Proposition 2, we establish SA.

Proof of Proposition 5. Given any initial states and actions ( S (0) , A (0) , ̂ S (0) , ̂ A (0)) = ( s, a, ̂ s, ̂ a ) ∈ S × A × S × A , we construct a sequence of positive probability events that leads to S ( t ) = ̂ S ( t ) :

The leader arm reaches the state s ∗ after t 1 time slots;

The leader arm stays in the state s ∗ and keeps applying action 1 for t 2 time slots; meanwhile, the follower arm also applies action 1 and reaches the state s ∗ , so the two arms synchronize.

The transitions of the two arms during the above sequence of events are illustrated in Figure 4. We argue that there exists suitable t 1 and t 2 such that the above two events happen with a positive probability. The first event can happen for a suitably large t 1 because s ∗ is in the recurrent class of ¯ π . In the second event, the leader arm can stay in the state s ∗ and keeps applying action 1 because ¯ π (1 | s ∗ ) > 0 and P ( s ∗ , 1 , s ∗ ) > 0 . Meanwhile, the follower arm can reach s ∗ after a suitably large t 2 time slots because s ∗ is also in the recurrent class of the all-one policy ¯ π 1 . Therefore, we have proved that for any ( s, a, ̂ s, ̂ a ) ,

C.3 Sufficient conditions based on cycles

The above sufficient conditions are based on self-loops, which can be viewed as cycles of minimal length. These conditions can be generalized to longer cycles, formally defined below.

Definition 1 (Cycle) . Consider the MDP ( S , A , P, r ) . We call an ordered set of states C = ( s 0 , s 1 , . . . , s L ) a cycle under the Markovian policy ¯ π if there is a positive probability of transitioning from s j to s j +1 for all j under the policy ¯ π (we identify s L +1 with s 0 ). We call the ordered set C a cycle under any policy if there is a positive probability of transitioning from s j to s j +1 for all j under any policy. In both cases, we call L the length of the cycle C .

We give two sufficient conditions based on cycles. These conditions are relaxations of the conditions in Proposition 4 and 5.

Proposition 6. Consider the single-armed problem with the unichain MDP ( S , A , P, r ) and budget α ∈ (0 , 1) . Let ¯ π be a single-armed policy with recurrent class S . Let the recurrent class of the all-one policy ¯ π 1 be denoted as S 1 . If the following conditions hold:

There exists a cycle C a under the policy ¯ π in S , and ¯ π (1 | s ) > 0 for all s ∈ C a ;

There exists cycle C b under any policy in S ∩ S 1 ;

The lengths of the two cycles C a and C b are relatively prime.

then Synchronization Assumption holds for the policy ¯ π .

Proposition 7. Consider the single-armed problem with the unichain MDP ( S , A , P, r ) and budget α ∈ (0 , 1) . Let ¯ π be a single-armed policy with recurrent class S . Let the recurrent class of the all-one policy ¯ π 1 be denoted as S 1 . If the following conditions hold:

There exists a cycle C ∗ under the policy ¯ π in S ∩ S 1 , and ¯ π (1 | s ) > 0 for all s ∈ C a ;

The policy ¯ π 1 is aperiodic,

then Synchronization Assumption holds for the policy ¯ π .

Similarly to Propositions 4 and 5, Propositions 6 and 7 have analogous version that are stated in terms of the all-zero policy ¯ π 0 . We omit the details.

Now we prove Propositions 6 and 7.

leader

leader

3

Cb

2

follower

Y

*

2

follower

1

1

2

3

3

2

S

S1

S

S1

Proof of Proposition 6. Given any initial states and actions ( S (0) , A (0) , ̂ S (0) , ̂ A (0)) = ( s, a, ̂ s, ̂ a ) ∈ S × A × S × A , we construct a sequence of positive probability events that leads to S ( t ) = ̂ S ( t ) :

The leader arm reaches the cycle C a after t 1 time slots;

The leader arm transitions along the cycle and keeps applying action 1 ; meanwhile, the follower arm also applies action 1 and reaches the cycle C b ; this happens in t 2 times slots; let the states of the leader and follower arm at this moment be s a and s b ;

The leader arm transitions along the cycle C a for another c laps, for some positive integer c , and then spends another t 3 time slots to go from s a to s b ; meanwhile, the follower arm transitions along the cycle C b for another d laps, for some positive integer d .

The transition of the arms during the sequence of events are as illustrated in Figure 5. We argue that there exists some suitable t 1 , t 2 , t 3 , c , and d such that the above four events happen with a positive probability, and the two arms synchronize after the four events.

The first event happens with a positive probability for suitably large t 1 because C a is in S . The second event can happen for suitably large t 2 because π (1 | s ) > 0 for all s ∈ C a , and C b is in S 1 .

After the first two events, suppose the state of the leader arm is s a , and the state of the follower arm is s b , then s a ∈ C a and s b ∈ C b . Because both cycles are in S , there exists a positive probability path from s a to s b under ¯ π . Let t 3 be the length of the path. Let L a and L b be the length of cycles C a and C b . Because L a and L b are relatively prime, we can take c and d such that dL b -cL a = t 3 . With this choice of c, d , and t 3 , the two arms will synchronize after the third and fourth events.

Now we argue that the third and fourth events happen with a positive probability. Because C a is a cycle under the policy ¯ π , the leader arm can transition along the cycle for an arbitrary number of time slots. Similarly, because C b is a cycle under any policy, the follower arm can transition along the cycle for an arbitrary number of time slots; the leader arm can reach s b from s a because of the positive probability path described in the last paragraph.

Therefore, we have proved that for any ( s, a, ̂ s, ̂ a ) ,

By Proposition 2, we establish SA.

Proof of Proposition 7. Given any initial states and actions ( S (0) , A (0) , ̂ S (0) , ̂ A (0)) = ( s, a, ̂ s, ̂ a ) ∈ S × A × S × A , we construct a sequence of positive probability events that leads to S ( t ) = ̂ S ( t ) :

The leader arm reaches the cycle C ∗ after t 1 time slots;

The leader arm transitions along the cycle and keeps applying action 1 ; meanwhile, the follower arm also applies action 1 and reaches the same state as the leader arm after t 2 time slots.

The transition of the arms during the sequence of events are as illustrated in Figure 5. We argue that the above two events happen with a positive probability for suitable t 1 and t 2 . The first event can happen for suitably large t 1 because the leader arm follows the policy ¯ π the cycle C ∗ is in the recurrent class of ¯ π . After reaching C ∗ , the leader arm can remain on C ∗ because C ∗ is a cycle under the policy ¯ π , and ¯ π (1 | s ) > 0 for all s ∈ C ∗ . The follower arm can reach the same state as the leader arm because the cycle C ∗ is also in the recurrent class of ¯ π 1 , and ¯ π 1 is aperiodic. Therefore, we have proved that for any ( s, a, ̂ s, ̂ a ) ,

By Proposition 2, we establish SA.

D On relaxing the unichain condition

Throughout this paper, we have imposed the (all-policy) unichain condition, i.e., all Markovian single-armed policies have a single recurrent class. Making this blanket assumption simplifies our presentation, but it can be substantially relaxed for many of our results, as we discuss below.

The main change caused by dropping the unichain assumption is the definition of the single-armed problem, because there could be multiple recurrent classes under a single-armed policy, each corresponding to a different long-run average reward. To adapt to this change, we can let both the single-armed policy ¯ π and the initial state distribution µ be optimization variables of the single-armed problem. We denote the average reward given ¯ π and µ as V ¯ π,µ 1 . We still denote the optimal value of the single-armed problem as V rel 1 . Note that the single-armed problem can still be solved by the same linear program in (LP) or (LP-CT): The optimal initial state distribution µ ∗ can be taken as the optimal stationary state distribution ( y ∗ ( s, 1) + y ∗ ( s, 0)) s ∈ S , given the optimal solution of LP, y ∗ .

The definitions of FTVA and FTVA-CT do not require any change, considering that they are already using the stationary distribution of ¯ π to initialize the virtual states. The conversion losses, now denoted as V ¯ π,µ 1 -V FTVA (¯ π,µ ) N and V ¯ π,µ 1 -V FTVA-CT (¯ π,µ ) N , have the same bound as what we currently have in equation (11) in Theorem 1 for discrete-time RBs and equation (20) in Theorem 2 for continuous-time RBs. These bounds hold as long as the given input policy ¯ π has finite expected synchronization times , which can be seen by inspecting the proofs of the theorems given in Appendix E and Appendix F.

The optimality gap bounds in (12) in Theorem 1 and (21) in Theorem 2 still follow from the conversion loss bounds when the unichain assumption is dropped. To see this, for each of the discrete-time and continuous-time settings, consider an optimal single-armed policy ¯ π ∗ and optimal initial state distribution µ ∗ . Note that V ¯ π ∗ ,µ ∗ 1 = V rel 1 ≥ V ∗ N . If V ¯ π ∗ ,µ ∗ 1 -V FTVA (¯ π ∗ ,µ ∗ ) N = O (1 / √ N ) , then V ∗ N -V FTVA (¯ π ∗ ,µ ∗ ) N = O (1 / √ N ) ; if V ¯ π ∗ ,µ ∗ 1 -V FTVA-CT (¯ π ∗ ,µ ∗ ) N = O (1 / √ N ) , then V ∗ N -V FTVA-CT (¯ π ∗ ,µ ∗ ) N = O (1 / √ N ) .

In the discrete-time setting, our sufficient conditions for finite expected synchronization times, presented in Propositions 4, 5, 6 and 7, are valid as long as each of the two policies ¯ π and ¯ π 1 has a single recurrent class and the two classes intersect. Similarly, Lemma 5 in Appendix F, which establishes finite expected synchronization times for the continuous-time setting, holds under the same condition on ¯ π and ¯ π 1 .

E Proofs for discrete-time RBs

E.1 Overview

In this section, we focus on proving the bound (11) on the conversion loss V ¯ π 1 -V FTVA (¯ π ) N for any single-armed policy ¯ π . The asymptotic optimality result (12) in Theorem 1 will be a direct consequence of (11) if we take ¯ π to be any optimal single-armed policy ¯ π ∗ .

We first show that under FTVA ( ¯ π ) defined in Algorithm 1, each virtual process is an independent Markov chain induced by applying ¯ π to the single-armed system ( S , A , P, r ) , as stated in Lemma 1 below. We will rigorously prove it in the next subsection.

Lemma 1. Under FTVA ( ¯ π ) given in Algorithm 1, for each t = 0 , 1 , 2 . . . , i ∈ [ N ] , ̂ s ∈ S N , and ̂ a ∈ A N ,

Let ( y ( s, a )) s ∈ S ,a ∈ A be the steady-state state-action distribution of the single-armed system under ¯ π . By (23)(24) , for each t ≥ 0 , ( ̂ S i ( t ) , ̂ A i ( t )) for i ∈ [ N ] are i.i.d. with the distribution ( y ( s, a )) s ∈ S ,a ∈ A .

Next, as sketched in Section 4.2, we bound V ¯ π 1 -V FTVA (¯ π ) N as

̸

̸

We say an arm i is a bad arm at time t if ( ̂ S i ( t ) , ̂ A i ( t ) ) = ( S i ( t ) , A i ( t ) ) , and a good arm Then E [ ∑ N i =1 ✶ { ( ̂ S i ( ∞ ) , ̂ A i ( ∞ ) ) = ( S i ( ∞ ) , A i ( ∞ ) ) }] = E [ # bad arms ] in steady state. By Little's Law, we have the following relationship:

E [ # bad arms ] = (rate of generating bad arms) × E [ time duration of a bad arm ] .

To make the quantities in the above expression precise, we make the following definitions.

̸

We say there is a disagreement event of the arm i happening at time t if ̂ A i ( t ) = A i ( t ) . The disagreement events are the only cause that turns good arms into bad arms because otherwise by the construction of our algorithm, ̂ S i ( t ) will remain the same as S i ( t ) . The number of disagreement events in each time slot is determined by how much the budget required by virtual actions violates the constraint. We call the expected number of disagreement events when the virtual states are ̂ s the instantaneous disagreement rate , denoted as d ( ̂ s ) .

We also define disagreement period of the arm i as a continuous period of time when arm i is a bad arm, separated by disagreement events. Formally,

Definition 2 (Disagreement period) . Given a sample path of the arm i 's real states and virtual states ( S i ( t ) , ̂ S i ( t )) t ≥ 0 , we define the disagreement period of the arm i as a time interval [ t begin , t end -1] such that

̸

̸

We use D avg to denote the long-run average length of the disagreement periods, i.e., the average of t end -t begin of all disagreement periods.

With the definitions, we can state Little's Law in the context of disagreement periods. We omit its proof since it is a direct consequence of Little's Law.

Lemma 2 (Little's Law for disagreement periods) .

̸

where d ( ̂ s ) denotes the instantaneous disagreement rate when the virtual states are ̂ S ( t ) = ̂ s ; D avg denotes the long-run average length of the disagreement periods.

̸

̸

otherwise.

E.2 Lemmas and the proof of Theorem 1

In this section, we will prove Theorem 1. We first rigorously prove Lemma 1.

̸

Proof of Lemma 1. The first equation (23) is obvious because each virtual action ̂ A i ( t ) is sampled from ¯ π ( ·| ̂ S i ( t )) independent of anything else. We only need to show (24). Recall that by the definition of FTVA ( ¯ π ), ̂ S i ( t +1) could either be equal to S i ( t +1) if ( ̂ S i ( t ) , ̂ A i ( t )) = ( S i ( t ) , A i ( t )) , or generated afresh independent of anything else if ( ̂ S i ( t ) , ̂ A i ( t )) = ( S i ( t ) , A i ( t )) . Let I good ( t ) = { i : ( ̂ S i ( t ) , ̂ A i ( t )) = ( S i ( t ) , A i ( t )) } . Then

where the last equality is by the definition of I good ( t ) .

Combining (23) and (24), we have confirmed that each virtual process is an independent singlearmed process with transition kernel P and policy ¯ π . Because the initial virtual states { ̂ S i (0) } i ∈ [ N ] are sampled i.i.d. from the stationary distribution of ¯ π , { ̂ S i ( t ) } i ∈ [ N ] are also i.i.d. with the same distribution. Therefore, { ( ̂ S i ( t ) , ̂ A i ( t )) } i ∈ [ N ] are i.i.d. with distribution ( y ( s, a )) s ∈ S ,a ∈ A .

The next two lemmas bound the two quantities in Lemma 2, the long-run average disagreement rate and the long-run average length of the disagreement periods.

Lemma 3 (Average disagreement rates) . The instantaneous disagreement rate is equal to

The long-run average disagreement rate is bounded as

̸

Proof. We first prove the expression of instantaneous disagreement rate in (26). By definition, the instantaneous disagreement rate is equal to the expected number of arms such that A i ( t ) = ̂ A i ( t ) conditioning on the virtual states being ̂ s . Because FTVA tries to match as many A i ( t ) 's to ̂ A i ( t ) ′ s as possible, there are exactly | ∑ N i =1 ̂ A i ( t ) -αN | arms such that A i ( t ) = ̂ A i ( t ) . This proves (26).

̸

To bound the average disagreement rate, letting ̂ s = ̂ S ( ∞ ) in (26) and taking expectation, we have

By Lemma 1, the virtual actions ̂ A i ( ∞ ) 's are i.i.d. binomial random variables such that P ( ̂ A i ( ∞ ) = 1) = ∑ s ∈ S y ( s, 1) = α , ∑ N i =1 ̂ A i ( ∞ ) has distribution Binomial ( N,α ) , whose mean and variance are αN and Nα (1 -α ) ≤ N/ 4 . Therefore the expression in (28) is equal to Var [ ∑ N i =1 ̂ A i ( ∞ ) ] 1 / 2 = √ N/ 2 .

Lemma 4 (Average length of disagreement periods) .

Proof. To bound D avg, it suffices to bound the expected length of a disagreement period with arbitrary initial states. Without loss of generality, consider a disagreement period on arm i that starts at time t begin = 0 , with initial states ( S i (0) , A i (0) , ̂ S i (0) , ̂ A i (0)) = ( s, a, ̂ s, ̂ a ) . During the disagreement period, the i -th arm can be seen as a leader-and-follower system, where the real state-action pair ( S i ( t ) , A i ( t )) corresponds to the follower arm and the virtual state-action pair ( ̂ S i ( t ) , ̂ A i ( t )) corresponds to the leader arm. By Assumption 1, the leader arm and the follower arm will synchronize in a finite expected time. When they synchronize, the disagreement period stops, which means t end ≤ τ sync ( s, a, ̂ s, ̂ a ) . Therefore, the expected length of the disagreement period satisfies

This holds for arbitrary initial states ( s, a, ̂ s, ̂ a ) , so D avg ≤ τ sync max .

Given the three lemmas above, we can prove Theorem 1.

Proof of Theorem 1. Combining Lemma 2, 4 and 3, we have

̸

Plugging the above inequality into the bound on conversion loss in (13), we get

̸

This finishes the proof.

F Proofs for continuous-time RBs

F.1 Preliminary: finite synchronization time

In this section, we prove that the synchronization time has a finite expectation in the continuous-time setting. Unlike the discrete-time setting, we do not need to make any additional assumptions other than the standard unichain condition. Our proof is based on the observation that the holding time distribution of a continuous-time Markov chain has support on the whole positive real line, so an arbitrary number of transitions can happen in any time interval, which, from the uniformization perspective, implies self-loops in all states. We can thus prove that synchronization time has a finite expectation using similar logic as in the proof of Proposition 3 and 4.

Lemma 5 (Synchronization in continuous time) . Consider the single-armed policy ¯ π and the corresponding leader-and-follower system. Given the initial states ( S (0) , ̂ S (0)) = ( s, ̂ s ) , the synchronization time is bounded as

Proof of Lemma 5. Consider the always1 policy ¯ π 1 given by

By the unichain property, ¯ π 1 only has a single recurrent class, which we denote as S 1 . We denote the recurrent class under ¯ π as S .

̸

We first prove by contradiction that S ∩ S 1 = ∅ . Suppose S ∩ S 1 = ∅ . Then we define a new policy ¯ π ′ as

̸

Then S and S 1 are two distinct recurrent classes under ¯ π ′ . This is because by definition of ¯ π ′ , an arm with the initial state in S remains in S , so it cannot reach S 1 ; similarly an arm cannot reach S from S 1 . The existence of two recurrent classes contradicts the unichain condition. Therefore, we must have S ∩ S 1 = ∅ .

We show that given any pair of initial states s, ̂ s ∈ S , the probability that τ sync ( s, ̂ s ) < 3 is positive. We construct a sequence of positive probability events that leads to S ( t ) = ̂ S ( t ) before time 3 :

The leader arm reaches a state s a such that ¯ π (1 | s a ) > 0 by time 1 and chooses action 1 ;

The leader arm has no transition during [1 , 2] , so ̂ S ( t ) = s a and ̂ A ( t ) = 1 for all t ∈ [1 , 2] ;

The follower arm applies the same action as the leader arm during [1 , 2] , so A ( t ) = 1 for all t ∈ [1 , 2] ; and reaches the state s b by time 2 for some s b ∈ S ∩ S 1 ;

The follower arm stays at s b during [2 , 3] , and the leader arm reaches s b by time 3 , so the two arms synchronize.

The first two events have positive probabilities because the policy ¯ π applies action 1 with a positive fraction of time. To see why the third event has a positive probability, observe that the follower arm applies action 1 for all t ∈ [1 , 2] regardless of its state, so it is effectively under policy ¯ π 1 and could traverse all states in S 1 . The fourth event has a positive probability because the leader arm under ¯ π can traverse all states in S . Note that in the above arguments, we are implicitly assuming that the two arms do not synchronize in the middle of the four events, because otherwise we are done. Also, we use the fact that in a continuous-time Markov chain, there can be an arbitrary number of transitions during any time interval. Therefore, we have proved that for all s, ̂ s ∈ S ,

Now we prove by induction that for all s, ̂ s ∈ S and k = 0 , 1 , 2 , . . . ,

̸

̂

where in the second equality we have used the Markov property of the system, and in the last inequality we apply the induction hypothesis. This proves (30).

The bound on the expectation E [ τ sync ( s, ̂ s )] follows summing the tail bound (30) over k = 0 , 1 , 2 , . . . :

where the second inequality is due to (30) and the fact that max s ′ , ̂ s ′ ∈ S P ( τ sync ( s ′ , ̂ s ′ ) ≥ 3) ≤ 1 -min s ′ , ̂ s ′ ∈ S P ( τ sync ( s ′ , ̂ s ′ ) < 3) < 1 . This finishes the proof.

F.2 Overview of the proof of Theorem 2

In this section, we focus on proving the bound (20) on the conversion loss V ¯ π 1 -V FTVA-CT (¯ π ) N for any single-armed policy ¯ π . The asymptotic optimality result (21) in Theorem 2 will be a direct consequence of (20) if we take ¯ π to be any optimal single-armed policy ¯ π ∗ .

We first show that under FTVA-CT ( ¯ π ) defined in Algorithm 2, { ( S ( t ) , ̂ S ( t )) } t ≥ 0 is a continuous-time Markov chain and each virtual process is an independent Markov chain induced by applying ¯ π to the single-armed system, as stated in Lemma 6 below. We will prove it in the next subsection.

Lemma 6. Under the FTVA-CT ( ¯ π ) defined in Algorithm 2, we have the following:

{ ( S ( t ) , ̂ S ( t )) } t ≥ 0 is a continuous-time Markov chain.

̸

{ ̂ S i ( t ) } t ≥ 0 for i ∈ [ N ] are N independent Markov chains, whose transition rate from state s to s ′ is ∑ a ∈ A G ( s, a, s ′ )¯ π ( a | s ) , for s, s ′ ∈ S s.t. s = s ′ .

Let ( y ( s, a )) s ∈ S ,a ∈ A be the steady-state state-action distribution of the single-armed system under ¯ π . Then the second result above implies that for each t ≥ 0 , ( ̂ S i ( t ) , ̂ A i ( t )) for i ∈ [ N ] are i.i.d. with the distribution ( y ( s, a )) s ∈ S ,a ∈ A .

Next, we can upper bound the conversion loss V ¯ π -V FTVA-CT (¯ π ) as

̸

̸

̸

The bound is slightly different from the (13) since we want to separately deal with the number of arms whose real and virtual states do not agree and those whose real and virtual actions do not agree, as shown in the lemma below.

Lemma 7. It holds that

̸

Before showing the proof of Lemma 7, we will first use Lemma 7 to prove Theorem 2.

Proof of Theorem 2. Combining (31) and Lemma 7, we have

By Lemma 6, the virtual actions ̂ A i ( ∞ ) 's are i.i.d. binomial random variables such that P ( ̂ A i ( ∞ ) = 1) = ∑ s ∈ S y ( s, a ) = α , so the distribution of ∑ N i =1 ̂ A i ( ∞ ) is Binomial ( N,α ) , whose mean and variance are αN and Nα (1 -α ) ≤ N/ 4 . Therefore,

̸

Combining the above calculations, we get

F.3 Proof of Lemma 6

Proof. We first show that { ( S ( t ) , ̂ S ( t )) } t ≥ 0 is a continuous-time Markov chain. Observe that { ( S ( t ) , ̂ S ( t )) } t ≥ 0 is piecewise constant between decision epochs { t k } k =0 , 1 ,... , so it suffices to consider the time between the decision epochs t k +1 -t k and the states at the decision epochs ( S ( t k ) , ̂ S ( t k )) for k = 0 , 1 , 2 , . . .

We claim that at each decision epoch t k , the time until the next decision epoch t k +1 -t k is exponentially distributed conditioned on ( S ( t k ) , ̂ S ( t k )) . Observe from the pseudo-code in Algorithm 2 that at the decision epoch t k , conditioned on ( S ( t k ) , A ( t k ) , ̂ S ( t k )) , the time until the next decision epoch t k +1 is the minimum of two independent exponential random variables, corresponding to the exponential timer and the transition of real states. The rates of the two exponential random variables are 2 Ng max -g real k and g real k , where g real k = ∑ N i =1 G ( S i ( t k ) , A i ( t k )) . Therefore, conditioned on ( S ( t k ) , A ( t k ) , ̂ S ( t k )) , t k +1 -t k has an exponential distribution with rate 2 Ng max -g real k + g real k = 2 Ng max . Because the rate 2 Ng max is a constant, if we take expectation over A ( t k ) and only conditioned on ( S ( t k ) , ̂ S ( t k )) , the time until the next decision epoch t k +1 -t k is still exponentially distributed with rate 2 Ng max .

Also, it is obvious from the pseudo-code in Algorithm 2 that conditioned on ( S ( t k ) , ̂ S ( t k )) , the distribution of ( S ( t k +1 ) , ̂ S ( t k +1 )) is independent of t k +1 -t k and ( S ( t ) , ̂ S ( t )) for t < t k . Therefore, we conclude that { ( S ( t ) , ̂ S ( t )) } t ≥ 0 is a continuous-time Markov chain.

̸

Next, we show that the { ̂ S i ( t ) } t ≥ 0 for i ∈ [ N ] are N independent Markov chains, whose transition rate from state s to s ′ is ∑ a ∈ A G ( s, a, s ′ )¯ π ( a | s ) for s ′ = s . Because we have shown that { ( S ( t ) , ̂ S ( t )) } t ≥ 0 is a continuous-time Markov chain with a constant transition rate, it suffices to focus on the embedded chain { ( S ( t k ) , ̂ S ( t k )) } k =0 , 1 ,... and examine the probability of the transitions that change the virtual states ̂ S ( t k ) . Note that between two decision epochs t k and t k +1 , there is at most one i such that ̂ S i ( t ) changes. Let I good ( t k ) = { i ∈ [ N ] : ̂ S i ( t k ) = S i ( t k ) , ̂ A i ( t k ) = A i ( t k ) } . For any arm i ∈ I good ( t k ) , the transitions of its virtual and real states are coupled, which implies that the probability that ̂ S i ( t k +1 ) = s ′ is the same as the probability that S i ( t k +1 ) = s ′ , for any s ′ = ̂ S i ( t k ) , conditioned on the states and actions at t k . Formally,

̸

̸

where the second equality uses the fact that the transition probability after uniformization is equal to the ratio between the transition rate and the uniformization rate; the last equality is by the definition of I good ( t k ) . For i / ∈ I good ( t k ) and s ′ = ̂ S i ( t k ) , we have ̂ S i ( t k +1 ) = s ′ only when the exponential

clock ticks and the pair ( i, s ′ ) is sampled, so

̸

Because ̂ A i ( t k ) is independently sampled from the distribution ¯ π ( ·| ̂ S i ( t k )) , taking expectation over ( A ( t k ) , ̂ A ( t k )) in the above equation, we get

̸

for any i ∈ [ N ] and s ′ = ̂ S i ( t k ) . Because the uniformization rate is 2 Ng max , the rate for ̂ S i ( t ) to transition to s ′ is equal to ∑ a ∈ A G ( ̂ S i ( t ) , a, s ′ )¯ π ( a | ̂ S i ( t ) ) conditioned on ( S ( t ) , ̂ S ( t )) . Observe that this transition rate only depends on ̂ S i ( t ) , which implies that ̂ S i ( t ) 's for i ∈ [ N ] are N i.i.d. Markov chains.

Finally, recall that we initialize the virtual states ̂ S i (0) 's as N i.i.d. samples from the stationary distribution of the single-armed system under the policy ¯ π . Since we have proved that { ̂ S i ( t ) } t ≥ 0 's are N i.i.d. Markov chains induced by applying ¯ π to the single-armed systems, for each t ≥ 0 , ̂ S i ( t ) 's remain stationary and i.i.d. Therefore, for each t ≥ 0 , ( ̂ S i ( t ) , ̂ A i ( t )) 's are i.i.d., and for each i , the distribution of ( ̂ S i ( t ) , ̂ A i ( t )) is equal to the steady-state state-action distribution ( y ( s, a )) s ∈ S ,a ∈ A .

F.4 Proof of Lemma 7

In this section, we prove the key intermediate result, Lemma 7. The second equation (33) in Lemma 7 is obvious from the definition of the policy. We can therefore focus on proving (32), i.e., bounding the long-run average number of arms whose real states are not equal to their virtual states. We call such arms bad arms , and the rest of the arms good arms .

We will use a similar approach as Section E: we invoke Little's Law to write the average number of bad arms as a product of the average disagreement rate and the average length of disagreement periods , based on a suitable definition of the disagreement events .

̸

We define the disagreement event in a different way than in the discrete-time RB setting, because unlike in the discrete-time RB setting where A i ( t ) = ̂ A i ( t ) can immediately cause S i ( t ) = ̂ S i ( t ) , in the continuous-time RB setting, actions that last for only a short time may not have an effect on the states. Therefore, we say a disagreement event for arm i happens at time t only when arm i behaves differently than it would have behaved if A i ( t ) = ̂ A i ( t ) .

̸

Specifically, the disagreement event of a good arm can be defined as having a state transition that turns it into a bad arm, since a good arm is supposed to remain good if we always have A i ( t ) = ̂ A i ( t ) ; for bad arms, how the arm 'would have behaved' need to be defined with the help of an extra structure, namely, exponential timers, as described below.

̸

̸

Definition 3 (Exponential timers for simulating the leader-and-follower system) . For each i ∈ [ N ] , we run a timer for each arm i that ticks every random amount of time. When the timer of arm i starts, it decides the time of its next tick based on the i -th arm ( S i ( t ) , A i ( t ) , ̂ S i ( t ) , ̂ A i ( t )) : if S i ( t ) = ̂ S i ( t ) and A i ( t ) = ̂ A i ( t ) , the timer ticks after a time that is exponentially distributed with rate G ( S i ( t ) , ̂ A i ( t )) ; otherwise, the timer pauses. The timer restarts when it ticks or when there is any event happening in the system.

̸

̸

The purpose of the exponential timer is to simulate the transition time of the leader-and-follower system, which characterizes how the arm would have behaved if A i ( t ) = ̂ A i ( t ) . Specifically, when arm i has S i ( t ) = ̂ S i ( t ) (so it is a bad arm) and A i ( t ) = ̂ A i ( t ) , we construct an imaginary leaderand-follower system whose state-action pairs are ( S i ( t ) , ̂ A i ( t ) , ̂ S i ( t ) , ̂ A i ( t )) . We let the transition times of the virtual state ̂ S i ( t ) in the two systems be identical, and the transition times of the real state S i ( t ) in the two systems be independent. Then the transition time of S i ( t ) in the leader-and-follower system has an exponential distribution with rate G ( S i ( t ) , ̂ A i ( t )) , which is equal to the time that the exponential timer ticks.

̸

̸

̸

Therefore, there are two ways that the transition of arm i with S i ( t ) = ̂ S i ( t ) and A i ( t ) = ̂ A i ( t ) can deviate from the leader-and-follower system described above: either arm i itself has a transition in the real state S i ( t ) , or when the exponential timer ticks. This statement is actually also true if S i ( t ) = ̂ S i ( t ) and A i ( t ) = ̂ A i ( t ) because in that case, the exponential timer pauses.

̸

We can thus formally define the disagreement event of arm i as below.

̸

Definition 4 (Disagreement event for continuous-time RBs) . For each i ∈ [ N ] , a disagreement event of arm i happens when its real state S i ( t ) transitions or its exponential timer ticks, while A i ( t ) = ̂ A i ( t ) .

̸

After defining the disagreement events, we can define the disagreement period of arm i as the period of time when S i ( t ) = ̂ S i ( t ) , separated by disagreement events, formally stated below.

Definition 5 (Disagreement period for continuous-time RBs) . Given a sample path of the arm i 's real states and virtual states ( S i ( t ) , ̂ S i ( t )) , and an exponential timer, we define the disagreement period of the arm i as a time interval [ t begin , t end ) such that

There is a disagreement event at t begin ;

There is no disagreement event during ( t begin , t end ) , and S i ( t ) = S i ( t ) for t ∈ ( t begin , t end );

There is a disagreement event at t end or S i ( t end ) = S i ( t end ) .

We let d ( s , ̂ s ) be the instantaneous rate of disagreement events (instantaneous disagreement rate) when the system has real and virtual states ( S ( t ) , ̂ S ( t )) = ( s , ̂ s ) . Let D avg be the long-run average length of the disagreement periods.

̸

Observe that the number of bad arms (the arms such that S i ( t ) = ̂ S i ( t ) ) is the number of arms in disagreement periods, so we can apply Little's Law to the number of bad arms.

Lemma 8 (Little's Law for disagreement periods, continuous-time version) . It holds that

̸

where d ( s , ̂ s ) denotes the instantaneous disagreement rate when the virtual states are ̂ S ( t ) = ̂ s ; D avg denotes the long-run average length of the disagreement periods.

Lemma 9 (Average disagreement rates) . The instantaneous disagreement rate is equal to

̸

Proof. For each arm i such that A i ( t ) = ̂ A i ( t ) , a disagreement event happens if its real state transitions, which happens at the rate G ( S i ( t ) , A i ( t )) , or if its exponential timer ticks, which happens at the rate 0 (if it is a good arm) or G ( S i ( t ) , ̂ A i ( t )) (if it is a bad arm). Therefore, the rate that disagreement events happen at arm i is no more than

By the definition of the policy, there are in expectation E [∣ ∣ ∣ ∑ N i =1 ̂ A i ( t ) -αN ∣ ∣ ∣ | ̂ S ( t ) = ̂ s ] arms with A i ( t ) = ̂ A i ( t ) , so the instantaneous disagreement rate of the system is as given in (39).

̸

Lemma 10 (Average length of disagreement periods) . It holds that

Proof. To bound D avg, it suffices to bound the expected length of a disagreement period with arbitrary initial states. Without loss of generality, consider a disagreement period on arm i that starts at time t begin = 0 , with initial states ( S i (0) , ̂ S i (0)) = ( s, ̂ s ) . During the disagreement period, there is no disagreement event, so as argued in the paragraph after Definition 3, the transitions of the i -th arm is identical to a leader-and-follower system. Therefore, we either have S i ( t ) = ̂ S i ( t ) after τ sync ( s, ̂ s ) amount of time, or have a disagreement event before that. In either case, t end ≤ τ sync ( s, ̂ s ) . Therefore, the expected length of the disagreement period satisfies

This holds for arbitrary initial states ( s, ̂ s ) , so D avg ≤ τ sync max .

Proof of Lemma 7. Combining Lemma 8, 9, and 10, we have

̸

This proves (32).

Observe that by the definition of our algorithm, at any time t ,

̸

Taking the steady-state expectation, we get (33).

G Experiment details and additional experiments

In this section, we include the details of the experiments in the main body, as well as some additional experiments. In Appendix G.1 and G.2, we describe the details of the experiments of Figure 1 and 3. We conduct the simulations of these two experiments with different initial points and present the results in Appendix G.3. To give an intuitive understanding of the difference between these policies, we also display more visualization of the sample paths under different policies in Appendix G.4. 3

G.1 Experiment details of Figure 1

Restless bandits setting In Figure 1, we consider the discrete-time N -arm restless bandits represented by the tuple ( N, S N , A N , P, r, αN ) . We vary the number of arms N , and keep the rest of the parameters fixed. The state space of each arm is S = { 1 , 2 , 3 } . The action space is A = { 0 , 1 } . The transition kernel P is given by

where the number in s -th row and s ′ -th column in each matrix represents P ( s, 0 , s ′ ) or P ( s, 1 , s ′ ) , for s, s ′ ∈ { 1 , 2 , 3 } . The reward function r is given by

where the s -th entry of each vector represents r ( s, 0) or r ( s, 1) , for s ∈ { 1 , 2 , 3 } . The budget parameter α = 0 . 4 , so 0 . 4 N arms are pulled in each time slot. This setting is taken from the Appendix E of [GGY20] as a counterexample to the UGAP assumption. We note that this RB problem obviously satisfies Synchronization Assumption for any policy: observe that P ( s, a, s ′ ) > 0 for all s, s ′ ∈ S , a ∈ A , so two arms with any initial states have a positive probability of synchronizing in the next time slot, which implies finite synchronization time by Proposition 2.

Simulation setting We plot the long-run average reward against the number of arms for different policies. The number of arms N varies from 100 , 200 , . . . , 1000 . Three policies are considered: our policy FTVA ( ¯ π ∗ ), the Whittle's index policy [Whi88], and an LP-Priority policy [Ver16]. For each data point, we obtain the long-run average reward and its confidence interval by simulating 50 independent trajectories, each with a length of 1000 time slots. The initial states of all arms are simply chosen to be 1 , because simulation results do not quite depend on the initial states (see Appendix G.3).

Detail of policies We implement Whittle index policy and LP-Priority in the standard way. The resulting priorities of both policies turn out to be 1 > 2 > 3 (state 1 has the highest priority). This is not a coincidence given that the solution of the single-armed problem (LP) is:

where the s -th row represents y ∗ ( s, 0) and y ∗ ( s, 1) , for s ∈ { 1 , 2 , 3 } . This solution implies that a reasonable policy should almost always pull arms in state 1 , pull arms in state 2 for a certain fraction of time, and almost never pull arms in state 3 . Therefore the only reasonable priority in this RB setting is 1 > 2 > 3 .

Our policy FTVA ( ¯ π ∗ ) is implemented according to the pseudocode in Section 3. The non-trivial detail here is the tie-breaking rule for selecting the set of arms to activate based on the virtual actions. Our tie-breaking rule is to select A i ( t ) 's such that the number of good arms, i.e., the arms such that S i ( t ) = ̂ S i ( t ) and A i ( t ) = ̂ A i ( t ) , is maximized. We have experimented with alternative tiebreaking rules, whose performance matches our theoretical results, though sometimes their asymptotic optimality requires larger values of N to be observed. The relationship between various tie-breaking rules and their impact on finiteN performances remains unclear in the present analysis, leaving it as a topic for future investigation.

G.2 Experiment details of Figure 3

Restless bandits setting In Figure 3, we consider the discrete-time N -arm bandits constructed as below. Suppose the state space for each arm is S = { 0 , 1 , . . . , 7 } . Each state has a preferred action , which is action 1 for states 0 , 1 , 2 , 3 , and action 0 otherwise. For an arm in state s , applying the preferred action moves the arm to state ( s +1) mod 8 with probability p s, R, and applying the other action moves the arm to state ( s -1) + with probability p s, L. 4 The probabilities p · , R and p · , L are given by

When the arm transitions from 7 to 0 , one unit of reward is generated. Equivalently, if we consider the expected reward of applying an action at a certain state, we can define the reward function as r (7 , 0) = p 7 , R, and r ( s, a ) = 0 for all other s ∈ S , a ∈ A . The parameter α = 1 / 2 , so N/ 2 arms are activated in each time slot.

Simulation setting We plot the long-run average reward against the number of arms for different policies. The number of arms N varies from 100 , 200 , . . . , 1000 . Three policies are considered: our policy FTVA ( ¯ π ∗ ), a random tie-breaking policy, and a particular LP-Priority policy that prioritizes arms with larger Lagrangian optimal indices (see the definition of Lagrangian optimal indices in

[HF17, BS20, GGY22]). This particular LP-Priority policy is also referred to as the LP-Index policy in the literature. But for simplicity, we just refer to it as LP-Priority in this paper. For each data point, we obtain the long-run average reward and its confidence interval by simulating 50 independent trajectories, each with a length of 1000 time slots. The initial states for the simulations are fixed: N/ 3 arms are in state 1 and 2 N/ 3 arms are in state 2 . Note that we fix this initial point in the simulation of Figure 3 in order to demonstrate that both the random tie-breaking and LP-Priority can get nearly zero rewards in this example. For other choices of fixed points, there is also a strong separation between the performance of FTVA ( ¯ π ∗ ) and random tie-breaking or LP-Priority, which we will show in the next section.

Details of policies The optimal solution of (LP) is y ∗ ( s, 1) = 1 / 8 for s = 0 , 1 , 2 , 3 , y ∗ ( s, 0) = 1 / 8 for s = 4 , 5 , 6 , 7 , and y ∗ ( s, a ) = 0 for other s ∈ S and a ∈ A . Note that the same y ∗ remains optimal even if we remove the budget constraint. The optimal LP solution suggests that a reasonable policy should almost always pull arms in states 0 , 1 , 2 , 3 , and almost never pull arms in states 4 , 5 , 6 , 7 .

The random tie-breaking policy that we consider prioritizes arms whose states are in { 0 , 1 , 2 , 3 } over arms whose states are in { 4 , 5 , 6 , 7 } , and it breaks ties uniformly at random when there are more than N/ 2 arms in either of the two sets.

The LP-Priority policy we consider prioritizes arms with larger Lagrangian optimal indices [BS20, GGY22]. Specifically, it involves solving the Lagrangian relaxation of the original LP, which replaces the budget constraint with a penalty term determined by the optimal Lagrange multiplier. In our setting, because the optimal solution y ∗ remains optimal even without the budget constraint, we can simply remove the budget constraint to get the Lagrangian relaxation. 5 The resulting Lagrangian optimal indices are given by:

and the priority is 1 > 2 > 3 > 0 > 7 > 6 > 5 > 4 .

The implementation of our policy FTVA ( ¯ π ∗ ) is the same as in the last experiment.

Further discussions of the policies As shown in Figure 3, the random tie-breaking policy and the LP-Priority policy get nearly zero rewards. We have discussed why this happens for the random tie-breaking policy in Section 3.3. For the LP-Priority policy based on the Lagrangian optimal index, the reason why it does not work is less obvious. Some sample paths suggest that under this policy, the arms will concentrate on { 3 , 4 , 5 } most of the time and thus cannot get a reward. Here is a possible explanation for why arms cannot easily escape from { 3 , 4 , 5 } : when some of the arms transition to state 6 , by the priority 6 > 5 > 4 , those arms in state 6 are likely to be activated. Because states 6 do not prefer action 1 , those arms will transition back to state 5 and thus get trapped.

In addition to the numerical results, we also note that our policy FTVA ( ¯ π ∗ ) is provably asymptotically optimal in this RB problem even though its transition kernel is not unichain. This stems from the fact that the optimal single-armed policy ¯ π ∗ has a single recurrent class S , and satisfies Synchronization Assumption. See Appendix D for the discussion on the conditions under which Synchronization Assumption and Theorem 1 hold when unichain is not assumed.

G.3 Varying the initial points of the Figure 1 and 3 examples

In Figure 1 and Figure 3, we have fixed one initial point for each simulation, for the ease of presentation. In this section, we rerun the simulations for these two settings with more initial points. For the simulations of each setting, we generate initial states in the following way.

We first choose a probability distribution on the state space S from all the possible probability distributions on S uniformly at random. Let π ( s ) denote the probability of state s under the chosen distribution. This distribution is chosen independently across the 20 sets of simulations.

For each N -armed problem, we set the initial state such that N · π ( s ) arms are in state s for each s, with proper rounding.

For each N -armed problem, we run all the policies with this initial state.

(a) The experiment of Figure 1

(b) The experiment of Figure 3

0.125

0.0200

FTVA(π*),minof20sets

0.120

0.0175

Randomtie-breaking,maxof20sets

LP-Priority,maxof20sets

0.0150

0.115

Upperbound

0.0125

0.110

R

0.0100

Avg

g

M

0.105

0.0075

0.100

1

FTVA(π*),minof20sets

0.0050

?

WhittleIndex/LP-Priority,maxof20sets

0.0025

0.095

Upperbound

0.0000

200

400

600

800

1000

200

400

600

800

1000

N

N

We illustrate the results of the 20 sets of simulations in the Figure 6a and 6b in the following way: For FTVA (¯ π ∗ ) , for each N , we take the minimum average reward over the 20 sets of simulations and plot them in the figure; For other policies, for each N , we take the maximum average reward over the 20 sets of simulations. We can see that the min curve for FTVA (¯ π ∗ ) still approaches the optimal value as N increases, whereas the max curves for other policies are strictly separated from the optimal value.

G.4 Visualization of the policies

In Figure 2 of Section 3.3, we illustrate what LP-Priority and FTVA (¯ π ∗ ) does differently by visualizing their sample paths. Here we include more such figures.

We still use the same example defined in Section 3.3. To recap, this example has 8 states, numbered as { 0 , 1 , 2 , 3 , 4 , 5 , 6 , 7 } . The optimal single-armed policy always chooses the preferred action in each state, which is 1 for states 0 , 1 , 2 , 3 , and 0 for states 4 , 5 , 6 , 7 . Under the optimal single-armed policy, each arm moves from state s to ( s +1) mod 8 with probability 0 . 1 or stays at s with probability 0 . 9 . The optimal distribution for the single-armed system is uniform over the 8 states.

When simulating the policies in the N -armed system, we can tell that a policy is good if the fraction of arms in each state is roughly uniform, and bad if the arms concentrate on a small number of states. We can also see why this happens by looking at whether the preferred action is chosen, and whether the arms are moving from s to ( s +1) mod 8 .

The three heatmaps in Figure 7a illustrate how the fractions of arms in each state change over time under the three policies: random tie-breaking, LP-Priority, and FTVA (¯ π ∗ ) . The x-axis represents the time slot, which ranges from 0 to 499 ; the y-axis represents the states; the brighter color represents a larger fraction of arms in this state at this time, and the specific value represented by each color can be found in the color bar on the right.

We can see that under the random tie-breaking, the arms concentrate around state 0 ; under LPPriority, the arms concentrate around states { 3 , 4 , 5 } after a burn-in period; under FTVA (¯ π ∗ ) , the arms uniformly spread out over the 8 states.

In Figure 7b, we take a closer look at the sample path of the random tie-breaking policy from time 250 to 289 and contrast it with the sample path if the system uses FTVA (¯ π ∗ ) from time 250 onwards. We also add some arrows indicating the drift of the arms, i.e., the average direction that the arms in this state are moving into. The colors of the arrows represent the direction: the blue arrow implies that the arm takes the preferred action and moves in the correct direction, while the red arm implies that the arm takes the non-preferred action and moves in the wrong direction. The magnitudes of the arrows represent how fast they are moving.

We can see that under random tie-breaking, although the arms have a strong tendency to move up from state 0 to state 1 , they move back when they reach state 1 and thus get stuck at state 0 . In

0

1

2

3

4

5

6

7

state

0

1

2

3

4

5

6

7

state

0

100

200

300

400

499

time slot

0

1

2

3

4

5

6

7

state

0

1/8

2/8

3/8

4/8

5/8

6/8

7/8

1

(a) Time evolution of the fractions of arms in each state under the three policies during the first 500 time slots. Top to bottom: Random Tie-Breaking, LP-Priority, and FTVA (¯ π ∗ ) .

0

1

2

3

4

5

6

7

state

250

254

258

262

266

270

274

278

282

286

289

time slot

0

1

2

3

4

5

6

7

state

0

1/8

2/8

3/8

4/8

5/8

6/8

7/8

1

(b) Time evolution of the fraction of arms in each state under Random Tie-Breaking (upper), or after switching to FTVA (¯ π ∗ ) (lower) since time slot 250 . The color and magnitude of the arrows represent the average movement of the arms in each state.

contrast, when switching to FTVA (¯ π ∗ ) , most arrows point in the right direction, which implies that the arms consistently apply the preferred actions. As a result, FTVA (¯ π ∗ ) helps the arms to escape from state 0 and converge to the uniform distribution over the state space.

Figure 8a illustrates the fraction of arms in state s taking action a in a few time slots after 250 in the same sample path of the random tie-breaking policy in Figure 7a. For each s and a . The upper bar plot is under the random tie-breaking policy, while the lower bar plot is after switching to FTVA (¯ π ∗ ) . The x-axis represents the state; the y-axis represents the fraction; there are four bars in each state, corresponding to 4 time-steps. Each bar has two segments, where the length of the blue segment indicates the fraction of arms taking the preferred action in this state, and the length of the red segment indicates the fraction of arms taking the non-preferred action in this state; the lower segment of the bar corresponds to action 0 , and the higher segment of the bar corresponds to action 1 .

We can see that under FTVA (¯ π ∗ ) , more arms choose the preferred actions for state 1 than under the random tie-breaking policy, which prevents the arms in state 1 from moving back to state 0 as we see in Figure 7b under the random tie-breaking policy.

Figure 8b is similar to Figures 8a, except that the random tie-breaking policy is replaced by the LP-Priority policy. We can again see that under LP-Priority there is a force that causes the arms to concentrate on a state, whereas switching to FTVA (¯ π ∗ ) helps the system to escape from that state.

fraction of arms fraction of arms

0.0

0.2

0.4

0.6

0.8

preferred action

non-preferred action

0

1

2

3

4

5

6

7

state

0.0

0.2

0.4

0.6

0.8

preferred action

non-preferred action

fraction of arms fraction of arms

0.0

0.1

0.2

0.3

0.4

0.5

0.6

preferred action

non-preferred action

0

1

2

3

4

5

6

7

state

0.0

0.1

0.2

0.3

0.4

0.5

preferred action

non-preferred action

The common failure mode for random tie-breaking and LP-Priority in this setting is that the arms concentrate on a bad state s , which prevents the arms on state ( s +1) from applying the preferred action due to the budget constraint. This causes a livelock where the arms move back and forth between state s and ( s +1) , and fail to gain a reward by moving past state 7 . FTVA (¯ π ∗ ) solves this issue by letting more arms in state ( s +1) follow the preferred action, which breaks the livelock and helps all arms converge to the optimal distribution.

H Generalization to heterogeneous arms

In this section, we show how FTVA and its analysis can be extended to the case with heterogenous arms. We focus on the discrete-time case for simplicity. The continuous-time case can be analyzed in a similar fashion.

H.1 Setting and algorithm

Suppose the arms are divided into K types, with β k N arms in each type k ∈ { 1 , 2 , . . . , K } ≜ [ K ] , and each type is associated with an MDP. The transition kernels and reward functions can be different across types. Suppose each arm is indexed by i ∈ [ N ] , and we denote the type of the i -th arm as k ( i ) . For any type k arm, we let P k ( s, a, s ′ ) be its probability of transitioning from state s to s ′ upon taking action a , and let r k ( s, a ) be its reward for taking action a in state s .

Consider the linear program below, whose variables y k ( s, a ) represents the steady-state probability that a type k arm is in state s taking action a , for k ∈ [ K ] , s ∈ S , a ∈ A .

where the three constraints (41), (42), and (43) correspond to (5), (6), and (7) in (LP); when writing summations, we omit ∈ [ K ] , ∈ S and ∈ A for simplicity.

(LP-Het) can be viewed as a relaxation of the N -armed problem. To see this, take any N -armed policy π and set y k ( s, a ) to be the fraction of arms in state s taking action a among type k arms in steady state under π , i.e.,

Whevener π satisfies the budget constraint (2), { y k ( s, a ) } k ∈ [ K ] ,s ∈ S ,a ∈ A satisfies (41)-(43). Therefore, the optimal value of (LP-Het), which we denote as V rel 1 , is an upper bound of the optimal value of the N -armed problem, i.e., V rel 1 ≥ V ∗ N .

The optimal solution to (LP-Het), { y ∗ k ( s, a ) } k ∈ [ K ] ,s ∈ S ,a ∈ A , induces a optimal single-armed policy ¯ π ∗ k for each type k ∈ [ K ] :

Standard argument in [Put05] show that if a type k arm runs ¯ π ∗ k , it achieves the steady-state expected reward ∑ s,a r k ( s, a ) y ∗ k ( s, a ) , and requires ∑ s y ∗ k ( s, 1) unit of budget in steady-state. If each type k arm could independently run the optimal single-armed policy of its type, since the fraction of type k arms in the N -armed system is β k , in steady state, the expected reward per arm would be V rel 1 , and the budget requirement per arm would be ∑ K k =1 ∑ s y ∗ k ( s, 1) = α , which is analogous to the homogeneous case where the optimal policy of the relaxed problem in (3) achieves V rel 1 reward and requires α expected budget in steady-state.

To convert from the single-armed policies to an N -armed policy, FTVA lets each arm of type k independently simulate a virtual single-armed process following ¯ π ∗ k . Lines 3-14 of Algorithm 1 stay the same. We denote the resulting policy as FTVA ( { ¯ π ∗ k } k ∈ [ K ] ) . The resulting policy can be proved to achieve O (1 / √ N ) optimality gap, under the assumption that Synchronization Assumption is satisfied for each ¯ π ∗ k , as stated below:

Theorem 3. In restless bandits with heterogeneous arms, let { ¯ π ∗ k } k ∈ [ K ] be the optimal single-armed policies induced by (LP-Het) . Assume that for each k ∈ [ K ] , ¯ π ∗ k satisfies Synchronization Assumption with the synchronization times { τ sync k ( s, a, ̂ s, ̂ s ) } ( s,a, ̂ s, ̂ a ) ∈ S × A × S × A . For any N ≥ 1 , the optimality gap of FTVA ( { ¯ π ∗ k } k ∈ [ K ] ) is upper bounded as

where r max ≜ max s ∈ S ,a ∈ A | r ( s, a ) | and τ sync max ≜ max k ∈ [ K ] , ( s,a, ̂ s, ̂ a ) ∈ S × A × S × A E [ τ sync k ( s, a, ̂ s, ̂ a ) ] .

H.2 Proof for Theorem 3

In this section, we prove Theorem 3.

Since V rel 1 ≥ V ∗ N , it suffices to show that

We start with an inequality that has the same form as (13) in the homogeneous case:

̸

where in the first equality, we used the fact that the virtual processes are independently running the single-armed policy of the corresponding types, so they achieve the average reward V rel 1 .

Defining the disagreement events, disagreement periods, and disagreement rates in the same way as Appendix E and applying Little's law, we have

̸

where d ( ̂ s ) is the instantaneous disagreement rate, and D avg is the long-run average length of disagreement periods. With (47) and (48), it remains to bound E [ d ( ̂ S ( ∞ )) ] and D avg .

By the definition of the policy, we have E [ d ( ̂ S ( ∞ )) ] = E [∣ ∣ ∣ ∑ N i =1 ̂ A i ( ∞ ) -αN ∣ ∣ ∣ ] , ̂ A i ( ∞ ) 's are independent Bernoulli random variables, and E [ ̂ A i ( ∞ ) ] = y ∗ k ( s, 1) if arm i is of type k . As a result,

Using the same Cauchy-Schwartz argument as in the proof of the homogeneous case in Appendix E, it is not hard to show that E [ d ( ̂ S ( ∞ )) ] ≤ √ N/ 2 .

As for D avg, it is not hard to see from the definition that the length of each disagreement period of a type k arm is stochastically dominated by τ sync k ( s, a, ̂ s, ̂ a ) , where ( s, a, ̂ s, ̂ a ) are the initial states and actions of that disagreement period. Therefore, the average length of disagreement period is bounded by

Combining the above calculations, we get (46), which finishes the proof.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the infinite-horizon restless bandits problem and proposed a simulation-based framework, i.e., Follow-the-Virtual-Advice, to leverage single armed policy to solve a multi-armed problem, which gets rid of the difficult-to-verify condition, i.e., the uniform global attractor property. Removing this pre-condition is a significant improvement for RMABs.\n\nSTRENGTHS:\nThis paper studies the infinite-horizon restless bandits problem and proposed a simulation-based framework, i.e., Follow-the-Virtual-Advice, to leverage single armed policy to solve a multi-armed problem, which gets rid of the difficult-to-verify condition, i.e., the uniform global attractor property. Removing this pre-condition is a significant improvement for RMABs.\n\nWEAKNESSES:\n1. The arguments on FTVA in Section 3.3 look to be a bit hands-waving since there is no clear evidence to support their claims. Based on the reviewer’s understandings, those arguments are possibly wrong.  For example, in lines 181-184, the authors claim that even if the initial virtual state and real state of an arm are different, they will become identical in finite time by chance under mild assumptions in Section 4.1. However, Assumption 1 relies on the “observation” in lines 233-238, which is not true. All the transitions are stochastic, not deterministic. How could we guarantee that $S(t+1) =\\hat{S}(t+1)$ if $S(t)$ is different with $\\hat{S}(t)$?  If Assumption 1 fails, the argument in lines 183-184 does not hold. The arguments in lines 185-189 are also not true. Even when the virtual process can always satisfy the budget constraint, how do we guarantee that the real process following the virtual actions does not violate the budget constraints. \n\n2. The example in lines 194-215 is also not fully supported by evidence. The single-agent policy defined in eq. (8) is stationary stochastic policy, and agent selects actions according to certain probabilities. What do you mean by the preferred action at each state? Though this particular example may only have one action at each state, there is no preferred action in general. How to guarantee the descriptions in lines 207-209 to be true?  This example shows that the real and virtual process has different state distributions, which contradicts Section 3.3. See comment 1.\n\n3. A very important related work is missing. [Ghosh22] also gets rid of the global attractor assumption and considers a much more challenging setting with heterogeneous arms and multi-actions.  An outstanding limitation of this work is that all arms must be the same. \n\nGhosh, A., Nagaraj, D., Jain, M., & Tambe, M. (2022). Indexability is Not Enough for Whittle: Improved, Near-Optimal Algorithms for Restless Bandits. arXiv preprint arXiv:2211.00112.\n\n4. The references cited in this paper are not precise. For example, as far as the reviewer knows, [Ver16] does not consider the single-armed problem. Hence, the argument in lines 140-141 are not precise.  Another example is in lines 123-124. The reviewer is not aware of the meaning of these sentences. How are they related to the RMAB problem in eqs. (1)-(2)?\n\nQUESTIONS:\nSee comments in Weaknesses.\n\nLIMITATIONS:\nno negative societal impact.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-03"}

comments: SUMMARY:
This paper presents an algorithm providing an asymptotically optimal policy to solve restless bandits with N arms when N goes to infinity without using classical assumptions on the expected deterministic dynamics (uniform global attractor).

STRENGTHS:
1. The idea to couple the actual trajectory under the relaxed policy with a virtual one, sampled independently for each bandit is a very nice idea.

2. Removing the UGAP assumption is also a strong point, although this is not the only paper going in this direction (see for example Zhang X, Frazier PI (2022) Near-optimality for infinite-horizon restless bandits with many arms. arXiv preprint arXiv:2203.15853).


3. The paper makes a good literature review and a good  use of previous work in this domain.

WEAKNESSES:
1. The discussion on the synchronization assumption, and especially the sufficient conditions could be made more precise. The link with UGAP is also missing.
Also there is a gap between the synchronization of each arm with the synchronization of the whole bandit (see for example the PhD thesis of Kimang Khun). 

2. The fact the algorithm is based on independent sampling of each arm makes the \sqrt{N} bound tight, wuth little hope for improvement, while other approaches based on the fluid dynamics could still work with UGAP (using Cezaro averaging for example) with a higher chance to have a better convergence rate.

3. The comparison with previous algorithms is a little unfair, especially because of initial conditions.

QUESTIONS:
1. Can you change the initial conditions of the numerical experiments to make a fairer comparisons with existing solutions.

2. Provide answers to the weak points listed above.

LIMITATIONS:
I did not see any limitations.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-05

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn the late 80s and early 90s, Whittle formalized the restless multi-armed bandit framework and Weber & Weiss proved that Whittle’s index policy, which solves a relaxed version of the optimization problem, is asymptotically optimal under conditions of indexability. Since then, RMAB problems have attracted much attention for their wide application in diverse areas. This paper contributes a new technique that is not an index policy. Instead, the simulation framework requires a new “synchronization assumption” for the discretized time case. The contributions of this paper are purely theoretical, including a new algorithmic approach (Follow-the-Virtual-Advice) and optimality gap bounds.\n\nSTRENGTHS:\nThis paper introduces an original technique for solving two-action RMAB problems in both discretized and continuous time spaces, and provides bounds on conversion loss and gaps in optimality for both. It is clear that the only assumption that must be enforced in the discrete case is the new “synchronization assumption”. There is a nice discussion about this assumption and when it holds in the appendix for novice readers. The submission appears to be technically sound, and claims are well supported by theoretical analysis.\n\nWEAKNESSES:\nAfter decades of essentially little-to-no change in the state-of-the-art techniques that are deployed in practice, it is wonderful to see new approaches. The authors should be made aware of [1], which aims to address the same problem and also does so in a new, novel way.\n\nI am surprised by the use of “uniform global attractor property” rather than “indexability” throughout the paper. With regards to clarity, I encourage the authors to include more explanations to inform the reader. \n\nThe contributions of this paper are purely theoretical. Open-source code as well as empirical results comparing FTVA against other state-of-the-art implementations would be helpful for the research community.\n\n[1] Abheek Ghosh, Dheeraj Nagaraj, Manish Jain, and Milind Tambe. 2023. Indexability is Not Enough for Whittle: Improved, Near-Optimal Algorithms for Restless Bandits. In Proceedings of the 2023 International Conference on Autonomous Agents and Multiagent Systems (AAMAS '23). International Foundation for Autonomous Agents and Multiagent Systems, Richland, SC, 1294–1302.\n\nQUESTIONS:\nAre the authors aware of [1]? That paper also claims to contribute a novel new approach for the discrete-time problem. Not only do they consider average-reward, but also any discounted total expected reward in both the finite and infinite horizon case. How does your approach differ? How do your optimality guarantees differ?\n\n[1] Abheek Ghosh, Dheeraj Nagaraj, Manish Jain, and Milind Tambe. 2023. Indexability is Not Enough for Whittle: Improved, Near-Optimal Algorithms for Restless Bandits. In Proceedings of the 2023 International Conference on Autonomous Agents and Multiagent Systems (AAMAS '23). International Foundation for Autonomous Agents and Multiagent Systems, Richland, SC, 1294–1302.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studied, for both the discrete-time and continuous-time settings, the average-reward restless bandit problem without the uniform global attractor property. They proposed the simulation-based policies Follow-the-Virtual-Advice (FTVA) and FTVA-CT, which converts the optimal single-armed policy into the N-armed constrained policy by simulating a virtual process that follows the single-armed policy, and then by letting the real process act closely to the virtual process but satisfying the constraints. The policies provide vanishing performance loss.\n\nSTRENGTHS:\nNovelties: In this reviewer's knowledge, this paper is the first paper to provide an algorithm for the average reward constraint RB problem without uniform global attractor property, which is a property that is hard to verify. The proposed FTVA policy is quite straightforward and also easy to understand. The idea of using virtual processes in this problem seems original to me.\n\nPresentation: The paper is well-written, with precise language and clear structure. The paper is clear and easy to understand. And there are a lot of intuitive explanations.\n\nSignificance: The paper provides the first algorithm for the average reward constraint RB problem without uniform global attractor property.\n\nWEAKNESSES:\nThe following assumption is very strong and important to this policy: the author assumed that all the arms with the same state and action have the same reward. And this policy seems very hard to extend to the case where the arms have different reward functions. (More comments in Questions.)\n\nThe paper feels incomplete, which prevents a clear conclusion at the end.\n\nThe simulation is not sufficient, the author only showed the average reward comparison for different policies, but I would also want to know how is the action look like for different policies or is there any intuition in it. \n\nThe paper presentation is too high-level and has missing details. The author claimed at first, but didn’t explain later, how the FTVA policy can be implemented in a distributed manner. The author claimed the algorithm can run at a linear computational cost but didn’t explain how, later.\n\nQUESTIONS:\nIn the setup, the author assumed that all the arms with the same state and action have the same reward. With this assumption, the problem (with the \\alpha N constraint) is greatly simplified. This assumption makes me feel that even with the \\alpha N constraint, those arms are still acting very independently, not affecting each other too much. And this policy seems very hard to extend to the case where the arms have different reward functions.\n\nIf possible, instead of only the average reward comparison, I am also curious about how the FTVA action policy look like compared to other non-asymptotically optimal policies in the example. And is there any significant different actions on some special states that makes the policy be optimal?\n\nIn the beginning, the author said that FTVA policy can be implemented in a distributed manner for arms, could you add more about it after the algorithm descriptions?\n\nFigure1 is missing one label.\n\nLIMITATIONS:\nThere is no negative societal impact. Discussion on limitations is not extensive.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work studies the restless bandit problem. In this setting the agent controls a set of $N$ arms modelled as Markov Decision Processes with two actions (active, inactive). The MDPs are coupled with a constraint that forces the agent to maintain a fixed arm activation rate. The authors propose a novel algorithm for this problem that does not use the commonly used uniform global attractor property (UGAP). The key insight of the approach is to transform a learned single arm policy to a $N$ armed policy. While this may lead to a discrepancy between the learned policy and what is actually played because of the constraint, the proposed algorithm aims to bring these two states closer together. The authors provide convergence guarantees and rates for both discrete and continuous time variants of the algorithm.\n\nSTRENGTHS:\nTo the best of my knowledge, the proposed algorithm is novel. The authors also presented very clearly the setup of the problem which is especially important given that many members of the community may not be familiar. While I did not follow all the technical details in this work, the synchronization assumption seems like a significantly weaker assumption than UGAP, indicating this a very strong contribution. On top of that, the way the algorithm solves the problem is itself highly non trivial.\n\nWEAKNESSES:\nI think clarifying a little bit more the importance of synchronization when we are using the constraint would be very helpful. I will try to outline my understanding in case it is helpful in improving the presentation:\n\n* In Section 4.1, based on the stated assumptions it is clear that achieving synchronization once is sufficient to achieve synchronization forever. But for the restless bandit setting it seems that the real and virtual states synchronize and desynchronize in a loop. Indeed the states start synchronized and some arms may immediately get desynchronized (become bad in the terminology of Theorem 1). \n* I understand that the the SA assumption helps us understand desynchronization is resolved in finite time. But if synchronization lasts only a single round, then the arms will still be bad most of the time. So SA alone is not sufficient.\n* Based on the proof sketch of Theorem 1, it seems like desynchronization is inherently rare for large $N$ so this is why SA is sufficient. Intuitively, SA is only needed to bound the effect of a single good-bad-good arm loop.\n\nIf my understanding is indeed correct, it would be helpful to make this more explicit somewhere in the text, preferably in the context of the example of Section 3.3.\n\nQUESTIONS:\nSee above for some suggestions.\n\nLIMITATIONS:\nAll limitations are addressed.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the problem of optimizing a policy for restless bandits in the infinite horizon case. A new algorithm is proposed that can convert a single-arm policy into an N-arm policy using virtual trajectories of the states and actions of the single-arm policy. The main idea is to generate virtual actions using the single-arm policy and then attempt to match these virtual actions with real actions that match the budget constraint. The algorithm is shown to have suboptimality that degrades with N. Both discrete time and continuous time versions are given. One of the main contributions is that this paper forgoes the UGAP condition.\n\nThis is an emergency review. I am not familiar enough with this literature to definitively judge the significance and novelty.\n\nSTRENGTHS:\nThe paper appears to tackle an important problem in the literature broadly (restless bandits) and provides a potential solution to optimize policies with vanishing suboptimality as N increases. \n\nAchieving this without the UGAP condition that appears in prior work is interesting.\n\nOverall the paper is well written and generally easy to understand. The examples and organization of the paper help convey these results.\n\nWEAKNESSES:\nThe main weakness appears to be that the entire premise hinges on the synchronization assumption that the real states and virtual states will eventually match, if the same actions are played, after which point they will remain identical. This seems reasonable in some settings, but a more clear contrast with conditions in other papers would be helpful.\n\nIn general, it would be helpful to better understand the significance of this change of assumptions and how this might translate into practice.\n\nQUESTIONS:\nIt is mentioned that Algorithm 1 should “steer” the trajectories towards the virtual trajectories but still respect the budget constraints. However, in Line 4, the selection of the smaller set to respect the budget does not appear to filter arms in any particular way. What prevents the same arms from being activated while starving others and thus causing the virtual trajectory to differ from the real one by a lot?\n\nSuggestion: the paper could benefit from further discussion of the last theorem and a conclusion.\n\nLIMITATIONS:\nNo negative societal impact, but it would be helpful to have more discussion of limitations.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-27"}