SlowGuess/NeurIPS_2023_OCR huggingface dataset:
In this section, we include the details of the experiments in the main body, as well as some additional experiments. In Appendix G.1 and G.2, we describe the details of the experiments of Figure 1 and 3. We conduct the simulations of these two experiments with different initial points and present the results in Appendix G.3. To give an intuitive understanding of the difference between these policies, we also display more visualization of the sample paths under different policies in Appendix G.4.
G.1 Experiment details of Figure 1
Restless bandits setting In Figure 1, we consider the discrete-time  $N$ -arm restless bandits represented by the tuple  $(N,\mathbb{S}^N,\mathbb{A}^N,P,r,\alpha N)$ . We vary the number of arms  $N$ , and keep the rest of the parameters fixed. The state space of each arm is  $\mathbb{S} = \{1,2,3\}$ . The action space is  $\mathbb{A} = \{0,1\}$ . The transition kernel  $P$  is given by
$$
P (\cdot , 0, \cdot) = \left[ \begin{array}{c c c} 0. 0 2 2 3 2 1 4 2 & 0. 1 0 2 2 9 2 8 3 & 0. 8 7 5 3 8 5 7 5 \\ 0. 0 3 4 2 6 6 0 5 & 0. 1 7 1 7 5 7 0 4 & 0. 7 9 3 9 7 6 9 1 \\ 0. 5 2 3 2 4 7 5 6 & 0. 4 5 5 2 3 2 9 8 & 0. 0 2 1 5 1 9 4 7 \end{array} \right],
$$
$$
P (\cdot , 1, \cdot) = \left[ \begin{array}{c c c} 0. 1 4 8 7 4 6 0 1 & 0. 3 0 4 3 5 8 0 9 & 0. 5 4 6 8 9 5 8 9 \\ 0. 5 6 8 4 5 7 5 4 & 0. 4 1 1 1 7 3 3 1 & 0. 0 2 0 3 6 9 1 5 \\ 0. 2 5 2 6 5 5 7 0 & 0. 2 7 3 1 0 4 3 9 & 0. 4 7 4 2 3 9 9 \end{array} \right],
$$
where the number in  $s$ -th row and  $s'$ -th column in each matrix represents  $P(s,0,s')$  or  $P(s,1,s')$ , for  $s,s' \in \{1,2,3\}$ . The reward function  $r$  is given by
$$
r (\cdot , 0) = \left[ \begin{array}{c c c} 0 & 0 & 0 \end{array} \right],
$$
3Our simulation code can be found in the link https://github.com/YigeHong/rb-break-ugap-ftva.
$$
r (\cdot , 1) = \left[ \begin{array}{c c c} 0. 3 7 4 0 1 5 5 2 & 0. 1 1 7 4 0 8 1 4 & 0. 0 7 8 6 6 1 3 5 \end{array} \right],
$$
where the  $s$ -th entry of each vector represents  $r(s,0)$  or  $r(s,1)$ , for  $s \in \{1,2,3\}$ . The budget parameter  $\alpha = 0.4$ , so  $0.4N$  arms are pulled in each time slot. This setting is taken from the Appendix E of [GGY20] as a counterexample to the UGAP assumption. We note that this RB problem obviously satisfies Synchronization Assumption for any policy: observe that  $P(s,a,s') > 0$  for all  $s, s' \in \mathbb{S}, a \in \mathbb{A}$ , so two arms with any initial states have a positive probability of synchronizing in the next time slot, which implies finite synchronization time by Proposition 2.
Simulation setting We plot the long-run average reward against the number of arms for different policies. The number of arms  $N$  varies from 100, 200, ..., 1000. Three policies are considered: our policy  $\mathrm{FTVA}(\bar{\pi}^{*})$ , the Whittle's index policy [Whi88], and an LP-Priority policy [Ver16]. For each data point, we obtain the long-run average reward and its confidence interval by simulating 50 independent trajectories, each with a length of 1000 time slots. The initial states of all arms are simply chosen to be 1, because simulation results do not quite depend on the initial states (see Appendix G.3).
Detail of policies We implement Whittle index policy and LP-Priority in the standard way. The resulting priorities of both policies turn out to be  $1 > 2 > 3$  (state 1 has the highest priority). This is not a coincidence given that the solution of the single-armed problem (LP) is:
$$
y ^ {*} (\cdot , \cdot) = \left[ \begin{array}{c c} 0 & 0. 2 9 9 4 3 \\ 0. 2 3 7 6 8 & 0. 1 0 0 5 7 \\ 0. 3 6 2 3 2 & 0 \end{array} \right],
$$
where the  $s$ -th row represents  $y^{*}(s,0)$  and  $y^{*}(s,1)$ , for  $s \in \{1,2,3\}$ . This solution implies that a reasonable policy should almost always pull arms in state 1, pull arms in state 2 for a certain fraction of time, and almost never pull arms in state 3. Therefore the only reasonable priority in this RB setting is  $1 > 2 > 3$ .
Our policy  $\mathrm{FTVA}(\bar{\pi}^{*})$  is implemented according to the pseudocode in Section 3. The non-trivial detail here is the tie-breaking rule for selecting the set of arms to activate based on the virtual actions. Our tie-breaking rule is to select  $A_{i}(t)$ 's such that the number of good arms, i.e., the arms such that  $S_{i}(t) = \widehat{S}_{i}(t)$  and  $A_{i}(t) = \widehat{A}_{i}(t)$ , is maximized. We have experimented with alternative tie-breaking rules, whose performance matches our theoretical results, though sometimes their asymptotic optimality requires larger values of  $N$  to be observed. The relationship between various tie-breaking rules and their impact on finite- $N$  performances remains unclear in the present analysis, leaving it as a topic for future investigation.
G.2 Experiment details of Figure 3
Restless bandits setting In Figure 3, we consider the discrete-time  $N$ -arm bandits constructed as below. Suppose the state space for each arm is  $\mathbb{S} = \{0,1,\dots ,7\}$ . Each state has a preferred action, which is action 1 for states  $0,1,2,3$ , and action 0 otherwise. For an arm in state  $s$ , applying the preferred action moves the arm to state  $(s + 1)$  mod 8 with probability  $p_{s,\mathrm{R}}$ , and applying the other action moves the arm to state  $(s - 1)^+$  with probability  $p_{s,\mathrm{L}}$ . The probabilities  $p_{\cdot ,\mathrm{R}}$  and  $p_{\cdot ,\mathrm{L}}$  are given by
$$
p _ {\cdot , \mathrm {R}} = \left[ \begin{array}{c c c c c c c c} 0. 1 & 0. 1 & 0. 1 & 0. 1 & 0. 1 & 0. 1 & 0. 1 & 0. 1 \end{array} \right],
$$
$$
p _ {\cdot , \mathrm {L}} = \left[ \begin{array}{c c c c c c c c} 1. 0 & 1. 0 & 0. 4 8 & 0. 4 7 & 0. 4 6 & 0. 4 5 & 0. 4 4 & 0. 4 3 \end{array} \right].
$$
When the arm transitions from 7 to 0, one unit of reward is generated. Equivalently, if we consider the expected reward of applying an action at a certain state, we can define the reward function as  $r(7,0) = p_{7,\mathrm{R}}$ , and  $r(s,a) = 0$  for all other  $s \in \mathbb{S}, a \in \mathbb{A}$ . The parameter  $\alpha = 1/2$ , so  $N/2$  arms are activated in each time slot.
Simulation setting We plot the long-run average reward against the number of arms for different policies. The number of arms  $N$  varies from 100, 200, ..., 1000. Three policies are considered: our policy  $\mathrm{FTVA}(\bar{\pi}^{*})$ , a random tie-breaking policy, and a particular LP-Priority policy that prioritizes arms with larger Lagrangian optimal indices (see the definition of Lagrangian optimal indices in
Here the subscript L means "left", and R means "right". We are imagining the arms being lined up in a row from state 0 to state 7, and the preferred action moves an arm to the right.
[HF17, BS20, GGY22]). This particular LP-Priority policy is also referred to as the LP-Index policy in the literature. But for simplicity, we just refer to it as LP-Priority in this paper. For each data point, we obtain the long-run average reward and its confidence interval by simulating 50 independent trajectories, each with a length of 1000 time slots. The initial states for the simulations are fixed:  $N/3$  arms are in state 1 and  $2N/3$  arms are in state 2. Note that we fix this initial point in the simulation of Figure 3 in order to demonstrate that both the random tie-breaking and LP-Priority can get nearly zero rewards in this example. For other choices of fixed points, there is also a strong separation between the performance of FTVA( $\bar{\pi}^*$ ) and random tie-breaking or LP-Priority, which we will show in the next section.
Details of policies The optimal solution of (LP) is  $y^{*}(s,1) = 1 / 8$  for  $s = 0,1,2,3$ ,  $y^{*}(s,0) = 1 / 8$  for  $s = 4,5,6,7$ , and  $y^{*}(s,a) = 0$  for other  $s\in \mathbb{S}$  and  $a\in \mathbb{A}$ . Note that the same  $y^{*}$  remains optimal even if we remove the budget constraint. The optimal LP solution suggests that a reasonable policy should almost always pull arms in states 0, 1, 2, 3, and almost never pull arms in states 4, 5, 6, 7.
The random tie-breaking policy that we consider prioritizes arms whose states are in  $\{0,1,2,3\}$  over arms whose states are in  $\{4,5,6,7\}$ , and it breaks ties uniformly at random when there are more than  $N / 2$  arms in either of the two sets.
The LP-Priority policy we consider prioritizes arms with larger Lagrangian optimal indices [BS20, GGY22]. Specifically, it involves solving the Lagrangian relaxation of the original LP, which replaces the budget constraint with a penalty term determined by the optimal Lagrange multiplier. In our setting, because the optimal solution  $y^{*}$  remains optimal even without the budget constraint, we can simply remove the budget constraint to get the Lagrangian relaxation. The resulting Lagrangian optimal indices are given by:
$$
\left[ \begin{array}{c c c c c c c} 0. 0 1 2 5 & 0. 1 3 7 5 & 0. 0 7 2 5 & 0. 0 7 1 2 5 & - 0. 0 7 - 0. 0 6 8 7 5 & - 0. 0 6 7 5 & - 0. 0 6 6 2 5 \end{array} \right],
$$
and the priority is  $1 > 2 > 3 > 0 > 7 > 6 > 5 > 4$ .
The implementation of our policy  $\mathrm{FTVA}(\bar{\pi}^{*})$  is the same as in the last experiment.
Further discussions of the policies As shown in Figure 3, the random tie-breaking policy and the LP-Priority policy get nearly zero rewards. We have discussed why this happens for the random tie-breaking policy in Section 3.3. For the LP-Priority policy based on the Lagrangian optimal index, the reason why it does not work is less obvious. Some sample paths suggest that under this policy, the arms will concentrate on  $\{3,4,5\}$  most of the time and thus cannot get a reward. Here is a possible explanation for why arms cannot easily escape from  $\{3,4,5\}$ : when some of the arms transition to state 6, by the priority  $6 > 5 > 4$ , those arms in state 6 are likely to be activated. Because states 6 do not prefer action 1, those arms will transition back to state 5 and thus get trapped.
In addition to the numerical results, we also note that our policy  $\mathrm{FTVA}(\bar{\pi}^{*})$  is provably asymptotically optimal in this RB problem even though its transition kernel is not unichain. This stems from the fact that the optimal single-armed policy  $\bar{\pi}^{*}$  has a single recurrent class  $\mathbb{S}$ , and satisfies Synchronization Assumption. See Appendix D for the discussion on the conditions under which Synchronization Assumption and Theorem 1 hold when unichain is not assumed.
G.3 Varying the initial points of the Figure 1 and 3 examples
In Figure 1 and Figure 3, we have fixed one initial point for each simulation, for the ease of presentation. In this section, we rerun the simulations for these two settings with more initial points. For the simulations of each setting, we generate initial states in the following way.
- We first choose a probability distribution on the state space  $\mathbb{S}$  from all the possible probability distributions on  $\mathbb{S}$  uniformly at random. Let  $\pi(s)$  denote the probability of states under the chosen distribution. This distribution is chosen independently across the 20 sets of simulations.
- For each  $N$ -armed problem, we set the initial state such that  $N \cdot \pi(s)$  arms are in state  $s$  for each  $s$ , with proper rounding.
- For each  $N$ -armed problem, we run all the policies with this initial state.
<sup>5</sup>A nuance is that the optimal Lagrange multiplier for the budget constraint is not unique in this setting, so there can be different Lagrange relaxations. We focus on the simplest one.
(a) The experiment of Figure 1
(b) The experiment of Figure 3
Figure 6: We re-run the two experiments of Figure 1 and 3 with 20 sets of simulation with randomly generated initial points. For FTVA, we calculate the minimum average reward over the 20 sets; for the other policies, we calculate the maximum average reward over the 20 sets.
We illustrate the results of the 20 sets of simulations in the Figure 6a and 6b in the following way: For  $\mathrm{FTVA}(\bar{\pi}^{*})$ , for each  $N$ , we take the minimum average reward over the 20 sets of simulations and plot them in the figure; For other policies, for each  $N$ , we take the maximum average reward over the 20 sets of simulations. We can see that the min curve for  $\mathrm{FTVA}(\bar{\pi}^{*})$  still approaches the optimal value as  $N$  increases, whereas the max curves for other policies are strictly separated from the optimal value.
G.4 Visualization of the policies
In Figure 2 of Section 3.3, we illustrate what LP-Priority and FTVA  $(\bar{\pi}^{*})$  does differently by visualizing their sample paths. Here we include more such figures.
We still use the same example defined in Section 3.3. To recap, this example has 8 states, numbered as  $\{0,1,2,3,4,5,6,7\}$ . The optimal single-armed policy always chooses the preferred action in each state, which is 1 for states  $0,1,2,3$ , and 0 for states  $4,5,6,7$ . Under the optimal single-armed policy, each arm moves from state  $s$  to  $(s + 1)$  mod 8 with probability 0.1 or stays at  $s$  with probability 0.9. The optimal distribution for the single-armed system is uniform over the 8 states.
When simulating the policies in the  $N$ -armed system, we can tell that a policy is good if the fraction of arms in each state is roughly uniform, and bad if the arms concentrate on a small number of states. We can also see why this happens by looking at whether the preferred action is chosen, and whether the arms are moving from  $s$  to  $(s + 1)$  mod 8.
The three heatmaps in Figure 7a illustrate how the fractions of arms in each state change over time under the three policies: random tie-breaking, LP-Priority, and FTVA $(\bar{\pi}^{*})$ . The x-axis represents the time slot, which ranges from 0 to 499; the y-axis represents the states; the brighter color represents a larger fraction of arms in this state at this time, and the specific value represented by each color can be found in the color bar on the right.
We can see that under the random tie-breaking, the arms concentrate around state 0; under LP-Priority, the arms concentrate around states  $\{3,4,5\}$  after a burn-in period; under FTVA  $(\bar{\pi}^{*})$ , the arms uniformly spread out over the 8 states.
In Figure 7b, we take a closer look at the sample path of the random tie-breaking policy from time 250 to 289 and contrast it with the sample path if the system uses  $\mathrm{FTVA}(\bar{\pi}^{*})$  from time 250 onwards. We also add some arrows indicating the drift of the arms, i.e., the average direction that the arms in this state are moving into. The colors of the arrows represent the direction: the blue arrow implies that the arm takes the preferred action and moves in the correct direction, while the red arm implies that the arm takes the non-preferred action and moves in the wrong direction. The magnitudes of the arrows represent how fast they are moving.
We can see that under random tie-breaking, although the arms have a strong tendency to move up from state 0 to state 1, they move back when they reach state 1 and thus get stuck at state 0. In
(a) Time evolution of the fractions of arms in each state under the three policies during the first 500 time slots. Top to bottom: Random Tie-Breaking, LP-Priority, and FTVA $(\bar{\pi}^{*})$ .
(b) Time evolution of the fraction of arms in each state under Random Tie-Breaking (upper), or after switching to  $\mathrm{FTVA}(\bar{\pi}^{*})$  (lower) since time slot 250. The color and magnitude of the arrows represent the average movement of the arms in each state.
Figure 7: Visualization of the sample paths under different policies for the example in Section 3.3.
contrast, when switching to  $\mathrm{FTVA}(\bar{\pi}^{*})$ , most arrows point in the right direction, which implies that the arms consistently apply the preferred actions. As a result,  $\mathrm{FTVA}(\bar{\pi}^{*})$  helps the arms to escape from state 0 and converge to the uniform distribution over the state space.
Figure 8a illustrates the fraction of arms in state  $s$  taking action  $a$  in a few time slots after 250 in the same sample path of the random tie-breaking policy in Figure 7a. For each  $s$  and  $a$ , The upper bar plot is under the random tie-breaking policy, while the lower bar plot is after switching to  $\mathrm{FTVA}(\bar{\pi}^{*})$ . The x-axis represents the state; the y-axis represents the fraction; there are four bars in each state, corresponding to 4 time-steps. Each bar has two segments, where the length of the blue segment indicates the fraction of arms taking the preferred action in this state, and the length of the red segment indicates the fraction of arms taking the non-preferred action in this state; the lower segment of the bar corresponds to action 0, and the higher segment of the bar corresponds to action 1.
We can see that under  $\mathrm{FTVA}(\bar{\pi}^{*})$ , more arms choose the preferred actions for state 1 than under the random tie-breaking policy, which prevents the arms in state 1 from moving back to state 0 as we see in Figure 7b under the random tie-breaking policy.
Figure 8b is similar to Figures 8a, except that the random tie-breaking policy is replaced by the LP-Priority policy. We can again see that under LP-Priority there is a force that causes the arms to concentrate on a state, whereas switching to  $\mathrm{FTVA}(\bar{\pi}^{*})$  helps the system to escape from that state.
(a) The fraction of arms in each state taking each action in the 4 time steps following time 250, under Random Tie-Breaking (upper plot) or after switching to  $\mathrm{FTVA}(\bar{\pi}^{*})$  (lower plot).
(b) The fraction of arms in each state taking each action in the 4 time steps following time 250, under LP-Priority (upper plot) or after switching to FTVA  $(\bar{\pi}^{*})$  (lower plot).
Figure 8: Actions taken by different policies for the example in Section 3.3. In each bar plot, the X-axis represents the states; the Y-axis represents the fractions. Each state has bars for the 4 time steps. The blue segment of each bar is the fraction of arms taking the preferred action; the red segment corresponds to the non-preferred action. The lower segment of each bar corresponds to action 0; the higher segment corresponds to action 1.
The common failure mode for random tie-breaking and LP-Priority in this setting is that the arms concentrate on a bad state  $s$ , which prevents the arms on state  $(s + 1)$  from applying the preferred action due to the budget constraint. This causes a livelock where the arms move back and forth between state  $s$  and  $(s + 1)$ , and fail to gain a reward by moving past state 7. FTVA  $(\bar{\pi}^{*})$  solves this issue by letting more arms in state  $(s + 1)$  follow the preferred action, which breaks the livelock and helps all arms converge to the optimal distribution.
H Generalization to heterogeneous arms
In this section, we show how FTVA and its analysis can be extended to the case with heterogenous arms. We focus on the discrete-time case for simplicity. The continuous-time case can be analyzed in a similar fashion.
H.1 Setting and algorithm
Suppose the arms are divided into  $K$  types, with  $\beta_{k}N$  arms in each type  $k\in \{1,2,\dots ,K\} \triangleq [K]$  and each type is associated with an MDP. The transition kernels and reward functions can be different across types. Suppose each arm is indexed by  $i\in [N]$ , and we denote the type of the  $i$ -th arm as  $k(i)$ . For any type  $k$  arm, we let  $P_{k}(s,a,s^{\prime})$  be its probability of transitioning from state  $s$  to  $s^{\prime}$  upon taking action  $a$ , and let  $r_k(s,a)$  be its reward for taking action  $a$  in state  $s$ .
Consider the linear program below, whose variables  $y_{k}(s,a)$  represents the steady-state probability that a type  $k$  arm is in state  $s$  taking action  $a$ , for  $k\in [K]$ ,  $s\in \mathbb{S}$ ,  $a\in \mathbb{A}$ .
$$
\underset {\{y _ {k} (s, a) \} _ {k \in [ K ], s \in \mathbb {S}, a \in \Lambda}} {\text {m a x i m i z e}} \sum_ {k} \sum_ {s, a} \beta_ {k} r _ {k} (s, a) y _ {k} (s, a) \tag {LP-Het}
$$
$$
\text {s u b j e c t} \quad \sum_ {k} \sum_ {s} \beta_ {k} y _ {k} (s, 1) = \alpha , \tag {41}
$$
$$
\sum_ {s ^ {\prime}, a} y _ {k} \left(s ^ {\prime}, a\right) P _ {k} \left(s ^ {\prime}, a, s\right) = \sum_ {a} y _ {k} (s, a), \quad \forall k \in [ K ], s \in \mathbb {S}, \tag {42}
$$
$$
\sum_ {s, a} y _ {k} (s, a) = 1 \forall k \in [ K ]; \tag {43}
$$
$$
y _ {k} (s, a) \geq 0, \forall k \in [ K ], s \in \mathbb {S}, a \in \mathbb {A}.
$$
where the three constraints (41), (42), and (43) correspond to (5), (6), and (7) in (LP); when writing summations, we omit  $\in [K], \in \mathbb{S}$  and  $\in \mathbb{A}$  for simplicity.
(LP-Het) can be viewed as a relaxation of the  $N$ -armed problem. To see this, take any  $N$ -armed policy  $\pi$  and set  $y_{k}(s,a)$  to be the fraction of arms in state  $s$  taking action  $a$  among type  $k$  arms in steady state under  $\pi$ , i.e.,
$$
y _ {k} (s, a) = \frac {1}{\beta_ {k} N} \mathbb {E} \left[ \sum_ {k (i) = k} \mathbb {1} _ {\{S _ {i} ^ {\pi} (\infty) = s, A _ {i} ^ {\pi} (\infty) = a \}} \right].
$$
Whevener  $\pi$  satisfies the budget constraint (2),  $\{y_{k}(s,a)\}_{k\in [K],s\in \mathbb{S},a\in \mathbb{A}}$  satisfies (41)-(43). Therefore, the optimal value of (LP-Het), which we denote as  $V_{1}^{\mathrm{rel}}$ , is an upper bound of the optimal value of the  $N$ -armed problem, i.e.,  $V_{1}^{\mathrm{rel}}\geq V_{N}^{*}$
The optimal solution to (LP-Het),  $\{y_k^*(s, a)\}_{k \in [K], s \in \mathbb{S}, a \in \mathbb{A}}$ , induces a optimal single-armed policy  $\bar{\pi}_k^*$  for each type  $k \in [K]$ :
$$
\bar {\pi} _ {k} ^ {*} (a | s) = \left\{ \begin{array}{l l} y _ {k} ^ {*} (s, a) / \left(y _ {k} ^ {*} (s, 0) + y _ {k} ^ {*} (s, 1)\right), & \text {i f} y _ {k} ^ {*} (s, 0) + y _ {k} ^ {*} (s, 1) > 0, \\ 1 / 2, & \text {i f} y _ {k} ^ {*} (s, 0) + y _ {k} ^ {*} (s, 1) = 0. \end{array} \quad \text {f o r} s \in \mathbb {S}, a \in \mathbb {A}. \right. \tag {44}
$$
Standard argument in [Put05] show that if a type  $k$  arm runs  $\bar{\pi}_k^*$ , it achieves the steady-state expected reward  $\sum_{s,a}r_k(s,a)y_k^* (s,a)$ , and requires  $\sum_{s}y_{k}^{*}(s,1)$  unit of budget in steady-state. If each type  $k$  arm could independently run the optimal single-armed policy of its type, since the fraction of type  $k$  arms in the  $N$ -armed system is  $\beta_{k}$ , in steady state, the expected reward per arm would be  $V_{1}^{\mathrm{rel}}$ , and the budget requirement per arm would be  $\sum_{k = 1}^{K}\sum_{s}y_{k}^{*}(s,1) = \alpha$ , which is analogous to the homogeneous case where the optimal policy of the relaxed problem in (3) achieves  $V_{1}^{\mathrm{rel}}$  reward and requires  $\alpha$  expected budget in steady-state.
To convert from the single-armed policies to an  $N$ -armed policy, FTVA lets each arm of type  $k$  independently simulate a virtual single-armed process following  $\bar{\pi}_k^*$ . Lines 3-14 of Algorithm 1 stay the same. We denote the resulting policy as FTVA( $\{\bar{\pi}_k^*\}_{k \in [K]}$ ). The resulting policy can be proved to achieve  $O(1/\sqrt{N})$  optimality gap, under the assumption that Synchronization Assumption is satisfied for each  $\bar{\pi}_k^*$ , as stated below:
Theorem 3. In restless bandits with heterogeneous arms, let  $\{\bar{\pi}_k^*\}_{k\in [K]}$  be the optimal single-armed policies induced by (LP-Het). Assume that for each  $k\in [K]$ ,  $\bar{\pi}_k^*$  satisfies Synchronization Assumption with the synchronization times  $\{\tau_k^{\mathrm{sync}}(s,a,\widehat{s},\widehat{s})\}_{(s,a,\widehat{s},\widehat{a})\in \mathbb{S}\times \mathbb{A}\times \mathbb{S}\times \mathbb{A}}$ . For any  $N\geq 1$ , the optimality gap of FTVA  $(\{\bar{\pi}_k^*\}_{k\in [K]})$  is upper bounded as
$$
V _ {N} ^ {*} - V _ {N} ^ {\mathrm {F T V A} \left(\left\{\bar {\pi} _ {k} ^ {*} \right\} _ {k \in [ K ]}\right)} \leq \frac {r _ {\max } \bar {\tau} _ {\max } ^ {\text {s y n c}}}{\sqrt {N}}, \tag {45}
$$