SlowGuess/NeurIPS_2023_OCR huggingface dataset:
1. The leader arm reaches the cycle  $C^a$  after  $t_1$  time slots;
2. The leader arm transitions along the cycle and keeps applying action 1; meanwhile, the follower arm also applies action 1 and reaches the cycle  $C^b$ ; this happens in  $t_2$  times slots; let the states of the leader and follower arm at this moment be  $s^a$  and  $s^b$ ;
3. The leader arm transitions along the cycle  $C^a$  for another  $c$  laps, for some positive integer  $c$ , and then spends another  $t_3$  time slots to go from  $s^a$  to  $s^b$ ; meanwhile, the follower arm transitions along the cycle  $C^b$  for another  $d$  laps, for some positive integer  $d$ .
The transition of the arms during the sequence of events are as illustrated in Figure 5. We argue that there exists some suitable  $t_1, t_2, t_3, c$ , and  $d$  such that the above four events happen with a positive probability, and the two arms synchronize after the four events.
The first event happens with a positive probability for suitably large  $t_1$  because  $C^a$  is in  $S$ . The second event can happen for suitably large  $t_2$  because  $\pi(1|s) > 0$  for all  $s \in C^a$ , and  $C^b$  is in  $S_1$ .
After the first two events, suppose the state of the leader arm is  $s^a$ , and the state of the follower arm is  $s^b$ , then  $s^a \in C^a$  and  $s^b \in C^b$ . Because both cycles are in  $S$ , there exists a positive probability path from  $s^a$  to  $s^b$  under  $\bar{\pi}$ . Let  $t_3$  be the length of the path. Let  $L^a$  and  $L^b$  be the length of cycles  $C^a$  and  $C^b$ . Because  $L^a$  and  $L^b$  are relatively prime, we can take  $c$  and  $d$  such that  $dL^b - cL^a = t_3$ . With this choice of  $c, d$ , and  $t_3$ , the two arms will synchronize after the third and fourth events.
Now we argue that the third and fourth events happen with a positive probability. Because  $C^a$  is a cycle under the policy  $\bar{\pi}$ , the leader arm can transition along the cycle for an arbitrary number of time slots. Similarly, because  $C^b$  is a cycle under any policy, the follower arm can transition along the cycle for an arbitrary number of time slots; the leader arm can reach  $s^b$  from  $s^a$  because of the positive probability path described in the last paragraph.
Therefore, we have proved that for any  $(s,a,\widehat{s},\widehat{a})$
$$
\mathbb {P} \left(\tau^ {\text {s y n c}} (s, a, \widehat {s}, \widehat {a}) \leq t _ {1} + t _ {2} + c L ^ {a} + t _ {3}\right) > 0.
$$
Proof of Proposition 7. Given any initial states and actions  $(S(0), A(0), \widehat{S}(0), \widehat{A}(0)) = (s, a, \widehat{s}, \widehat{a}) \in \mathbb{S} \times \mathbb{A} \times \mathbb{S} \times \mathbb{A}$ , we construct a sequence of positive probability events that leads to  $S(t) = \widehat{S}(t)$ :
1. The leader arm reaches the cycle  $C^*$  after  $t_1$  time slots;
2. The leader arm transitions along the cycle and keeps applying action 1; meanwhile, the follower arm also applies action 1 and reaches the same state as the leader arm after  $t_2$  time slots.
The transition of the arms during the sequence of events are as illustrated in Figure 5. We argue that the above two events happen with a positive probability for suitable  $t_1$  and  $t_2$ . The first event can happen for suitably large  $t_1$  because the leader arm follows the policy  $\bar{\pi}$  the cycle  $C^*$  is in the recurrent class of  $\bar{\pi}$ . After reaching  $C^*$ , the leader arm can remain on  $C^*$  because  $C^*$  is a cycle under the policy  $\bar{\pi}$ , and  $\bar{\pi}(1|s) > 0$  for all  $s \in C^*$ . The follower arm can reach the same state as the leader arm because the cycle  $C^*$  is also in the recurrent class of  $\bar{\pi}_1$ , and  $\bar{\pi}_1$  is aperiodic. Therefore, we have proved that for any  $(s, a, \widehat{s}, \widehat{a})$ ,
D On relaxing the unichain condition
Throughout this paper, we have imposed the (all-policy) unichain condition, i.e., all Markovian single-armed policies have a single recurrent class. Making this blanket assumption simplifies our presentation, but it can be substantially relaxed for many of our results, as we discuss below.
The main change caused by dropping the unichain assumption is the definition of the single-armed problem, because there could be multiple recurrent classes under a single-armed policy, each corresponding to a different long-run average reward. To adapt to this change, we can let both the single-armed policy  $\bar{\pi}$  and the initial state distribution  $\mu$  be optimization variables of the single-armed problem. We denote the average reward given  $\bar{\pi}$  and  $\mu$  as  $V_{1}^{\bar{\pi},\mu}$ . We still denote the optimal value of the single-armed problem as  $V_{1}^{\mathrm{rel}}$ . Note that the single-armed problem can still be solved by the same linear program in (LP) or (LP-CT): The optimal initial state distribution  $\mu^{*}$  can be taken as the optimal stationary state distribution  $(y^{*}(s,1) + y^{*}(s,0))_{s\in \mathbb{S}}$ , given the optimal solution of LP,  $y^{*}$ .
The definitions of FTVA and FTVA-CT do not require any change, considering that they are already using the stationary distribution of  $\bar{\pi}$  to initialize the virtual states. The conversion losses, now denoted as  $V_{1}^{\bar{\pi},\mu} - V_{N}^{\mathrm{FTVA}(\bar{\pi},\mu)}$  and  $V_{1}^{\bar{\pi},\mu} - V_{N}^{\mathrm{FTVA - CT}(\bar{\pi},\mu)}$ , have the same bound as what we currently have in equation (11) in Theorem 1 for discrete-time RBs and equation (20) in Theorem 2 for continuous-time RBs. These bounds hold as long as the given input policy  $\bar{\pi}$  has finite expected synchronization times, which can be seen by inspecting the proofs of the theorems given in Appendix E and Appendix F.
The optimality gap bounds in (12) in Theorem 1 and (21) in Theorem 2 still follow from the conversion loss bounds when the unichain assumption is dropped. To see this, for each of the discrete-time and continuous-time settings, consider an optimal single-armed policy  $\bar{\pi}^*$  and optimal initial state distribution  $\mu^{*}$ . Note that  $V_{1}^{\bar{\pi}^{*},\mu^{*}} = V_{1}^{\mathrm{rel}}\geq V_{N}^{*}$ . If  $V_{1}^{\bar{\pi}^{*},\mu^{*}} - V_{N}^{\mathrm{FTVA}(\bar{\pi}^{*},\mu^{*})} = O(1 / \sqrt{N})$ , then  $V_{N}^{*} - V_{N}^{\mathrm{FTVA}(\bar{\pi}^{*},\mu^{*})} = O(1 / \sqrt{N})$ ; if  $V_{1}^{\bar{\pi}^{*},\mu^{*}} - V_{N}^{\mathrm{FTVA - CT}(\bar{\pi}^{*},\mu^{*})} = O(1 / \sqrt{N})$ , then  $V_{N}^{*} - V_{N}^{\mathrm{FTVA - CT}(\bar{\pi}^{*},\mu^{*})} = O(1 / \sqrt{N})$ .
In the discrete-time setting, our sufficient conditions for finite expected synchronization times, presented in Propositions 4, 5, 6 and 7, are valid as long as each of the two policies  $\bar{\pi}$  and  $\bar{\pi}_1$  has a single recurrent class and the two classes intersect. Similarly, Lemma 5 in Appendix F, which establishes finite expected synchronization times for the continuous-time setting, holds under the same condition on  $\bar{\pi}$  and  $\bar{\pi}_1$ .
E Proofs for discrete-time RBs
In this section, we focus on proving the bound (11) on the conversion loss  $V_{1}^{\bar{\pi}} - V_{N}^{\mathrm{FTVA}(\bar{\pi})}$  for any single-armed policy  $\bar{\pi}$ . The asymptotic optimality result (12) in Theorem 1 will be a direct consequence of (11) if we take  $\bar{\pi}$  to be any optimal single-armed policy  $\bar{\pi}^{*}$ .
We first show that under  $\mathrm{FTVA}(\bar{\pi})$  defined in Algorithm 1, each virtual process is an independent Markov chain induced by applying  $\bar{\pi}$  to the single-armed system  $(\mathbb{S},\mathbb{A},P,r)$ , as stated in Lemma 1 below. We will rigorously prove it in the next subsection.
Lemma 1. Under FTVA(π) given in Algorithm 1, for each  $t = 0,1,2\ldots$ $i\in [N]$ ,  $\widehat{s}\in \mathbb{S}^N$ , and  $\widehat{\pmb{a}}\in \mathbb{A}^N$ ,
$$
\mathbb {P} \left(\widehat {\boldsymbol {A}} (t) = \widehat {\boldsymbol {a}} \mid \widehat {\boldsymbol {S}} (t), \dots , \widehat {\boldsymbol {S}} (0)\right) = \prod_ {i \in [ N ]} \bar {\pi} \left(\widehat {a} _ {i} \mid \widehat {S} _ {i} (t)\right) \tag {23}
$$
$$
\mathbb {P} \left(\widehat {\boldsymbol {S}} (t + 1) = \widehat {\boldsymbol {s}} \mid \widehat {\boldsymbol {S}} (t), \widehat {\boldsymbol {A}} (t), \dots , \widehat {\boldsymbol {S}} (0), \widehat {\boldsymbol {A}} (0)\right) = \prod_ {i \in [ N ]} P \left(\widehat {S} _ {i} (t), \widehat {A} _ {i} (t), \widehat {s} _ {i}\right). \tag {24}
$$
Let  $(y(s,a))_{s\in \mathbb{S},a\in \mathbb{A}}$  be the steady-state state-action distribution of the single-armed system under  $\bar{\pi}$ . By (23)(24), for each  $t\geq 0$ ,  $(\widehat{S}_i(t),\widehat{A}_i(t))$  for  $i\in [N]$  are i.i.d. with the distribution  $(y(s,a))_{s\in \mathbb{S},a\in \mathbb{A}}$ .
Next, as sketched in Section 4.2, we bound  $V_{1}^{\bar{\pi}} - V_{N}^{\mathrm{FTVA}(\bar{\pi})}$  as
$$
\begin{array}{l} V _ {1} ^ {\bar {\pi}} - V _ {N} ^ {\mathrm {F T V A} (\bar {\pi})} = \frac {1}{N} \mathbb {E} \left[ \sum_ {i = 1} ^ {N} r (\widehat {S} _ {i} (\infty), \widehat {A} _ {i} (\infty)) - \sum_ {i = 1} ^ {N} r (S _ {i} (\infty), A _ {i} (\infty)) \right] \\ \leq \frac {2 r _ {\max }}{N} \mathbb {E} \left[ \sum_ {i = 1} ^ {N} \mathbb {1} \left\{\left(\widehat {S} _ {i} (\infty), \widehat {A} _ {i} (\infty)\right) \neq \left(S _ {i} (\infty), A _ {i} (\infty)\right) \right\} \right]. \tag {13} \\ \end{array}
$$
We say an arm  $i$  is a bad arm at time  $t$  if  $\left(\widehat{S}_i(t),\widehat{A}_i(t)\right)\neq \left(S_i(t),A_i(t)\right)$ , and a good arm otherwise. Then  $\mathbb{E}\left[\sum_{i = 1}^{N}\mathbb{1}\Big\{\big(\widehat{S}_i(\infty),\widehat{A}_i(\infty)\big)\neq \big(S_i(\infty),A_i(\infty)\big)\Big\} \right] = \mathbb{E}\big[\# \text{bad arms}\big]$  in steady state.
$$
\mathbb {E} [ \# \text {b a d a r m s} ] = (\text {r a t e o f g e n e r a t i n g b a d a r m s}) \times \mathbb {E} [ \text {t i m e d u r a t i o n o f a b a d a r m} ].
$$
To make the quantities in the above expression precise, we make the following definitions.
We say there is a disagreement event of the arm  $i$  happening at time  $t$  if  $\widehat{A}_i(t) \neq A_i(t)$ . The disagreement events are the only cause that turns good arms into bad arms because otherwise by the construction of our algorithm,  $\widehat{S}_i(t)$  will remain the same as  $S_i(t)$ . The number of disagreement events in each time slot is determined by how much the budget required by virtual actions violates the constraint. We call the expected number of disagreement events when the virtual states are  $\widehat{s}$  the instantaneous disagreement rate, denoted as  $d(\widehat{s})$ .
We also define disagreement period of the arm  $i$  as a continuous period of time when arm  $i$  is a bad arm, separated by disagreement events. Formally,
Definition 2 (Disagreement period). Given a sample path of the arm  $i$ 's real states and virtual states  $(S_{i}(t),\widehat{S}_{i}(t))_{t\geq 0}$ , we define the disagreement period of the arm  $i$  as a time interval  $[t_{\mathrm{begin}},t_{\mathrm{end}} - 1]$  such that
$$
A _ {i} \left(t _ {\text {b e g i n}}\right) \neq \widehat {A} _ {i} \left(t _ {\text {b e g i n}}\right);
$$
$$
A _ {i} (t) = \widehat {A} _ {i} (t) \text {a n d} S _ {i} (t) \neq \widehat {S} _ {i} (t) \quad \forall t \in \left[ t _ {\text {b e g i n}} + 1, t _ {\text {e n d}} - 1 \right];
$$
$$
A _ {i} \left(t _ {\text {e n d}}\right) \neq \widehat {A} _ {i} \left(t _ {\text {e n d}}\right) \text {o r} S _ {i} \left(t _ {\text {e n d}}\right) = \widehat {S} _ {i} \left(t _ {\text {e n d}}\right).
$$
We use  $D_{\mathrm{avg}}$  to denote the long-run average length of the disagreement periods, i.e., the average of  $t_{\mathrm{end}} - t_{\mathrm{begin}}$  of all disagreement periods.
With the definitions, we can state Little's Law in the context of disagreement periods. We omit its proof since it is a direct consequence of Little's Law.
Lemma 2 (Little's Law for disagreement periods).
$$
\mathbb {E} \left[ \sum_ {i = 1} ^ {N} \mathbb {1} \left\{\left(\widehat {S} _ {i} (\infty), \widehat {A} _ {i} (\infty)\right) \neq \left(S _ {i} (\infty), A _ {i} (\infty)\right) \right\} \right] = \mathbb {E} \left[ d \left(\widehat {\boldsymbol {S}} (\infty)\right) \right] \cdot D _ {\text {a v g}}, \tag {25}
$$
where  $d(\widehat{s})$  denotes the instantaneous disagreement rate when the virtual states are  $\widehat{S}(t) = \widehat{s}$ ;  $D_{avg}$  denotes the long-run average length of the disagreement periods.
E.2 Lemmas and the proof of Theorem 1
In this section, we will prove Theorem 1. We first rigorously prove Lemma 1.
Proof of Lemma 1. The first equation (23) is obvious because each virtual action  $\widehat{A}_i(t)$  is sampled from  $\bar{\pi} (\cdot |\widehat{S}_i(t))$  independent of anything else. We only need to show (24). Recall that by the definition of FTVA  $(\bar{\pi})$ ,  $\widehat{S}_i(t + 1)$  could either be equal to  $S_{i}(t + 1)$  if  $(\widehat{S}_i(t),\widehat{A}_i(t)) = (S_i(t),A_i(t))$ , or generated afresh independent of anything else if  $(\widehat{S}_i(t),\widehat{A}_i(t))\neq (S_i(t),A_i(t))$ . Let  $\mathcal{I}_{\mathrm{good}}(t) = \{i\colon (\widehat{S}_i(t),\widehat{A}_i(t)) = (S_i(t),A_i(t))\}$ . Then
$$
\begin{array}{l} \mathbb {P} \left(\widehat {\boldsymbol {S}} (t + 1) = \widehat {\boldsymbol {s}} \mid \widehat {\boldsymbol {S}} (t), \widehat {\boldsymbol {A}} (t), \dots , \widehat {\boldsymbol {S}} (0), \widehat {\boldsymbol {A}} (0)\right) \\ = \prod_ {i \in \mathcal {I} _ {\text {g o o d}} (t)} \mathbb {P} \left(S _ {i} (t + 1) = \widehat {s} _ {i} \mid S _ {i} (t), A _ {i} (t)\right) \prod_ {i \notin \mathcal {I} _ {\text {g o o d}} (t)} \mathbb {P} \left(\widehat {S} _ {i} (t + 1) = \widehat {s} _ {i} \mid \widehat {S} _ {i} (t), \widehat {A} _ {i} (t)\right) \\ = \prod_ {i \in \mathcal {I} _ {\text {g o o d}} (t)} P \left(S _ {i} (t), A _ {i} (t), \widehat {s} _ {i}\right) \prod_ {i \notin \mathcal {I} _ {\text {g o o d}} (t)} P \left(\widehat {S} _ {i} (t), \widehat {A} _ {i} (t), \widehat {s} _ {i}\right) \\ = \prod_ {i \in [ N ]} P \left(\widehat {S} _ {i} (t), \widehat {A} _ {i} (t), \widehat {s} _ {i}\right), \\ \end{array}
$$
where the last equality is by the definition of  $\mathcal{I}_{\mathrm{good}}(t)$ .
Combining (23) and (24), we have confirmed that each virtual process is an independent single-armed process with transition kernel  $P$  and policy  $\bar{\pi}$ . Because the initial virtual states  $\{\widehat{S}_i(0)\}_{i\in [N]}$  are sampled i.i.d. from the stationary distribution of  $\bar{\pi}$ ,  $\{\widehat{S}_i(t)\}_{i\in [N]}$  are also i.i.d. with the same distribution. Therefore,  $\{(\widehat{S}_i(t),\widehat{A}_i(t))\}_{i\in [N]}$  are i.i.d. with distribution  $(y(s,a))_{s\in \mathbb{S},a\in \mathbb{A}}$ .
The next two lemmas bound the two quantities in Lemma 2, the long-run average disagreement rate and the long-run average length of the disagreement periods.
Lemma 3 (Average disagreement rates). The instantaneous disagreement rate is equal to
$$
d (\widehat {\boldsymbol {s}}) = \mathbb {E} \left[ \left| \sum_ {i = 1} ^ {N} \widehat {A} _ {i} (t) - \alpha N \right| \mid \widehat {\boldsymbol {S}} (t) = \widehat {\boldsymbol {s}} \right]. \tag {26}
$$
The long-run average disagreement rate is bounded as
$$
\mathbb {E} \left[ d (\widehat {\boldsymbol {S}} (\infty)) \right] \leq \frac {1}{2} \sqrt {N}. \tag {27}
$$
Proof. We first prove the expression of instantaneous disagreement rate in (26). By definition, the instantaneous disagreement rate is equal to the expected number of arms such that  $A_{i}(t) \neq \widehat{A}_{i}(t)$  conditioning on the virtual states being  $\widehat{s}$ . Because FTVA tries to match as many  $A_{i}(t)$ 's to  $\widehat{A}_{i}(t)'s$  as possible, there are exactly  $|\sum_{i=1}^{N} \widehat{A}_{i}(t) - \alpha N|$  arms such that  $A_{i}(t) \neq \widehat{A}_{i}(t)$ . This proves (26).
To bound the average disagreement rate, letting  $\widehat{s} = \widehat{S} (\infty)$  in (26) and taking expectation, we have
$$
\begin{array}{l} \mathbb {E} \left[ d (\widehat {\boldsymbol {S}} (\infty)) \right] = \mathbb {E} \left[ \left| \sum_ {i = 1} ^ {N} \widehat {A} _ {i} (\infty) - \alpha N \right| \right] \\ \leq \left(\mathbb {E} \left[ \left(\sum_ {i = 1} ^ {N} \widehat {A} _ {i} (\infty) - \alpha N\right) ^ {2} \right]\right) ^ {1 / 2}. \tag {28} \\ \end{array}
$$
By Lemma 1, the virtual actions  $\widehat{A}_i(\infty)$ 's are i.i.d. binomial random variables such that  $\mathbb{P}(\widehat{A}_i(\infty) = 1) = \sum_{s\in \mathbb{S}}y(s,1) = \alpha$ ,  $\sum_{i=1}^{N}\widehat{A}_i(\infty)$  has distribution Binomial  $(N,\alpha)$ , whose mean and variance are  $\alpha N$  and  $N\alpha(1-\alpha) \leq N/4$ . Therefore the expression in (28) is equal to  $\mathrm{Var}\left[\sum_{i=1}^{N}\widehat{A}_i(\infty)\right]^{1/2} = \sqrt{N}/2$ .
Lemma 4 (Average length of disagreement periods).
$$
D _ {\text {a v g}} \leq \bar {\tau} _ {\max } ^ {\text {s y n c}}. \tag {29}
$$
Proof. To bound  $D_{\mathrm{avg}}$ , it suffices to bound the expected length of a disagreement period with arbitrary initial states. Without loss of generality, consider a disagreement period on arm  $i$  that starts at time  $t_{\mathrm{begin}} = 0$ , with initial states  $(S_i(0), A_i(0), \widehat{S}_i(0), \widehat{A}_i(0)) = (s, a, \widehat{s}, \widehat{a})$ . During the disagreement period, the  $i$ -th arm can be seen as a leader-and-follower system, where the real state-action pair  $(S_i(t), A_i(t))$  corresponds to the follower arm and the virtual state-action pair  $(\widehat{S}_i(t), \widehat{A}_i(t))$  corresponds to the leader arm. By Assumption 1, the leader arm and the follower arm will synchronize in a finite expected time. When they synchronize, the disagreement period stops, which means  $t_{\mathrm{end}} \leq \tau^{\mathrm{sync}}(s, a, \widehat{s}, \widehat{a})$ . Therefore, the expected length of the disagreement period satisfies
$$
\mathbb {E} \left[ t _ {\text {e n d}} - t _ {\text {b e g i n}} \right] = \mathbb {E} \left[ t _ {\text {e n d}} \right] \leq \mathbb {E} \left[ \tau^ {\text {s y n c}} (s, a, \widehat {s}, \widehat {a}) \right] \leq \bar {\tau} _ {\max } ^ {\text {s y n c}}.
$$
This holds for arbitrary initial states  $(s,a,\widehat{s},\widehat{a})$  , so  $D_{\mathrm{avg}}\leq \overline{\tau}_{\mathrm{max}}^{\mathrm{sync}}$
Given the three lemmas above, we can prove Theorem 1.
Proof of Theorem 1. Combining Lemma 2, 4 and 3, we have
$$
\mathbb {E} \left[ \sum_ {i = 1} ^ {N} \mathbb {1} \Big \{(\widehat {S} _ {i} (\infty), \widehat {A} _ {i} (\infty)) \neq (S _ {i} (\infty), A _ {i} (\infty)) \Big \} \right] \leq \frac {1}{2} \overline {{\tau}} _ {\max } ^ {\mathrm {s y n c}} \sqrt {N}.
$$
Plugging the above inequality into the bound on conversion loss in (13), we get
$$
\begin{array}{l} V _ {1} ^ {\bar {\pi}} - V _ {N} ^ {\mathrm {F T V A} (\bar {\pi})} = \frac {1}{N} \mathbb {E} \left[ \sum_ {i = 1} ^ {N} r \left(\widehat {S} _ {i} (\infty), \widehat {A} _ {i} (\infty)\right) - \sum_ {i = 1} ^ {N} r \left(S _ {i} (\infty), A _ {i} (\infty)\right) \right] \\ \leq \frac {2 r _ {\mathrm {m a x}}}{N} \mathbb {E} \left[ \sum_ {i = 1} ^ {N} \mathbb {1} \Big \{\big (\widehat {S} _ {i} (\infty), \widehat {A} _ {i} (\infty) \big) \neq \big (S _ {i} (\infty), A _ {i} (\infty) \big) \Big \} \right] \\ \leq \frac {r _ {\mathrm {m a x}} \overline {{\tau}} _ {\mathrm {m a x}} ^ {\mathrm {s y n c}}}{\sqrt {N}}. \\ \end{array}
$$
F Proofs for continuous-time RBs
F.1 Preliminary: finite synchronization time
In this section, we prove that the synchronization time has a finite expectation in the continuous-time setting. Unlike the discrete-time setting, we do not need to make any additional assumptions other than the standard unichain condition. Our proof is based on the observation that the holding time distribution of a continuous-time Markov chain has support on the whole positive real line, so an arbitrary number of transitions can happen in any time interval, which, from the uniformization perspective, implies self-loops in all states. We can thus prove that synchronization time has a finite expectation using similar logic as in the proof of Proposition 3 and 4.
Lemma 5 (Synchronization in continuous time). Consider the single-armed policy  $\bar{\pi}$  and the corresponding leader-and-follower system. Given the initial states  $(S(0),\widehat{S} (0)) = (s,\widehat{s})$ , the synchronization time is bounded as
$$
\mathbb {E} \left[ \tau^ {\text {s y n c}} (s, \hat {s}) \right] <   \infty .
$$
Proof of Lemma 5. Consider the always-1 policy  $\bar{\pi}_1$  given by
$$
\bar {\pi} _ {1} (a | s) = \left\{ \begin{array}{l l} 1, & \text {i f} a = 1, \\ 0, & \text {i f} a = 0. \end{array} \right. \quad \text {f o r} s \in \mathbb {S}, a \in \mathbb {A}.
$$
By the unichain property,  $\bar{\pi}_1$  only has a single recurrent class, which we denote as  $S_1$ . We denote the recurrent class under  $\bar{\pi}$  as  $S$ .
We first prove by contradiction that  $\mathcal{S} \cap \mathcal{S}_1 \neq \emptyset$ . Suppose  $\mathcal{S} \cap \mathcal{S}_1 = \emptyset$ . Then we define a new policy  $\bar{\pi}'$  as
$$
\bar {\pi} ^ {\prime} (a | s) = \left\{ \begin{array}{l l} \bar {\pi} (a | s), & \text {i f} s \in \mathcal {S}, \\ \bar {\pi} _ {1} (a | s), & \text {o t h e r w i s e}. \end{array} \right. \quad \text {f o r} s \in \mathbb {S}, a \in \mathbb {A}.
$$
Then  $S$  and  $S_{1}$  are two distinct recurrent classes under  $\bar{\pi}'$ . This is because by definition of  $\bar{\pi}'$ , an arm with the initial state in  $S$  remains in  $S$ , so it cannot reach  $S_{1}$ ; similarly an arm cannot reach  $S$  from  $S_{1}$ . The existence of two recurrent classes contradicts the unichain condition. Therefore, we must have  $S \cap S_{1} \neq \emptyset$ .
We show that given any pair of initial states  $s, \widehat{s} \in \mathbb{S}$ , the probability that  $\tau^{\mathrm{sync}}(s, \widehat{s}) < 3$  is positive. We construct a sequence of positive probability events that leads to  $S(t) = \widehat{S}(t)$  before time 3:
1. The leader arm reaches a state  $s^a$  such that  $\bar{\pi}(1|s^a) > 0$  by time 1 and chooses action 1;
2. The leader arm has no transition during [1, 2], so  $\widehat{S}(t) = s^a$  and  $\widehat{A}(t) = 1$  for all  $t \in [1, 2]$ ;
3. The follower arm applies the same action as the leader arm during [1, 2], so  $A(t) = 1$  for all  $t \in [1, 2]$ ; and reaches the state  $s^b$  by time 2 for some  $s^b \in S \cap S_1$ ;
4. The follower arm stays at  $s^b$  during [2, 3], and the leader arm reaches  $s^b$  by time 3, so the two arms synchronize.
The first two events have positive probabilities because the policy  $\bar{\pi}$  applies action 1 with a positive fraction of time. To see why the third event has a positive probability, observe that the follower arm applies action 1 for all  $t \in [1,2]$  regardless of its state, so it is effectively under policy  $\bar{\pi}_1$  and could traverse all states in  $S_1$ . The fourth event has a positive probability because the leader arm under  $\bar{\pi}$  can traverse all states in  $S$ . Note that in the above arguments, we are implicitly assuming that the two arms do not synchronize in the middle of the four events, because otherwise we are done. Also, we use the fact that in a continuous-time Markov chain, there can be an arbitrary number of transitions during any time interval. Therefore, we have proved that for all  $s, \widehat{s} \in \mathbb{S}$ ,
$$
\mathbb {P} \left(\tau^ {\text {s y n c}} (s, \widehat {s}) <   3\right) > 0.
$$
Now we prove by induction that for all  $s, \widehat{s} \in \mathbb{S}$  and  $k = 0,1,2,\ldots$ ,
$$
\mathbb {P} \left(\tau^ {\text {s y n c}} (s, \widehat {s}) \geq 3 k\right) \leq \left(\max  _ {s ^ {\prime}, \widehat {s} ^ {\prime} \in \mathbb {S}} \mathbb {P} \left(\tau^ {\text {s y n c}} \left(s ^ {\prime}, \widehat {s} ^ {\prime}\right) \geq 3\right)\right) ^ {k}. \tag {30}
$$
The base case of  $k = 1$  is already known. Suppose we have proved (30) for a certain  $k$ , then