SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} \mathbb {P} \left(\tau^ {\text {s y n c}} (s, \widehat {s}) \geq 3 (k + 1)\right) = \mathbb {P} \left(\tau^ {\text {s y n c}} (s, \widehat {s}) \geq 3 k\right) \cdot \mathbb {P} \left(\tau^ {\text {s y n c}} (s, \widehat {s}) \geq 3 (k + 1) \mid \tau^ {\text {s y n c}} (s, \widehat {s}) \geq 3 k\right) \\ = \mathbb {P} \left(\tau^ {\text {s y n c}} (s, \widehat {s}) \geq 3 k\right) \cdot \mathbb {P} \left(\tau^ {\text {s y n c}} (S (3 k), \widehat {S} (3 k)) \geq 3 \mid S (3 k) \neq \widehat {S} (3 k)\right) \\ \leq \mathbb {P} \left(\tau^ {\text {s y n c}} (s, \widehat {s}) \geq 3 k\right) \cdot \max  _ {s ^ {\prime}, \widehat {s} ^ {\prime} \in \mathbb {S}} \mathbb {P} \left(\tau^ {\text {s y n c}} \left(s ^ {\prime}, \widehat {s} ^ {\prime}\right) \geq 3\right) \\ \leq \Big (\max  _ {s ^ {\prime}, \widehat {s} ^ {\prime} \in \mathbb {S}} \mathbb {P} \left(\tau^ {\text {s y n c}} \left(s ^ {\prime}, \widehat {s} ^ {\prime}\right) \geq 3\right) \Big) ^ {k + 1}. \\ \end{array}
$$
where in the second equality we have used the Markov property of the system, and in the last inequality we apply the induction hypothesis. This proves (30).
The bound on the expectation  $\mathbb{E}\left[\tau^{\mathrm{sync}}(s,\widehat{s})\right]$  follows summing the tail bound (30) over  $k = 0,1,2,\ldots$
$$
\begin{array}{l} \mathbb {E} \left[ \tau^ {\text {s y n c}} (s, \widehat {s}) \right] \leq \sum_ {k = 0} ^ {\infty} 3 \mathbb {P} \left(\tau^ {\text {s y n c}} (s, \widehat {s}) \geq 3 k\right) \\ \leq \frac {3}{1 - \max  _ {s ^ {\prime} , \widehat {s} ^ {\prime} \in \mathbb {S}} \mathbb {P} \left(\tau^ {\text {s y n c}} \left(s ^ {\prime} , \widehat {s} ^ {\prime}\right) \geq 3\right)} \\ <   \infty , \\ \end{array}
$$
where the second inequality is due to (30) and the fact that  $\max_{s',\widehat{s}'\in \mathbb{S}}\mathbb{P}(\tau^{\mathrm{sync}}(s',\widehat{s}')\geq 3)\leq 1 - \min_{s',\widehat{s}'\in \mathbb{S}}\mathbb{P}(\tau^{\mathrm{sync}}(s',\widehat{s}') < 3) < 1$ . This finishes the proof.
F.2 Overview of the proof of Theorem 2
In this section, we focus on proving the bound (20) on the conversion loss  $V_{1}^{\bar{\pi}} - V_{N}^{\mathrm{FTVA-CT}(\bar{\pi})}$  for any single-armed policy  $\bar{\pi}$ . The asymptotic optimality result (21) in Theorem 2 will be a direct consequence of (20) if we take  $\bar{\pi}$  to be any optimal single-armed policy  $\bar{\pi}^{*}$ .
We first show that under FTVA-CT(  $\bar{\pi}$  ) defined in Algorithm 2,  $\{(S(t),\widehat{S} (t))\}_{t\geq 0}$  is a continuous-time Markov chain and each virtual process is an independent Markov chain induced by applying  $\bar{\pi}$  to the single-armed system, as stated in Lemma 6 below. We will prove it in the next subsection.
Lemma 6. Under the FTVA-CT(π) defined in Algorithm 2, we have the following:
(1)  $\{(\pmb {S}(t),\widehat{\pmb{S}} (t))\}_{t\geq 0}$  is a continuous-time Markov chain.
(2)  $\{\widehat{S}_i(t)\}_{t\geq 0}$  for  $i\in [N]$  are  $N$  independent Markov chains, whose transition rate from state  $s$  to  $s^{\prime}$  is  $\sum_{a\in \mathbb{A}}G(s,a,s^{\prime})\bar{\pi} (a|s),$  for  $s,s^{\prime}\in \mathbb{S}$  s.t.  $s\neq s^{\prime}$
Let  $(y(s,a))_{s\in \mathbb{S},a\in \mathbb{A}}$  be the steady-state state-action distribution of the single-armed system under  $\bar{\pi}$ . Then the second result above implies that for each  $t\geq 0$ ,  $(\widehat{S}_i(t),\widehat{A}_i(t))$  for  $i\in [N]$  are i.i.d. with the distribution  $(y(s,a))_{s\in \mathbb{S},a\in \mathbb{A}}$ .
Next, we can upper bound the conversion loss  $V_{1}^{\bar{\pi}} - V_{N}^{\mathrm{FTVA-CT}(\bar{\pi})}$  as
$$
\begin{array}{l} V _ {1} ^ {\bar {\pi}} - V _ {N} ^ {\mathrm {F T V A - C T} (\bar {\pi})} = \frac {1}{N} \mathbb {E} \left[ \sum_ {i = 1} ^ {N} r \left(\widehat {S} _ {i} (\infty), \widehat {A} _ {i} (\infty)\right) - \sum_ {i = 1} ^ {N} r \left(S _ {i} (\infty), A _ {i} (\infty)\right) \right] \\ \leq \frac {2 r _ {\operatorname* {m a x}}}{N} \mathbb {E} \left[ \sum_ {i = 1} ^ {N} \mathbb {1} \left\{\left(\widehat {S} _ {i} (\infty), \widehat {A} _ {i} (\infty)\right) \neq \left(S _ {i} (\infty), A _ {i} (\infty)\right) \right\} \right] \\ \leq \frac {2 r _ {\operatorname* {m a x}}}{N} \mathbb {E} \left[ \sum_ {i = 1} ^ {N} \mathbb {1} \left\{\widehat {S} _ {i} (\infty) \neq S _ {i} (\infty) \right\} + \sum_ {i = 1} ^ {N} \mathbb {1} \left\{\widehat {A} _ {i} (\infty) \neq A _ {i} (\infty) \right\} \right]. \tag {31} \\ \end{array}
$$
The bound is slightly different from the (13) since we want to separately deal with the number of arms whose real and virtual states do not agree and those whose real and virtual actions do not agree, as shown in the lemma below.
Lemma 7. It holds that
$$
\mathbb {E} \left[ \sum_ {i = 1} ^ {N} \mathbb {1} \left\{\widehat {S} _ {i} (\infty) \neq S _ {i} (\infty) \right\} \right] \leq 2 g _ {\max } \bar {\tau} _ {\max } ^ {\text {s y n c}} \mathbb {E} \left[ \left| \sum_ {i = 1} ^ {N} \widehat {A} _ {i} (\infty) - \alpha N \right| \right] \tag {32}
$$
$$
\mathbb {E} \left[ \sum_ {i = 1} ^ {N} \mathbb {1} \left\{\widehat {A} _ {i} (\infty) \neq A _ {i} (\infty) \right\} \right] = \mathbb {E} \left[ \left| \sum_ {i = 1} ^ {N} \widehat {A} _ {i} (\infty) - \alpha N \right| \right] \tag {33}
$$
Before showing the proof of Lemma 7, we will first use Lemma 7 to prove Theorem 2.
Proof of Theorem 2. Combining (31) and Lemma 7, we have
$$
V _ {1} ^ {\bar {\pi}} - V _ {N} ^ {\mathrm {F T V A - C T} (\bar {\pi})} \leq \frac {2 r _ {\max}}{N} (1 + 2 g _ {\max} \overline {{\tau}} _ {\max} ^ {\mathrm {s y n c}}) \mathbb {E} \left[ \left| \sum_ {i = 1} ^ {N} \widehat {A} _ {i} (\infty) - \alpha N \right| \right].
$$
By Lemma 6, the virtual actions  $\widehat{A}_i(\infty)$ 's are i.i.d. binomial random variables such that  $\mathbb{P}(\widehat{A}_i(\infty) = 1) = \sum_{s\in \mathbb{S}}y(s,a) = \alpha$ , so the distribution of  $\sum_{i=1}^{N}\widehat{A}_i(\infty)$  is Binomial  $(N,\alpha)$ , whose mean and variance are  $\alpha N$  and  $N\alpha(1-\alpha) \leq N/4$ . Therefore,
$$
\mathbb {E} \left[ \left| \sum_ {i = 1} ^ {N} \widehat {A} _ {i} (\infty) - \alpha N \right| \right] \leq \left(\mathbb {E} \left[ \left(\sum_ {i = 1} ^ {N} \widehat {A} _ {i} (\infty) - \alpha N\right) ^ {2} \right]\right) ^ {1 / 2}
$$
$$
\begin{array}{l} = \operatorname {V a r} \left[ \sum_ {i = 1} ^ {N} \widehat {A} _ {i} (\infty) \right] ^ {1 / 2} \\ \leq \frac {1}{2} \sqrt {N}. \\ \end{array}
$$
Combining the above calculations, we get
$$
V _ {1} ^ {\bar {\pi}} - V _ {N} ^ {\mathrm {F T V A - C T} (\bar {\pi})} \leq \frac {r _ {\max} (1 + 2 g _ {\max} \overline {{\tau}} _ {\max} ^ {\mathrm {s y n c}})}{\sqrt {N}}.
$$
F.3 Proof of Lemma 6
Proof. We first show that  $\{(S(t),\widehat{S} (t))\}_{t\geq 0}$  is a continuous-time Markov chain. Observe that  $\{(S(t),\widehat{S} (t))\}_{t\geq 0}$  is piecewise constant between decision epochs  $\{t_k\}_{k = 0,1,\ldots}$ , so it suffices to consider the time between the decision epochs  $t_{k + 1} - t_k$  and the states at the decision epochs  $(S(t_{k}),\widehat{S} (t_{k}))$  for  $k = 0,1,2,\ldots$
We claim that at each decision epoch  $t_k$ , the time until the next decision epoch  $t_{k+1} - t_k$  is exponentially distributed conditioned on  $(S(t_k), \widehat{S}(t_k))$ . Observe from the pseudo-code in Algorithm 2 that at the decision epoch  $t_k$ , conditioned on  $(S(t_k), A(t_k), \widehat{S}(t_k))$ , the time until the next decision epoch  $t_{k+1}$  is the minimum of two independent exponential random variables, corresponding to the exponential timer and the transition of real states. The rates of the two exponential random variables are  $2Ng_{\max} - g_k^{\mathrm{real}}$  and  $g_k^{\mathrm{real}}$ , where  $g_k^{\mathrm{real}} = \sum_{i=1}^{N} G(S_i(t_k), A_i(t_k))$ . Therefore, conditioned on  $(S(t_k), A(t_k), \widehat{S}(t_k))$ ,  $t_{k+1} - t_k$  has an exponential distribution with rate  $2Ng_{\max} - g_k^{\mathrm{real}} + g_k^{\mathrm{real}} = 2Ng_{\max}$ . Because the rate  $2Ng_{\max}$  is a constant, if we take expectation over  $A(t_k)$  and only conditioned on  $(S(t_k), \widehat{S}(t_k))$ , the time until the next decision epoch  $t_{k+1} - t_k$  is still exponentially distributed with rate  $2Ng_{\max}$ .
Also, it is obvious from the pseudo-code in Algorithm 2 that conditioned on  $(S(t_k),\widehat{S} (t_k))$  , the distribution of  $(S(t_{k + 1}),\widehat{S} (t_{k + 1}))$  is independent of  $t_{k + 1} - t_k$  and  $(S(t),\widehat{S} (t))$  for  $t <   t_k$  . Therefore, we conclude that  $\{(S(t),\widehat{S} (t))\}_{t\geq 0}$  is a continuous-time Markov chain.
Next, we show that the  $\{\widehat{S}_i(t)\}_{t\geq 0}$  for  $i\in [N]$  are  $N$  independent Markov chains, whose transition rate from state  $s$  to  $s^{\prime}$  is  $\sum_{a\in \mathbb{A}}G(s,a,s^{\prime})\bar{\pi} (a|s)$  for  $s^{\prime}\neq s$ . Because we have shown that  $\{(S(t),\widehat{S} (t))\}_{t\geq 0}$  is a continuous-time Markov chain with a constant transition rate, it suffices to focus on the embedded chain  $\{(\pmb {S}(t_k),\widehat{\pmb{S}} (t_k))\}_{k = 0,1,\ldots}$  and examine the probability of the transitions that change the virtual states  $\widehat{S} (t_k)$ . Note that between two decision epochs  $t_k$  and  $t_{k + 1}$ , there is at most one  $i$  such that  $\widehat{S}_i(t)$  changes. Let  $\mathcal{I}_{\mathrm{good}}(t_k) = \{i\in [N]\colon \widehat{S}_i(t_k) = S_i(t_k),\widehat{A}_i(t_k) = A_i(t_k)\}$ . For any arm  $i\in \mathcal{I}_{\mathrm{good}}(t_k)$ , the transitions of its virtual and real states are coupled, which implies that the probability that  $\widehat{S}_i(t_{k + 1}) = s'$  is the same as the probability that  $S_{i}(t_{k + 1}) = s^{\prime}$ , for any  $s^{\prime}\neq \widehat{S}_i(t_k)$ , conditioned on the states and actions at  $t_k$ . Formally,
$$
\begin{array}{l} \mathbb {P} \left(\widehat {S} _ {i} \left(t _ {k + 1}\right) = s ^ {\prime} \mid \boldsymbol {S} \left(t _ {k}\right), \boldsymbol {A} \left(t _ {k}\right), \widehat {\boldsymbol {S}} \left(t _ {k}\right), \widehat {\boldsymbol {A}} \left(t _ {k}\right)\right) \mathbb {1} \left\{i \in \mathcal {I} _ {\text {g o o d}} \left(t _ {k}\right) \right\} \\ = \mathbb {P} \left(S _ {i} \left(t _ {k + 1}\right) = s ^ {\prime} \mid \boldsymbol {S} \left(t _ {k}\right), \boldsymbol {A} \left(t _ {k}\right), \widehat {\boldsymbol {S}} \left(t _ {k}\right), \widehat {\boldsymbol {A}} \left(t _ {k}\right)\right) \mathbb {1} \left\{i \in \mathcal {I} _ {\text {g o o d}} \left(t _ {k}\right) \right\} \\ = \frac {G \left(S _ {i} \left(t _ {k}\right) , A _ {i} \left(t _ {k}\right) , s ^ {\prime}\right)}{2 N g _ {\max }} \mathbb {1} \left\{i \in \mathcal {I} _ {\text {g o o d}} \left(t _ {k}\right) \right\} \\ = \frac {G \left(\widehat {S} _ {i} \left(t _ {k}\right) , \widehat {A} _ {i} \left(t _ {k}\right) , s ^ {\prime}\right)}{2 N g _ {\max }} \mathbb {1} \left\{i \in \mathcal {I} _ {\text {g o o d}} \left(t _ {k}\right) \right\}, \tag {34} \\ \end{array}
$$
where the second equality uses the fact that the transition probability after uniformization is equal to the ratio between the transition rate and the uniformization rate; the last equality is by the definition of  $\mathcal{I}_{\mathrm{good}}(t_k)$ . For  $i \notin \mathcal{I}_{\mathrm{good}}(t_k)$  and  $s' \neq \widehat{S}_i(t_k)$ , we have  $\widehat{S}_i(t_{k+1}) = s'$  only when the exponential
clock ticks and the pair  $(i, s')$  is sampled, so
$$
\begin{array}{l} \mathbb {P} \left(\widehat {S} _ {i} \left(t _ {k + 1}\right) = s ^ {\prime} \mid \boldsymbol {S} \left(t _ {k}\right), \boldsymbol {A} \left(t _ {k}\right), \widehat {\boldsymbol {S}} \left(t _ {k}\right), \widehat {\boldsymbol {A}} \left(t _ {k}\right)\right) \mathbb {1} \left\{i \notin \mathcal {I} _ {\text {g o o d}} \left(t _ {k}\right) \right\} \\ = \mathbb {E} \left[ \frac {2 N g _ {\max } - g _ {k} ^ {\text {r e a l}}}{2 N g _ {\max }} \frac {G (\widehat {S} _ {i} (t _ {k}) , \widehat {A} _ {i} (t _ {k}) , s ^ {\prime})}{2 N g _ {\max } - g _ {k} ^ {\text {r e a l}}} \Bigg | \boldsymbol {S} (t _ {k}), \boldsymbol {A} (t _ {k}), \widehat {\boldsymbol {S}} (t _ {k}), \widehat {\boldsymbol {A}} (t _ {k}) \right] \mathbb {1} \{i \notin \mathcal {I} _ {\operatorname {g o o d}} (t _ {k}) \} \\ = \frac {G \left(\widehat {S} _ {i} \left(t _ {k}\right) , \widehat {A} _ {i} \left(t _ {k}\right) , s ^ {\prime}\right)}{2 N g _ {\max }} \mathbb {1} \left\{i \notin \mathcal {I} _ {\text {g o o d}} \left(t _ {k}\right) \right\}. \tag {35} \\ \end{array}
$$
Summing up (34)(35), we have that for any  $i \in [N]$  and  $s' \neq \widehat{S}_i(t_k)$ ,
$$
\mathbb {P} \left(\widehat {S} _ {i} \left(t _ {k + 1}\right) = s ^ {\prime} \mid \boldsymbol {S} \left(t _ {k}\right), \boldsymbol {A} \left(t _ {k}\right), \widehat {\boldsymbol {S}} \left(t _ {k}\right), \widehat {\boldsymbol {A}} \left(t _ {k}\right)\right) = \frac {G \left(\widehat {S} _ {i} \left(t _ {k}\right) , \widehat {A} _ {i} \left(t _ {k}\right) , s ^ {\prime}\right)}{2 N g _ {\max }}. \tag {36}
$$
Because  $\widehat{A}_i(t_k)$  is independently sampled from the distribution  $\bar{\pi} (\cdot |\widehat{S}_i(t_k))$ , taking expectation over  $(\pmb {A}(t_k),\widehat{\pmb{A}} (t_k))$  in the above equation, we get
$$
\mathbb {P} \left(\widehat {S} _ {i} \left(t _ {k + 1}\right) = s ^ {\prime} \mid \boldsymbol {S} \left(t _ {k}\right), \widehat {\boldsymbol {S}} \left(t _ {k}\right)\right) = \frac {\sum_ {a \in \mathbb {A}} G \left(\widehat {S} _ {i} \left(t _ {k}\right) , a , s ^ {\prime}\right) \bar {\pi} \left(a \mid \widehat {S} _ {i} \left(t _ {k}\right)\right)}{2 N g _ {\max }}, \tag {37}
$$
for any  $i \in [N]$  and  $s' \neq \widehat{S}_i(t_k)$ . Because the uniformization rate is  $2Ng_{\mathrm{max}}$ , the rate for  $\widehat{S}_i(t)$  to transition to  $s'$  is equal to  $\sum_{a \in \mathbb{A}} G(\widehat{S}_i(t), a, s') \bar{\pi}(a | \widehat{S}_i(t))$  conditioned on  $(S(t), \widehat{S}(t))$ . Observe that this transition rate only depends on  $\widehat{S}_i(t)$ , which implies that  $\widehat{S}_i(t)$ 's for  $i \in [N]$  are  $N$  i.i.d. Markov chains.
Finally, recall that we initialize the virtual states  $\widehat{S}_i(0)$ 's as  $N$  i.i.d. samples from the stationary distribution of the single-armed system under the policy  $\bar{\pi}$ . Since we have proved that  $\{\widehat{S}_i(t)\}_{t\geq 0}$ 's are  $N$  i.i.d. Markov chains induced by applying  $\bar{\pi}$  to the single-armed systems, for each  $t\geq 0$ ,  $\widehat{S}_i(t)$ 's remain stationary and i.i.d. Therefore, for each  $t\geq 0$ ,  $(\widehat{S}_i(t),\widehat{A}_i(t))$ ’s are i.i.d., and for each  $i$ , the distribution of  $(\widehat{S}_i(t),\widehat{A}_i(t))$  is equal to the steady-state state-action distribution  $(y(s,a))_{s\in \mathbb{S},a\in \mathbb{A}}$ .
F.4 Proof of Lemma 7
In this section, we prove the key intermediate result, Lemma 7. The second equation (33) in Lemma 7 is obvious from the definition of the policy. We can therefore focus on proving (32), i.e., bounding the long-run average number of arms whose real states are not equal to their virtual states. We call such arms bad arms, and the rest of the arms good arms.
We will use a similar approach as Section E: we invoke Little's Law to write the average number of bad arms as a product of the average disagreement rate and the average length of disagreement periods, based on a suitable definition of the disagreement events.
We define the disagreement event in a different way than in the discrete-time RB setting, because unlike in the discrete-time RB setting where  $A_{i}(t) \neq \widehat{A}_{i}(t)$  can immediately cause  $S_{i}(t) \neq \widehat{S}_{i}(t)$ , in the continuous-time RB setting, actions that last for only a short time may not have an effect on the states. Therefore, we say a disagreement event for arm  $i$  happens at time  $t$  only when arm  $i$  behaves differently than it would have behaved if  $A_{i}(t) = \widehat{A}_{i}(t)$ .
Specifically, the disagreement event of a good arm can be defined as having a state transition that turns it into a bad arm, since a good arm is supposed to remain good if we always have  $A_{i}(t) = \widehat{A}_{i}(t)$ ; for bad arms, how the arm "would have behaved" need to be defined with the help of an extra structure, namely, exponential timers, as described below.
Definition 3 (Exponential timers for simulating the leader-and-follower system). For each  $i \in [N]$ , we run a timer for each arm  $i$  that ticks every random amount of time. When the timer of arm  $i$  starts, it decides the time of its next tick based on the  $i$ -th arm  $(S_{i}(t), A_{i}(t), \widehat{S}_{i}(t), \widehat{A}_{i}(t))$ : if  $S_{i}(t) \neq \widehat{S}_{i}(t)$  and  $A_{i}(t) \neq \widehat{A}_{i}(t)$ , the timer ticks after a time that is exponentially distributed with rate  $G(S_{i}(t), \widehat{A}_{i}(t))$ ; otherwise, the timer pauses. The timer restarts when it ticks or when there is any event happening in the system.
The purpose of the exponential timer is to simulate the transition time of the leader-and-follower system, which characterizes how the arm would have behaved if  $A_{i}(t) = \widehat{A}_{i}(t)$ . Specifically, when arm  $i$  has  $S_{i}(t) \neq \widehat{S}_{i}(t)$  (so it is a bad arm) and  $A_{i}(t) \neq \widehat{A}_{i}(t)$ , we construct an imaginary leader-and-follower system whose state-action pairs are  $(S_{i}(t), \widehat{A}_{i}(t), \widehat{S}_{i}(t), \widehat{A}_{i}(t))$ . We let the transition times of the virtual state  $\widehat{S}_{i}(t)$  in the two systems be identical, and the transition times of the real state  $S_{i}(t)$  in the two systems be independent. Then the transition time of  $S_{i}(t)$  in the leader-and-follower system has an exponential distribution with rate  $G(S_{i}(t), \widehat{A}_{i}(t))$ , which is equal to the time that the exponential timer ticks.
Therefore, there are two ways that the transition of arm  $i$  with  $S_{i}(t) \neq \widehat{S}_{i}(t)$  and  $A_{i}(t) \neq \widehat{A}_{i}(t)$  can deviate from the leader-and-follower system described above: either arm  $i$  itself has a transition in the real state  $S_{i}(t)$ , or when the exponential timer ticks. This statement is actually also true if  $S_{i}(t) = \widehat{S}_{i}(t)$  and  $A_{i}(t) \neq \widehat{A}_{i}(t)$  because in that case, the exponential timer pauses.
We can thus formally define the disagreement event of arm  $i$  as below.
Definition 4 (Disagreement event for continuous-time RBs). For each  $i \in [N]$ , a disagreement event of arm  $i$  happens when its real state  $S_{i}(t)$  transitions or its exponential timer ticks, while  $A_{i}(t) \neq \widehat{A}_{i}(t)$ .
After defining the disagreement events, we can define the disagreement period of arm  $i$  as the period of time when  $S_{i}(t) \neq \widehat{S}_{i}(t)$ , separated by disagreement events, formally stated below.
Definition 5 (Disagreement period for continuous-time RBs). Given a sample path of the arm  $i$ 's real states and virtual states  $(S_{i}(t),\widehat{S}_{i}(t))$ , and an exponential timer, we define the disagreement period of the arm  $i$  as a time interval  $[t_{\mathrm{begin}},t_{\mathrm{end}})$  such that
There is a disagreement event at  $t_{\mathrm{begin}}$
There is no disagreement event during  $(t_{\mathrm{begin}}, t_{\mathrm{end}})$ , and  $S_i(t) \neq S_i(t)$  for  $t \in (t_{\mathrm{begin}}, t_{\mathrm{end}})$ ;
There is a disagreement event at  $t_{\mathrm{end}}$  or  $S_{i}(t_{\mathrm{end}}) = S_{i}(t_{\mathrm{end}})$ .
We let  $d(s, \widehat{s})$  be the instantaneous rate of disagreement events (instantaneous disagreement rate) when the system has real and virtual states  $(\mathbf{S}(t), \widehat{\mathbf{S}}(t)) = (s, \widehat{s})$ . Let  $D_{\mathrm{avg}}$  be the long-run average length of the disagreement periods.
Observe that the number of bad arms (the arms such that  $S_{i}(t) \neq \widehat{S}_{i}(t)$ ) is the number of arms in disagreement periods, so we can apply Little's Law to the number of bad arms.
Lemma 8 (Little's Law for disagreement periods, continuous-time version). It holds that
$$
\mathbb {E} \left[ \sum_ {i = 1} ^ {N} \mathbb {1} \left\{\widehat {S} _ {i} (\infty) \neq S _ {i} (\infty) \right\} \right] = \mathbb {E} \left[ d \left(\boldsymbol {S} (\infty), \widehat {\boldsymbol {S}} (\infty)\right) \right] \cdot D _ {\text {a v g}}, \tag {38}
$$
where  $d(\boldsymbol{s}, \widehat{\boldsymbol{s}})$  denotes the instantaneous disagreement rate when the virtual states are  $\widehat{S}(t) = \widehat{\boldsymbol{s}}$ ;  $D_{avg}$  denotes the long-run average length of the disagreement periods.
Lemma 9 (Average disagreement rates). The instantaneous disagreement rate is equal to
$$
d (\boldsymbol {s}, \widehat {\boldsymbol {s}}) \leq 2 g _ {\max } \mathbb {E} \left[ \left| \sum_ {i = 1} ^ {N} \widehat {A} _ {i} (t) - \alpha N \right| | \widehat {\boldsymbol {S}} (t) = \widehat {\boldsymbol {s}} \right]. \tag {39}
$$
Proof. For each arm  $i$  such that  $A_{i}(t) \neq \widehat{A}_{i}(t)$ , a disagreement event happens if its real state transitions, which happens at the rate  $G(S_{i}(t), A_{i}(t))$ , or if its exponential timer ticks, which happens at the rate 0 (if it is a good arm) or  $G(S_{i}(t), \widehat{A}_{i}(t))$  (if it is a bad arm). Therefore, the rate that disagreement events happen at arm  $i$  is no more than
$$
G \left(S _ {i} (t), A _ {i} (t)\right) + G \left(S _ {i} (t), \widehat {A} _ {i} (t)\right) \leq 2 g _ {\max }
$$
By the definition of the policy, there are in expectation  $\mathbb{E}\left[\left|\sum_{i=1}^{N} \widehat{A}_i(t) - \alpha N\right| \mid \widehat{\pmb{S}}(t) = \widehat{\pmb{s}}\right]$  arms with  $A_i(t) \neq \widehat{A}_i(t)$ , so the instantaneous disagreement rate of the system is as given in (39).
Lemma 10 (Average length of disagreement periods). It holds that
$$
D _ {a v g} \leq \bar {\tau} _ {\max } ^ {\text {s y n c}}. \tag {40}
$$
Proof. To bound  $D_{\mathrm{avg}}$ , it suffices to bound the expected length of a disagreement period with arbitrary initial states. Without loss of generality, consider a disagreement period on arm  $i$  that starts at time  $t_{\mathrm{begin}} = 0$ , with initial states  $(S_i(0), \widehat{S}_i(0)) = (s, \widehat{s})$ . During the disagreement period, there is no disagreement event, so as argued in the paragraph after Definition 3, the transitions of the  $i$ -th arm is identical to a leader-and-follower system. Therefore, we either have  $S_i(t) = \widehat{S}_i(t)$  after  $\tau^{\mathrm{sync}}(s, \widehat{s})$  amount of time, or have a disagreement event before that. In either case,  $t_{\mathrm{end}} \leq \tau^{\mathrm{sync}}(s, \widehat{s})$ . Therefore, the expected length of the disagreement period satisfies
$$
\mathbb {E} \left[ t _ {\text {e n d}} - t _ {\text {b e g i n}} \right] = \mathbb {E} \left[ t _ {\text {e n d}} \right] \leq \mathbb {E} \left[ \tau^ {\text {s y n c}} (s, \widehat {s}) \right] \leq \overline {{\tau}} _ {\max } ^ {\text {s y n c}}.
$$
This holds for arbitrary initial states  $(s,\widehat{s})$ , so  $D_{\mathrm{avg}}\leq \overline{\tau}_{\mathrm{max}}^{\mathrm{sync}}$
Proof of Lemma 7. Combining Lemma 8, 9, and 10, we have
$$
\begin{array}{l} \mathbb {E} \left[ \sum_ {i = 1} ^ {N} \mathbb {1} \left\{\widehat {S} _ {i} (\infty) \neq S _ {i} (\infty) \right\} \right] = \mathbb {E} \left[ d (\boldsymbol {S} (\infty), \widehat {\boldsymbol {S}} (\infty)) \right] \cdot D _ {\text {a v g}} \\ \leq 2 g _ {\max } \bar {\tau} _ {\max } ^ {\text {s y n c}} \mathbb {E} \left[ \left| \sum_ {i = 1} ^ {N} \widehat {A} _ {i} (\infty) - \alpha N \right| \right]. \\ \end{array}
$$
This proves (32).
Observe that by the definition of our algorithm, at any time  $t$
$$
\sum_ {i = 1} ^ {N} \mathbb {1} \left\{\widehat {A} _ {i} (t) \neq A _ {i} (t) \right\} = \left| \sum_ {i = 1} ^ {N} \widehat {A} _ {i} (t) - \alpha N \right| \quad a. s.
$$
Taking the steady-state expectation, we get (33).
G Experiment details and additional experiments