SlowGuess/NeurIPS_2023_OCR huggingface dataset:
where  $r_{\mathrm{max}} \triangleq \max_{s \in \mathbb{S}, a \in \mathbb{A}} |r(s, a)|$  and  $\overline{\tau}_{\mathrm{max}}^{\mathrm{sync}} \triangleq \max_{k \in [K], (s, a, \widehat{s}, \widehat{a}) \in \mathbb{S} \times \mathbb{A} \times \mathbb{S} \times \mathbb{A}} \mathbb{E}\left[\tau_k^{\mathrm{sync}}(s, a, \widehat{s}, \widehat{a})\right]$ .
H.2 Proof for Theorem 3
In this section, we prove Theorem 3.
Since  $V_{1}^{\mathrm{rel}}\geq V_{N}^{*}$  , it suffices to show that
$$
V _ {1} ^ {\text {r e l}} - V _ {N} ^ {\mathrm {F T V A} \left(\left\{\bar {\pi} _ {k} ^ {*} \right\} _ {k \in [ K ]}\right)} \leq \frac {r _ {\max } \bar {\tau} _ {\max } ^ {\text {s y n c}}}{\sqrt {N}}. \tag {46}
$$
We start with an inequality that has the same form as (13) in the homogeneous case:
$$
V _ {1} ^ {\mathrm {r e l}} - V _ {N} ^ {\mathrm {F T V A} (\{\bar {\pi} _ {k} ^ {*} \} _ {k \in [ K ]})} = \frac {1}{N} \mathbb {E} \left[ \sum_ {i = 1} ^ {N} r \big (\widehat {S} _ {i} (\infty), \widehat {A} _ {i} (\infty) \big) - \sum_ {i = 1} ^ {N} r \big (S _ {i} (\infty), A _ {i} (\infty) \big) \right]
$$
$$
\leq \frac {2 r _ {\max }}{N} \mathbb {E} \left[ \sum_ {i = 1} ^ {N} \mathbb {1} \left\{\left(\widehat {S} _ {i} (\infty), \widehat {A} _ {i} (\infty)\right) \neq \left(S _ {i} (\infty), A _ {i} (\infty)\right) \right\} \right], \tag {47}
$$
where in the first equality, we used the fact that the virtual processes are independently running the single-armed policy of the corresponding types, so they achieve the average reward  $V_{1}^{\mathrm{rel}}$ .
Defining the disagreement events, disagreement periods, and disagreement rates in the same way as Appendix E and applying Little's law, we have
$$
\mathbb {E} \left[ \sum_ {i = 1} ^ {N} \mathbb {1} \left\{\left(\widehat {S} _ {i} (\infty), \widehat {A} _ {i} (\infty)\right) \neq \left(S _ {i} (\infty), A _ {i} (\infty)\right) \right\} \right] = \mathbb {E} \left[ d \left(\widehat {\boldsymbol {S}} (\infty)\right) \right] \cdot D _ {\text {a v g}}, \tag {48}
$$
where  $d(\widehat{s})$  is the instantaneous disagreement rate, and  $D_{\mathrm{avg}}$  is the long-run average length of disagreement periods. With (47) and (48), it remains to bound  $\mathbb{E}\left[d(\widehat{S} (\infty))\right]$  and  $D_{\mathrm{avg}}$ .
By the definition of the policy, we have  $\mathbb{E}\left[d(\widehat{\pmb{S}} (\infty))\right] = \mathbb{E}\left[\left|\sum_{i = 1}^{N}\widehat{A}_{i}(\infty) - \alpha N\right|\right]$ ,  $\widehat{A}_i(\infty)$ 's are independent Bernoulli random variables, and  $\mathbb{E}\left[\widehat{A}_i(\infty)\right] = y_k^* (s,1)$  if arm  $i$  is of type  $k$ . As a result,
$$
\mathbb {E} \left[ \sum_ {i = 1} ^ {N} \widehat {A} _ {i} (\infty) \right] = \sum_ {k} \beta_ {k} N \sum_ {s, a} y _ {k} ^ {*} (s, 1) = \alpha N.
$$
Using the same Cauchy-Schwartz argument as in the proof of the homogeneous case in Appendix E, it is not hard to show that  $\mathbb{E}\left[d(\widehat{S} (\infty))\right]\leq \sqrt{N} /2$
As for  $D_{\mathrm{avg}}$ , it is not hard to see from the definition that the length of each disagreement period of a type  $k$  arm is stochastically dominated by  $\tau_k^{\mathrm{sync}}(s, a, \widehat{s}, \widehat{a})$ , where  $(s, a, \widehat{s}, \widehat{a})$  are the initial states and actions of that disagreement period. Therefore, the average length of disagreement period is bounded by
$$
D_{\mathrm{avg}}\leq \max_{k\in [K],(s,a,\widehat{s},\widehat{a})\in \mathbb{S}\times \mathbb{A}\times \mathbb{S}\times \mathbb{A}}\mathbb{E}\left[\tau_{k}^{\mathrm{sync}}(s,a,\widehat{s},\widehat{a})\right] = \overline{\tau}_{\mathrm{max}}^{\mathrm{sync}}.
$$
Combining the above calculations, we get (46), which finishes the proof.