SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Conjecture D.8. Suppose  $\mathcal{A}(\bar{c}^{(1)},\dots,\bar{c}^{(N)};\bar{r})$  is a randomized  $\rho$ -replicable algorithm for the multiple coin problem and has a constant probability of success. Then, the sample complexity of  $\mathcal{A}$  is at least
$$
\Omega \left(\frac {N ^ {2} q (1 - q)}{\varepsilon^ {2} \rho^ {2}}\right).
$$
D.2.3 A Lower Bound for Replicable  $Q$ -Function Estimation
We now present the MDP construction that achieves the desired sample complexity lower bound. We define a family of MDPs  $\mathbb{M}$  as depicted in Figure D.1. This particular construction was first presented by Mannor and Tsitsiklis [2004] and generalized by Gheshlaghi Azar et al. [2013], Feng et al. [2019].
Any MDP  $M \in \mathbb{M}$  is parameterized by positive integers  $K_M, L_M$ , and some  $p_M^{(k,\ell)} \in [0,1]$  for  $k \in [K_M], \ell \in [L_M]$ . The state space of  $M$  is the disjoint union<sup>7</sup> of three sets  $\mathcal{S} = \mathcal{X} \sqcup \mathcal{Y} \sqcup \mathcal{Z}$ , where  $\mathcal{X}$  consists of  $K$  states  $\{x_1, \ldots, x_K\}$  and each of them has  $L$  available actions  $\{a_1, \ldots, a_L\} =: \mathcal{A}$ . All states in  $\mathcal{V}, \mathcal{Z}$  have a single action that the agent can take. Remark that each  $M \in \mathbb{M}$  has  $N = \sum_{s \in S} |\mathcal{A}^s| = 4K_M L_M$ .
For  $x \in \mathcal{X}$ , by taking action  $a \in \mathcal{A}$ , the agent transitions to a state  $y(x, a) \in \mathcal{Y}$  with probability 1. Let  $p_M(x_k, a_\ell) \coloneqq p_M^{(k,\ell)}$ . For state  $y(x, a) \in \mathcal{Y}$ , we transition back to  $y(x, a)$  with probability  $p_M(x, a)$  and to  $z(x, a) \in \mathcal{Z}$  with probability  $1 - p_M(x, a)$ . Finally, the agent always returns to  $z(x, a)$  for all  $z(x, a) \in \mathcal{Z}$ . The reward function  $r_M(s, a) = 1$  if  $s \in \mathcal{X} \cup \mathcal{Y}$  and is 0 otherwise. We remark that for every  $x \in \mathcal{X}, a \in \mathcal{A}$ , its  $Q^{\star}$  function can be computed in closed form by solving the Bellman optimality equation
$$
Q _ {M} ^ {\star} (x, a) = 1 + \gamma [ p _ {M} (x, a) \cdot Q _ {M} ^ {\star} (x, a) + (1 - p _ {M} (x, a)) \cdot 0 ] = \frac {1}{1 - \gamma p _ {M} (x , a)}.
$$
Figure D.1: The class of MDPs considered to prove the lower bound in Theorem D.9.
Recall we write  $N \coloneqq \sum_{s \in S} |\mathcal{A}^s|$ . to denote the total number of state-action pairs. Our main result in this section is the following.
Theorem D.9. Let  $\rho, \varepsilon \in (0,1)^2$ ,  $\gamma \in (1/2,1)$ , and  $\delta = 1/4$ . Suppose  $\mathcal{A}$  is a locally random  $\rho$ -replicable algorithm that returns an estimate  $\widehat{Q}$  for any MDP with discount factor  $\gamma$  such that  $|\widehat{Q}(s,a) - Q^{\star}(s,a)| \leq \varepsilon$  with probability at least  $1 - \delta/6$  for each  $s \in S, a \in \mathcal{A}^s$ . Then  $\mathcal{A}$  has a sample complexity of at least
$$
\Omega \left(\frac {N ^ {3}}{(1 - \gamma) ^ {3} \varepsilon^ {2} \rho^ {2}}\right).
$$
Remark D.10. If Conjecture D.8 holds, we obtain a sample complexity lower bound of
$$
\Omega \left(\frac {N ^ {2}}{(1 - \gamma) ^ {3} \varepsilon^ {2} \rho^ {2}}\right)
$$
for general randomized  $\rho$ -replicable algorithms for  $Q$  estimation.
7Denoted by  $\sqcup$
On a high level, we argue that a locally random  $\rho$ -replicable algorithm  $\mathcal{A}$  for estimating the  $Q$  function of arbitrary MDPs up to accuracy  $\varepsilon \approx \varepsilon_0 / (1 - \gamma)^2$  yields a locally random  $\rho$ -replicable algorithm for the multiple coin problem (cf. Problem D.5) with tolerance approximately  $\varepsilon_0 \approx (1 - \gamma)^2\varepsilon$  when we choose  $q \approx \gamma$  in Theorem D.6. We can then directly apply Theorem D.6 to conclude the proof. See Appendix D.4 for details.
D.2.4 A Lower Bound for Replicable Policy Estimation
Having established the lower bound for locally random replicable  $Q$ -function estimation, we now present our lower bound for deterministic replicable policy estimation. We argue that a deterministic  $\rho$ -replicable algorithm for optimal policy estimation yields a locally random  $\rho$ -replicable algorithm for optimal  $Q$ -function estimation after some post-processing that has sample complexity  $\tilde{o}\left(N^{3} / \varepsilon^{2}\rho^{2}(1 - \gamma)^{3}\right)$ . It follows that the sample complexity lower bound we derived for  $Q$ -function estimation holds for policy estimation as well.
In order to describe the post-processing step, we employ a locally random replicable rounding algorithm (cf. Theorem C.2) that is provided in Esfandiari et al. [2023b]. Intuitively, we show that estimating the value function  $V^{\pi}$  of  $\pi$  reduces to estimating the optimal  $Q$ -function of some single-action MDP. Given such an estimate  $\hat{V}^{\pi}$ , we can then estimate  $Q^{\pi}$  using the simple sample mean query given sufficient samples from the generative model. Lastly, the locally random replicable rounding subroutine from Theorem C.1 is used as a post-processing step.
We now state the formal lower bound regarding the sample complexity of deterministic replicable policy estimation. Its proof follows by combining the  $Q$ -function estimation lower bound and the reduction we described above. For the full proof, see Appendix D.5.
Theorem D.11. Let  $\varepsilon, \rho \in (0,1)^2$  and  $\delta = 1/4$ . Suppose  $\mathcal{A}$  is a deterministic  $\rho$ -replicable algorithm that outputs a randomized policy  $\pi$  such that  $|V^{\pi}(s) - V^{\star}(s)| \leq \varepsilon$  with probability at least  $1 - \delta/12$  for each  $s \in S$ . Then  $\mathcal{A}$  has a sample complexity of at least
Remark D.12. If Conjecture D.8 holds, we obtain a sample complexity lower bound of
for general randomized  $\rho$ -replicable algorithms for policy estimation.
D.3 The Coin Problem
As mentioned before, estimating the bias of a coin is an elementary statistical problem. In order to establish our lower bounds, we first aim to understand the sample complexity of locally random algorithms that replicably estimate the bias of multiple coins simultaneously.
Impagliazzo et al. [2022] explored the single coin version of the problem and established a version of the following result when the biases  $q, q - \varepsilon$  of the coins in question lie in the interval  $(1/4, 3/4)$ . We state and prove the more general result when we allow  $q$  to be arbitrarily close to 1.
Theorem D.13 (SQ-Replicability Lower Bound; [Impagliazzo et al., 2022]). Fix  $q, \varepsilon, \rho \in (0,1)^3$  such that  $q - \varepsilon \in (1/2,1)$  and  $\delta = 1/4$ . Let  $\mathcal{A}(\bar{c};\bar{r})$  be an algorithm for the (single) coin problem.
Suppose  $\mathcal{A}$  satisfies the following:
(i)  $\mathcal{A}$  outputs  $\{0,1\}$  where 1 indicates its guess that the bias of the coin which generated  $\overline{c}$  is  $q$
(ii)  $\mathcal{A}$  is  $\rho$ -replicable even when its samples are drawn from coins with bias  $p \in [q - \varepsilon, q]$  for all  $i \in [N]$ .
(iii) If  $p \in \{q - \varepsilon, q\}$ , then  $\mathcal{A}$  correctly guesses the bias of the  $i$ -th coin with probability at least  $1 - \delta / 6$ .
Then the sample complexity of  $\mathcal{A}$  is at least
$$
\Omega \left(\frac {q (1 - q)}{\varepsilon^ {2} \rho^ {2}}\right).
$$
We follow a similar proof process as Impagliazzo et al. [2022] towards a lower bound for multiple coins. In particular, we begin with the two following lemmas adapted from Impagliazzo et al. [2022].
Lemma D.14 ([Impagliazzo et al., 2022]). Let  $g: \{0,1\}^m \to \{0,1\}$  be a boolean function. Suppose the input bits are independently sampled from  $\mathrm{Be}(p)$  for some parameter  $p \in [0,1]$  and let  $\mathrm{Acc}: [0,1] \to [0,1]$  be given by
$$
\operatorname {A c c} (p) := \mathbb {P} _ {\bar {x} \sim_ {i. i. d.} \mathrm {B e} (p)} \{g (\bar {x}) = 1 \}.
$$
Then Acc is differentiable on  $(0,1)$  and for all  $p\in (0,1)$
$$
\mathrm {A c c} ^ {\prime} (p) \leq \sqrt {\frac {m}{p (1 - p)}}.
$$
Proof (Lemma D.14). Fix  $p \in (0, 1)$  and suppose  $\bar{x} \sim_{i.i.d.} \operatorname{Be}(p)$ . Define
$$
a _ {k} := \mathbb {P} \left\{g (\bar {x}) = 1 \left| \sum_ {i = 1} ^ {m} x _ {i} = k \right. \right\}.
$$
$$
\operatorname {A c c} (p) = \sum_ {k = 1} ^ {m} a _ {k} \binom {m} {k} p ^ {k} (1 - p) ^ {m - k}.
$$
In particular, Acc is differentiable.
By computation,
$$
\begin{array}{l} \operatorname {A c c} ^ {\prime} (p) = \sum_ {k = 1} ^ {m} a _ {k} \binom {m} {k} \left[ k p ^ {k - 1} (1 - p) ^ {m - k} - (m - k) p ^ {k} (1 - p) ^ {m - k - 1} \right] \\ = \sum_ {k} a _ {k} {\binom {m} {k}} p ^ {k} (1 - p) ^ {m - k} \left[ \frac {k}{p} - \frac {m - k}{1 - p} \right] \\ = \sum_ {k} a _ {k} \binom {m} {k} p ^ {k} (1 - p) ^ {m - k} \cdot \frac {k (1 - p) - (m - k) p}{p (1 - p)} \\ = \sum_ {k} a _ {k} \binom {m} {k} p ^ {k} (1 - p) ^ {m - k} \cdot \frac {k - m p}{p (1 - p)} \\ \leq \sum_ {k} \binom {m} {k} p ^ {k} (1 - p) ^ {m - k} \cdot \frac {| k - m p |}{p (1 - p)} \quad a _ {k} \in [ 0, 1 ] \\ = \frac {1}{p (1 - p)} \mathbb {E} [ | X - \mathbb {E} [ X ] | ] \quad X \sim \operatorname {B i n} (m, p) \\ \leq \frac {1}{p (1 - p)} \sqrt {\operatorname {V a r} [ X ]} \\ = \frac {1}{p (1 - p)} \sqrt {m p (1 - p)} \\ = \sqrt {\frac {m}{p (1 - p)}}. \\ \end{array}
$$
In the following arguments, we need to reason about probabilistic events with multiple sources of randomness. For the sake of clarity, we use the formal definitions of a probability space and random variable. Specifically, let  $(W,\mathcal{F},\mathbb{P})$  be an underlying probability space where  $W$  is some sample space,  $\mathcal{F}\subseteq 2^W$  is a  $\sigma$ -algebra, and  $\mathbb{P}$  is a probability measure. A random variable is simply a real-valued function  $W\to \mathbb{R}$  from the sample space.
Define  $C_p \coloneqq \operatorname{Be}(p)$ . Moreover, define  $C_- \coloneqq \operatorname{Be}(q - \varepsilon)$  and  $C_+ \coloneqq \operatorname{Be}(q)$  to be the distributions of the possible biased coins.
Lemma D.15 ([Impagliazzo et al., 2022]). Fix  $q, \varepsilon \in (0,1)$  such that  $q - \varepsilon \in (1/2,1)$  and  $\delta = 1/4$ . Suppose  $g: \{0,1\}^m \to \{0,1\}$  is a boolean function satisfying the following:
(i) For  $\bar{x}\sim_{i.i.d.}C_{-}$ ,  $\mathbb{P}\{w:g(\bar{x} (w)) = 0\} \geq 1 - \delta$
(ii) For  $\bar{x} \sim_{i.i.d.} C_+$ ,  $\mathbb{P}\{w : g(\bar{x}(w)) = 1\} \geq 1 - \delta$ .
Then for  $p\sim U[q - \varepsilon ,q]$  and  $\bar{x}^{(1)},\bar{x}^{(2)}\sim_{i.i.d.}C_p$
$$
\mathbb {P} \{w: g (\bar {x} ^ {(1)} (w)) \neq g (\bar {x} ^ {(2)} (w)) \} \geq \Omega \left(\frac {\sqrt {q (1 - q)}}{\varepsilon \sqrt {m}}\right).
$$
In other words, we require
$$
m \geq \Omega \left(\frac {q (1 - q)}{\varepsilon^ {2} \rho^ {2}}\right)
$$
if we wish to reduce the probability above to at most  $\rho$ .
We should interpret the input of the function  $g$  from Lemma D.15 as  $m$  realizations of coin flips from the same coin and the output of  $g$  as its guess whether the bias of the coin that generated the realizations is  $q$ . Lemma D.15 states that if the function  $g$  is "nice", i.e. is able to distinguish the two coins  $C_{-}, C_{+}$  with some fixed confidence  $1 - \delta$ , then the same function is not replicable with probability at least  $\Omega(\sqrt{q(1 - q)} / \varepsilon \sqrt{m})$  when the bias is chosen uniformly randomly in the interval  $[q - \varepsilon, q]$ . Let us view an arbitrary  $\rho$ -replicable algorithm  $\mathcal{A}(\bar{c}; \bar{r})$  as a distribution over functions  $g_{\bar{r}}(\bar{c})$ . Impagliazzo et al. [2022] argued that at least a constant fraction of  $g_{\bar{r}}$  is nice, leading to Theorem D.13. Unfortunately, this argument does not extend trivially to the case of multiple coins. However, we present an extension for the special case when  $\mathcal{A}$  is locally random.
Proof (Lemma D.15). Let  $g: \{0, 1\}^m \to \{0, 1\}$  be a boolean function that satisfies the condition of Lemma D.15. By Lemma D.14,
$$
\operatorname {A c c} (p) := \mathbb {P} _ {\bar {c} \sim_ {i. i. d.} C _ {p}} \{w: g (\bar {c} (w)) = 1 \}
$$
is differentiable (continuous) and for every  $p\in [q - \varepsilon ,q]$
$$
\operatorname {A c c} ^ {\prime} (p) \leq \sqrt {\frac {m}{q (1 - q)}}.
$$
This is because  $\frac{1}{p(1 - p)}$  is non-increasing on  $(1/2, 1)$ . In particular,  $\operatorname{Acc}(p)$  is  $\sqrt{\frac{m}{q(1 - q)}}$ -Lipschitz over the interval  $[q - \varepsilon, q]$ .
Now,  $\mathrm{Acc}(q - \varepsilon) \leq \delta < 1/4$  and  $\mathrm{Acc}(q) \geq 1 - \delta > 3/4$ , thus by the intermediate value theorem from elementary calculus, there is some  $q_0 \in (q - \varepsilon, q)$  such that  $\mathrm{Acc}(q_0) = 1/2$ . It follows by the Lipschitz condition and the mean value theorem that there is an interval  $I$  of length  $\Omega\left(\sqrt{q(1 - q)/m}\right)$  around  $q_0$  so that for all  $p \in I$ ,
$$
\operatorname {A c c} (p) \in \left( \begin{array}{c} 1 \\ \hline 3 \end{array} , \frac {2}{3}\right).
$$
For two independent sequences of i.i.d. samples, say  $\bar{x}^{(1)},\bar{x}^{(2)}$  , we know that  $g(\bar{x}^{(1)})$  and  $g(\bar{x}^{(2)})$  are independent. Thus for  $p\in I$  , there is a  $2\operatorname {Acc}(p)(1 - \operatorname {Acc}(p)) > 4 / 9$  probability that  $g(\bar{x}^{(1)})\neq g(\bar{x}^{(2)})$
$$
\begin{array}{l} \left. \mathbb {P} \left\{w: g (\bar {x} ^ {(1)} (w)) \neq g (\bar {x} ^ {(2)} (w)) \right\} \right\} \\ \geq \mathbb {P} \left\{w: g \left(\bar {x} ^ {(1)} (w)\right) \neq g \left(\bar {x} ^ {(2)} (w)\right) \mid p (w) \in I \right\} \cdot \mathbb {P} \left\{w: p (w) \in I \right\} \\ \geq \frac {4}{9} \cdot \Omega \left(\frac {\sqrt {q (1 - q) / m}}{2 \varepsilon}\right) \quad p \sim U [ q - \varepsilon , q ] \\ = \Omega \left(\frac {\sqrt {q (1 - q)}}{\varepsilon \sqrt {m}}\right). \\ \end{array}
$$
Lemma D.15 enables us to prove Theorem D.13.
Proof (Theorem D.13). Let  $G_{-}$  be the event that  $\mathcal{A}$  correctly guess the bias of the coin given  $p = q - \varepsilon$  and similarly for  $G_{+}$ . Thus
$$
G _ {-} := \{w: \mathscr {A} (\bar {c} (w); \bar {r} (w)) = 0 \} \quad p = q - \varepsilon
$$
$$
G _ {+} := \{w: \mathcal {A} (\bar {c} (w); \bar {r} (w)) = 1 \} \quad p = q.
$$
Here the randomness lies only in the samples  $\bar{c} \sim_{i.i.d.} C_p$  and the uniformly random  $\bar{r}$ . By (iii),
$$
\begin{array}{l} \frac {\delta}{3 \cdot 2} \geq \sum_ {\bar {r}} \mathbb {P} \left(G _ {-} ^ {c} \mid \bar {r}\right) \cdot \mathbb {P} (\bar {r}) \\ \geq \mathbb {E} _ {\bar {r}} \left[ \mathbb {P} \left(G _ {-} ^ {c} \mid \bar {r}\right) \right]. \\ \end{array}
$$
Thus Markov's inequality tells us that choosing  $\bar{r}$  uniformly at random satisfies
$$
\mathbb {P} \left(G _ {-} ^ {c} \mid \bar {r}\right) \leq \delta
$$
with probability at least  $1 - \frac{1}{(3\cdot 2)}$  and similarly for  $G_{+}$
By a union bound, choosing  $\bar{r}$  uniformly at random means  $G_{\pm}$  both occur with probability at least  $2/3$ . Let us write  $\bar{r} \in G := G_{+} \cap G_{-}$  as a shorthand for indicating the random draw  $\bar{r}$  satisfies both  $G_{\pm}$ .
Fix  $\bar{r}^{\star}\in G$  to be any particular realization and observe that  $g\coloneqq \mathcal{A}(\cdot ;\bar{r}^{\star})$  satisfies the condition of Lemma D.15. By Lemma D.15, if  $p\sim U[q - \varepsilon ,q]$  , and  $\bar{c}^{(1)},\bar{c}^{(2)}\sim_{i.i.d.}C_p,$
$$
\begin{array}{l} \mathbb {P} \left\{w: \mathscr {A} (\bar {c} ^ {(1)} (w); r (w)) \neq \mathscr {A} (\bar {c} ^ {(2)} (w); r (w)) \right\} \\ = \sum_ {\bar {r}} \mathbb {P} \left\{w: \mathcal {A} (\bar {c} ^ {(1)} (w); r (w)) \neq \mathcal {A} (\bar {c} ^ {(2)} (w); r (w)) \mid \bar {r} \right\} \cdot \mathbb {P} (\bar {r}) \\ \geq \sum_ {\bar {r} \in G} \mathbb {P} \left\{w: \mathcal {A} (\bar {c} ^ {(1)} (w); r (w)) \neq \mathcal {A} (\bar {c} ^ {(2)} (w); r (w)) \mid \bar {r} \right\} \cdot \mathbb {P} (\bar {r}) \\ \geq \frac {2}{3} \cdot \Omega \left(\frac {\sqrt {q (1 - q)}}{\varepsilon \sqrt {m}}\right) \\ = \Omega \left(\frac {\sqrt {q (1 - q)}}{\varepsilon \sqrt {m}}\right). \\ \end{array}
$$
We remark that in the derivation above, a crucial component is the fact that outputs of  $\mathcal{A}$  across two runs are conditionally independent given the internal randomness.
Since (ii) also holds when  $p$  is chosen uniformly at random in the interval  $[q - \varepsilon, q]$ , it follows that  $\Omega(\sqrt{q(1 - q)} / \varepsilon \sqrt{m}) \leq \rho$  is a lower bound for the replicability parameter. The total sample complexity is thus at least
We now use Theorem D.13 to prove the formal statement of Theorem D.6, a samples complexity lower bound for locally random algorithms that replicably distinguish the biases of  $N$  independent coins  $C^{(1)},\ldots ,C^{(N)}$ , assuming each of which is either  $C_-$  or  $C_+$ . The lower bound for locally random replicable  $Q$ -estimation follows. We formally state Theorem D.6 below.
Theorem D.16 (Theorem D.6; Formal). Fix  $q, \varepsilon, \rho \in (0,1)^3$  such that  $q - \varepsilon \in (1/2,1)$  and  $\delta = 1/4$ . Let  $\mathcal{A} = (\mathcal{A}^{(1)},\ldots,\mathcal{A}^{(N)})$  be a locally random algorithm for the multiple coin problem where  $\mathcal{A}^{(i)}(\bar{c}^{(1)},\ldots,\bar{c}^{(N)};\bar{r}^{(i)})$  is the output for the  $i$ -th coin that draws internal randomness independently from other coins.
(i)  $\mathcal{A}^{(i)}$  outputs  $\{0,1\}$  where  $I$  indicates its guess that the bias of the coin which generated  $\bar{c}^{(i)}$  is  $q$ .
(ii)  $\mathcal{A}$  is  $\rho$ -replicable even when its samples are drawn from coins where  $p^{(i)} \in [q - \varepsilon, q]$  for all  $i \in [N]$ .
(iii) If  $p^{(i)} \in \{q - \varepsilon, q\}$ , then  $\mathcal{A}$  correctly guesses the bias of the  $i$ -th coin with probability at least  $1 - \delta / 6$ .
We remark that the scaling with respect to  $q$  is important in order to establish a strong lower bound with respect to the parameter  $(1 - \gamma)^{-1}$ .
Proof (Theorem D.16). First, we remark that the  $i$ -th output of  $\mathcal{A}$  across two runs is independent of other outputs since both the coin flips and internal randomness are drawn independently per coin. Thus we may as well assume that  $\mathcal{A}^{(i)}$  only reads the coin flips from the  $i$ -th coin.
Let  $\rho_{i}, i \in [N]$  be the probability that the output of  $\mathcal{A}^{(i)}$  is inconsistent across two executions when the bias  $p^{(i)}$  is chosen uniformly in  $[q - \varepsilon, q]$ . We claim that there are at least  $N/2$  indices  $i \in [N]$  such that  $\rho_{i} \leq \rho/N$ . Indeed, by the independence due to the local randomness property, we have
$$
\begin{array}{l} \rho \geq 1 - \prod_ {i \in [ N ]} \left(1 - \rho_ {i}\right) \\ \geq 1 - \exp \left(- \sum_ {i} \rho_ {i}\right) \quad 1 + x \leq e ^ {x} \\ \geq \sum_ {i} \frac {\rho_ {i}}{2}. \quad e ^ {- x} \leq 1 - \frac {x}{2}, x \in [ 0, 1 ] \\ \end{array}
$$
Suppose towards a contradiction that more than  $N / 2$  indices  $i \in [N]$  satisfy  $\rho_i > \rho / N$ . But then
$$
\sum_ {i} \frac {\rho_ {i}}{2} > \frac {N}{2} \cdot 2 \frac {\rho}{N} = \rho ,
$$
Let  $I \subseteq [N]$  be the indices of the coins such that  $\rho_i \leq \rho / N$ . For each  $i \in I$ ,  $\mathcal{A}^{(i)}$  satisfies the conditions for Theorem D.13. Thus  $\mathcal{A}^{(i)}$  has sample complexity at least
$$
\Omega \left(\frac {q (1 - q)}{\varepsilon^ {2} (\rho / N) ^ {2}}\right).
$$
It follows that the total sample complexity is at least
$$
\Omega \left(\frac {N}{2} \cdot \frac {q (1 - q)}{\varepsilon^ {2} (\rho / N) ^ {2}}\right) = \Omega \left(\frac {N ^ {3} q (1 - q)}{\varepsilon^ {2} \rho^ {2}}\right).
$$
D.4 Replicable  $Q$ -Function Estimation Lower Bound
In this section, we restate and prove Theorem D.9, a lower bound on locally random replicable  $Q$ -function estimation.
Proof (Theorem D.9). Choose  $q \coloneqq 4\gamma - 1/3\gamma$  and  $\varepsilon_0 \in (0, (1 - \gamma)/\gamma)$  such that  $q - \varepsilon_0 \in (1/2, 1)$ . Let  $C^{(i)} \sim \mathrm{Be}(p^{(i)})$  be Bernoulli random variables (biased coins) for  $i \in [N]$  where  $p^{(i)} \in [q - \varepsilon_0, q]$ .
To see the reduction, first choose any  $K,L\in \mathbb{Z}_{+}$  such that  $KL = N$  and let  $M\in \mathbb{M}$  be such that the state action space has cardinality  $4N$  as in the construction in Figure D.1. We identity each  $i\in [N]$  with a pair  $(x,a)\in \mathcal{X}\times \mathcal{A}$  and write  $i_{x,a}$  to be the index corresponding to the pair  $(x,a)$ , i.e.  $p(x,a) = p^{(i_x,a)}$ . For each state  $y(x,a)\in \mathcal{V}$ , we flip the coin  $C^{(i_x,a)}\sim \mathrm{Be}(p(x,a))$  and transition back to  $y(x,a)$  if the coin lands on 1 and to  $z(x,a)\in \mathcal{Z}$  if it lands on 0. By construction,
$$
Q _ {M} ^ {\star} (x, a) = \frac {1}{1 - \gamma p _ {M} (x , a)}.
$$
Let us compare the absolute difference of  $Q^{\star}(x, a)$  when  $p(x, a) = q$  versus when  $p(x, a) = q - \varepsilon_0$ . By computation,
$$
\frac {1}{1 - \gamma q} - \frac {1}{1 - \gamma (q - \varepsilon_ {0})} = \frac {\gamma \varepsilon_ {0}}{[ 1 - \gamma (q - \varepsilon_ {0}) ] [ 1 - \gamma q ]}.
$$
Then the following inequalities hold.
$$
\begin{array}{l} q \geq \frac {2}{3} \\ 1 - q = \frac {1 - \gamma}{3 \gamma} \\ \geq \frac {2}{3} (1 - \gamma) \quad \gamma > \frac {1}{2} \\ 1 - \gamma q = \frac {4}{3} (1 - \gamma) \\ \gamma (q - \varepsilon_ {0}) = \frac {4}{3} (1 - \gamma) + \gamma \varepsilon_ {0} \\ \leq \frac {4}{3} (1 - \gamma) + (1 - \gamma) \quad \varepsilon_ {0} <   \frac {1 - \gamma}{\gamma} \\ = \frac {7}{3} (1 - \gamma). \\ \end{array}
$$
$$
\begin{array}{l} \left| \frac {1}{1 - \gamma q} - \frac {1}{1 - \gamma (q - \varepsilon_ {0})} \right| \geq \frac {9 \gamma \varepsilon_ {0}}{2 8 (1 - \gamma) ^ {2}} \\ \geq \frac {3 \varepsilon_ {0}}{1 4 (1 - \gamma) ^ {2}} \quad \gamma \geq \frac {2}{3} \\ =: \varepsilon . \\ \end{array}
$$
Suppose we run a locally random algorithm that replicably estimates  $Q_{M}^{\star}(x, a)$  to an accuracy of  $\varepsilon / 3$ . Then we are able to distinguish whether the biases of the coins  $C^{(i_{x,a})}$  is either  $q - \varepsilon_0$  or  $q$ . In addition, the internal randomness consumed in the estimation of  $C^{(i_{x,a})}$  comes only from the internal randomness used to estimate  $Q^{\star}(x, a)$ . Hence the scheme we described is a locally random algorithm for the multiple coin problem. Finally, the scheme is replicable even when the biases are chosen in the interval  $[q - \varepsilon_0, q]$ .
By Theorem D.16, the following sample complexity lower bound holds for  $\mathcal{A}$ :
$$
\Omega \left(\frac {N ^ {3} q (1 - q)}{\varepsilon_ {0} ^ {2} \rho^ {2}}\right) = \Omega \left(\frac {N ^ {3} (1 - \gamma)}{[ (1 - \gamma) ^ {2} \varepsilon ] ^ {2} \rho^ {2}}\right) = \Omega \left(\frac {N ^ {3}}{(1 - \gamma) ^ {3} \varepsilon^ {2} \rho^ {2}}\right).
$$
D.5 Replicable Policy Estimation Lower Bound
In order to prove a lower bound on locally random replicable policy estimation, we first describe a locally random replicable algorithm that estimates the state-action function  $Q^{\pi}$  for a given (possibly randomized) policy  $\pi$ . Recall  $N := \sum_{s \in S} |\mathcal{A}^s|$  denotes the number of state-action pairs.
Lemma D.17. Let  $\varepsilon, \rho \in (0, 1)$  and  $\delta \in (0, \rho/3)$ . Suppose  $\pi$  is an explicitly given randomized policy. There is a locally random  $\rho$ -replicable algorithm that outputs an estimate  $\widehat{Q}^{\pi}$  of  $Q^{\pi}$  such that with probability at least  $1 - \delta$ ,  $\| \widehat{Q}^{\pi} - Q^{\pi} \|_{\infty} \leq \varepsilon$ . Moreover, the algorithm has sample complexity
$$
\tilde {O} \left(\left(\frac {| \mathcal {S} | N ^ {2}}{(1 - \gamma) ^ {3} \varepsilon^ {2} \rho^ {2}} + \frac {N ^ {3}}{(1 - \gamma) ^ {2} \varepsilon^ {2} \rho^ {2}}\right) \log \frac {1}{\delta}\right) = \tilde {o} \left(\frac {N ^ {3}}{(1 - \gamma) ^ {3} \varepsilon^ {2} \rho^ {2}} \log \frac {1}{\delta}\right).
$$