SlowGuess/NeurIPS_2023_OCR huggingface dataset:
In this work, we establish sample complexity bounds for a several notions of replicability in the context of RL. We give an extensive comparison of the guarantees under these different notions in Appendix G. We believe that our work can open several directions for future research. One immediate next step would be to verify our lower bound conjecture for replicable estimation of multiple independent coins (cf. Conjecture D.8). Moreover, it would be very interesting to extend our results to different RL settings, e.g. offline RL with linear MDPs, offline RL with finite horizon, and online RL.
Amin Karbasi acknowledges funding in direct support of this work from NSF (IIS-1845032), ONR (N00014-19-1-2406), and the AI Institute for Learning-Enabled Optimization at Scale (TILOS). Lin Yang is supported in part by NSF Award 2221871 and an Amazon Faculty Award. Grigoris Velegkas is supported by TILOS, the Onassis Foundation, and the Bodossaki Foundation. Felix Zhou is supported by TILOS. The authors would also like to thank Yuval Dagan for an insightful discussion regarding the Gaussian mechanism.
M Mehdi Afsar, Trafford Crump, and Behrouz Far. Reinforcement learning based recommender systems: A survey. ACM Computing Surveys, 55(7):1-38, 2022.
Alekh Agarwal, Sham Kakade, and Lin F Yang. Model-based reinforcement learning with a generative model is minimax optimal. In Conference on Learning Theory, pages 67-83. PMLR, 2020.
Omer Angel and Yinon Spinka. Pairwise optimal coupling of multiple random variables. arXiv preprint arXiv:1903.00632, 2019.
Monya Baker. 1,500 scientists lift the lid on reproducibility. Nature, 533(7604), 2016.
Mohammad Bavarian, Badih Ghazi, Elad Haramaty, Pritish Kamath, Ronald L Rivest, and Madhu Sudan. Optimality of correlated sampling strategies. arXiv preprint arXiv:1612.01041, 2016.
Dimitri P Bertsekas. Dynamic programming and stochastic control. Academic Press, 1976.
Andrei Z Broder. On the resemblance and containment of documents. In Proceedings. Compression and Complexity of SEQUENCES 1997 (Cat. No. 97TB100171), pages 21-29. IEEE, 1997.
Mark Bun, Marco Gaboardi, Max Hopkins, Russell Impagliazzo, Rex Lei, Toniann Pitassi, Jessica Sorrell, and Satchit Sivakumar. Stability is stable: Connections between replicability, privacy, and adaptive generalization. arXiv preprint arXiv:2303.12921, 2023.
Moses S Charikar. Similarity estimation techniques from rounding algorithms. In Proceedings of the thirty-fourth annual ACM symposium on Theory of computing, pages 380-388, 2002.
Zachary Chase, Shay Moran, and Amir Yehudayoff. Replicability and stability in learning. arXiv preprint arXiv:2304.03757, 2023.
Peter Dixon, A Pavan, Jason Vander Woude, and NV Vinodchandran. List and certificate complexities in replicable learning. arXiv preprint arXiv:2304.02240, 2023.
Cynthia Dwork, Aaron Roth, et al. The algorithmic foundations of differential privacy. Foundations and Trends® in Theoretical Computer Science, 9(3-4):211-407, 2014.
Eric Eaton, Marcel Hussing, Michael Kearns, and Jessica Sorrell. Replicable reinforcement learning. arXiv preprint arXiv:2305.15284, 2023.
Alessandro Epasto, Mohammad Mahdian, Vahab Mirrokni, and Emmanouil Zampetakis. Optimal approximation-smoothness tradeoffs for soft-max functions. Advances in Neural Information Processing Systems, 33:2651-2660, 2020.
Hossein Esfandiari, Alkis Kalavasis, Amin Karbasi, Andreas Krause, Vahab Mirrokni, and Grigoris Velegkas. Replicable bandits. In The Eleventh International Conference on Learning Representations, 2023a.
Hossein Esfandiari, Amin Karbasi, Vahab Mirrokni, Grigoris Velegkas, and Felix Zhou. Replicable clustering. arXiv preprint arXiv:2302.10359, 2023b.
Eyal Even-Dar, Shie Mannor, and Yishay Mansour. Pac bounds for multi-armed bandit and markov decision processes. In  $COLT$ , volume 2, pages 255-270. Springer, 2002.
A Omitted Algorithms
Algorithm A.1 TV Indistinguishable Oracle for Multiple Query Estimation
1:  $\widehat{\mu} = (\widehat{\mu_1},\dots ,\widehat{\mu_d})\gets$  StatisticalQueryOracles  $\left(\frac{\varepsilon\rho}{2\sqrt{8d\cdot\log(4d / \delta)}},\frac{\delta}{2}\right)$  2: Sample  $\widehat{v}\sim \mathcal{N}(\widehat{\mu},\varepsilon^{2} / (8\cdot \log (4d / \delta))\cdot I_{d})$  3: Output  $\widehat{v}$    
Algorithm A.2 Sampling from Pairwise Optimal Coupling; [Angel and Spinka, 2019]   
1: Input: collection of random vectors  $S = \{X\}$  absolutely continuous with respect to a  $\sigma$  -finite measure  $\mu$  , with densities  $f_{X}:\mathbb{R}^{d}\to \mathbb{R}$  , some  $X\in S$    
2: Let  $\mathcal{R}$  denote the Poisson point process over  $\mathbb{R}^d\times \mathbb{R}_+ \times \mathbb{R}_+$  with intensity  $\mu \times$  Leb  $\times$  Leb.   
3: Sample  $r\coloneqq \{(x_i,y_i,t_i):i\in \mathbb{N}\} \sim \mathcal{R}$    
4: Let  $i^{\star}\gets \mathrm{argmin}_{i\in \mathbb{N}}\{t_i:f_S(x_i) > y_i\}$    
5: Output  $x_{i^{\star}}$  as a sample for  $X$
B Omitted Definitions
In this section, we discuss some dissimilarity measures for probability distributions that we use in this work.
Definition B.1 (Total Variation (TV) Distance). Let  $\Omega$  be a countable domain and  $P, Q$  be probability distributions over  $\Omega$ . The total variation distance between  $P, Q$ , denoted by  $d_{\mathrm{TV}}(P, Q)$ , is defined as
$$
\begin{array}{l} d_{\mathrm{TV}}(P,Q) = \sup_{A\in 2^{\Omega}}P(A) - Q(A) \\ = | | P - Q | | _ {1} \\ = \inf  _ {(X, Y) \sim \Pi (P, Q)} \mathbb {P} \{X \neq Y \}, \\ \end{array}
$$
where  $\Pi(P, Q)$  is a coupling between  $P, Q$ .
In the above definition, a coupling is defined to be a joint probability distribution  $(X,Y)\sim \Pi (P,Q)$  over the product space whose marginals distributions are  $P,Q$ , i.e., if we only view the individual components, then  $X\sim P,Y\sim Q$
Definition B.2 (Renyi Divergence). Let  $\Omega$  be a countable domain and  $P, Q$  be probability distributions over  $\Omega$ . For any  $\alpha > 1$  the Renyi divergence of order  $\alpha$  between  $P, Q$ , denoted by  $D_{\alpha}(P\|Q)$  is defined as
$$
D _ {a} (P \| Q) = \frac {1}{\alpha - 1} \log \left(\sum_ {\omega \in \Omega} \frac {P (\omega) ^ {\alpha}}{Q (\omega) ^ {\alpha - 1}}\right).
$$
We note that the quantity above is undefined when  $\alpha = 1$ . However, we can take its limit and define  $D_{1}(P||Q) = \sum_{\omega \in \Omega} P(\omega) \log \frac{P(\omega)}{Q(\omega)}$ , which recovers the definition of KL-divergence. Similarly, we can define  $D_{\infty}(P||Q) = \max_{\omega \in \Omega} \frac{P(\omega)}{Q(\omega)}$ .
Another important definition is that of Lipschitz continuity.
Definition B.3 (Lipschitz Continuity). Let  $\mathcal{X},\mathcal{Y}$  be two domains,  $f:\mathcal{X}\to \mathcal{Y}$  be some function, and  $d_{1}:\mathcal{X}\times \mathcal{X}\rightarrow \mathbb{R}_{\geq 0}$ ,  $d_{2}:\mathcal{Y}\times \mathcal{Y}\rightarrow \mathbb{R}_{\geq 0}$  be distance measures over  $\mathcal{X},\mathcal{Y}$ , respectively. We say that  $f$  is  $L$ -Lipschitz continuous with respect to  $d_{1},d_{2}$  if  $\forall x_1,x_2\in \mathcal{X}$  it holds that  $d_{2}(f(x_{1}),f(x_{2}))\leq L\cdot d_{1}(x_{1},x_{2})$ .
B.1 Local Randomness
Our algorithms in Section 3 satisfy a property which we call locally random. This roughly means that for every decision an algorithm makes based on external and internal randomness, the internal randomness is used once and discarded immediately after.
Definition B.4 (Locally Random). Let  $\mathcal{A} = (\mathcal{A}^{(1)},\dots,\mathcal{A}^{(N)}):\mathcal{I}^n\to \mathbb{R}^N$  be an  $n$ -sample randomized algorithm that takes as input elements from some domain  $\mathcal{I}$  and maps them to  $\mathbb{R}^N$ . We say that  $\mathcal{A}$  is locally random if:
(i) The  $i$ -th output component  $\mathcal{A}^{(i)}(\bar{S};\bar{r}^{(i)})$  is a function of all samples  $\bar{S}$  but only its own internal random string  $\bar{r}^{(i)}$ .
(ii) The sources  $\bar{r}^{(i)}$  of internal randomness are independent of each other and the external samples  $\bar{S}$ .
We will see that by restricting ourselves to locally random algorithms, it is necessary and sufficient to incur a sample cost of  $\tilde{\Theta}(N^3)$  for replicable  $Q$ -estimation. However, by relaxing this restriction and allowing for internal randomness that is correlated, we can achieve  $\tilde{O}(N^2)$  sample complexity.
C Omitted Preliminaries
C.1 Replicability
Impagliazzo et al. [2022] introduced the definition of replicability and demonstrated the following basic replicable operation.
Theorem C.1 (Replicable SQ-Oracle; [Impagliazzo et al., 2022]). Let  $\varepsilon, \rho \in (0,1)$  and  $\delta \in (0,\rho /3)$ . Suppose  $\phi$  is a sample mean statistical query with co-domain  $[0,1]$ . There is a locally random  $\rho$ -replicable  $SQ$  oracle to estimate its true value with tolerance  $\varepsilon$  and failure rate  $\delta$ . Moreover, the oracle has sample complexity
$$
\tilde {O} \left(\frac {1}{\varepsilon^ {2} \rho^ {2}} \log \frac {1}{\delta}\right).
$$
Esfandiari et al. [2023b] generalized the result above to multiple general queries with unbounded co-domain, assuming some regularity conditions on the queries.
Theorem C.2 (Replicable Rounding; [Impagliazzo et al., 2022, Esfandiari et al., 2023b]). Let  $\varepsilon, \rho \in (0,1)$  and  $\delta \in (0,\rho /3)$ . Suppose we have a finite class of statistical queries  $g_{1},\ldots,g_{N}$  with true values  $G_{1},\ldots,G_{N}$  and sampling  $n$  independent points from  $\mathcal{D}$  ensures that
$$
\sum_ {j = 1} ^ {N} \left| g _ {j} \left(x _ {1}, \dots , x _ {n}\right) - G _ {j} \right| \leq \varepsilon^ {\prime} := \frac {\varepsilon (\rho - 2 \delta)}{\rho + 1 - 2 \delta}
$$
There is a locally random  $\rho$ -replicable algorithm that outputs estimates  $\widehat{G}_j$  such that
$$
\left| \widehat {G} _ {j} - G _ {j} \right| \leq \varepsilon
$$
with probability at least  $1 - \delta$  for every  $j\in [N]$ . Moreover, it requires at most  $n$  samples.
C.2 Coupling and Correlated Sampling
Our exposition in this section follows Kalavasis et al. [2023]. Coupling is a fundamental notion in probability theory with many applications [Levin and Peres, 2017]. The correlated sampling problem, which has applications in various domains, e.g., in sketching and approximation algorithms [Broder, 1997, Charikar, 2002], is described in Bavarian et al. [2016] as follows: Alice and Bob are assigned probability distributions  $P_A$  and  $P_B$ , respectively, over a finite set  $W$ . Without any communication, using only shared randomness as the means to coordinate, Alice is required to output an element  $x$  distributed according to  $P_A$  and Bob is required to output an element  $y$  distributed according to  $P_B$ . Their goal is to minimize the disagreement probability  $\mathbb{P}\{x \neq y\}$ , which is comparable with  $d_{\mathrm{TV}}(P_A, P_B)$ . Formally, a correlated sampling strategy for a finite set  $W$  with error  $\varepsilon : [0,1] \to [0,1]$  is specified by a probability space  $\mathcal{R}$  and a pair of functions  $f, g : \Delta_W \times \mathcal{R} \to W$ , which are measurable in their second argument, such that for any pair  $P_A, P_B \in \Delta_W$  with  $d_{\mathrm{TV}}(P_A, P_B) \leq \delta$ , it holds that (i) the push-forward measure  $f(P_A, \cdot)$  (resp.  $g(P_B, \cdot)$ ) is  $P_A$  (resp.  $P_B$ ) and (ii)  $\mathbb{P}_{r \sim \mathcal{R}}\{f(P_A, r) \neq g(P_B, r)\} \leq \varepsilon(\delta)$ . We underline that a correlated
sampling strategy is not the same as a coupling, in the sense that the latter requires a single function  $h: \Delta_W \times \Delta_W \to \Delta_{W \times W}$  such that for any  $P_A, P_B$ , the marginals of  $h(P_A, P_B)$  are  $P_A$  and  $P_B$  respectively.
It is known that for any coupling function  $h$ , it holds that  $\mathbb{P}_{(x,y)\sim h(P_A,P_B)}\{x\neq y\} \geq d_{\mathrm{TV}}(P_A,P_B)$  and that this bound is attainable. Since  $(f(P_A,\cdot),g(P_B,\cdot))$  induces a coupling, it holds that  $\varepsilon (\delta)\geq \delta$  and, perhaps surprisingly, there exists a strategy with  $\varepsilon (\delta)\leq \frac{2\delta}{1 + \delta}$  [Broder, 1997, Kleinberg and Tardos, 2002, Holenstein, 2007] and this result is tight [Bavarian et al., 2016]. A second difference between coupling and correlated sampling has to do with the size of  $W$ : while correlated sampling strategies can be extended to infinite spaces  $W$ , it remains open whether there exists a correlated sampling strategy for general measure spaces  $(W,\mathcal{F},\mu)$  with any non-trivial error bound [Bavarian et al., 2016]. On the other hand, coupling applies to spaces  $W$  of any size. For further comparisons between coupling and the correlated sampling problem of Bavarian et al. [2016], we refer to the discussion in Angel and Spinka [2019] after Corollary 4.
Definition C.3 (Coupling). A coupling of two probability distributions  $P_A$  and  $P_B$  is a pair of random variables  $(X, Y)$ , defined on the same probability space, such that the marginal distribution of  $X$  is  $P_A$  and the marginal distribution of  $Y$  is  $P_B$ .
A very useful tool for our derivations is a coupling protocol that can be found in Angel and Spinka [2019].
Theorem C.4 (Pairwise Optimal Coupling [Angel and Spinka, 2019]). Let  $S$  be any collection of random variables that are absolutely continuous with respect to a  $\sigma$ -finite measure  $\mu$ . Then, there exists a coupling of the variables in  $S$  such that for any  $X, Y \in S$ ,
$$
\mathbb {P} \{X \neq Y \} \leq \frac {2 d _ {\mathrm {T V}} (X , Y)}{1 + d _ {\mathrm {T V}} (X , Y)}.
$$
Moreover, this coupling requires sample access to a Poisson point process with intensity  $\mu \times \mathrm{Leb} \times \mathrm{Leb}$ , where  $\mathrm{Leb}$  is the Lebesgue measure over  $\mathbb{R}_+$ , and full access to the densities of all the random variables in  $S$  with respect to  $\mu$ . Finally, we can sample from this coupling using Algorithm A.2.
D Omitted Details from Section 3
Here we fill in the details from Section 3, which describes upper and lower bounds for replicable  $Q$ -function and policy estimation. In Appendix D.1, we provide the rigorous analysis of an algorithm for locally random replicable  $Q$ -function estimation as well as replicable policy estimation. Appendix D.3 describes the local random replicable version of the multiple coin estimation problem, an elementary statistical problem that serves as the basis of our hardness proofs. Next, Appendix D.4 reduces locally random replicable  $Q$ -function estimation to the multiple coin estimation problem. Finally, Appendix D.5 reduces deterministic replicable policy estimation to locally random replicable  $Q$ -function estimation.
D.1 Omitted Details from Upper Bounds
First, we show that we can use any non-replicable  $Q$ -function estimation algorithm as a black-box to obtain a locally random replicable  $Q$ -function estimation algorithm.
Lemma D.1. Let  $\varepsilon', \delta' \in (0, 1)$ . Suppose there is an oracle  $\mathcal{O}$  that takes  $\widetilde{O}(f(N, \varepsilon', \delta'))$  samples from the generative model and outputs  $\widehat{Q}$  satisfying  $\| \widehat{Q} - Q^{\star} \|_{\infty} \leq \varepsilon'$  with probability at least  $1 - \delta'$ . Let  $\varepsilon, \rho \in (0, 1)$  and  $\delta \in (0, \rho/3)$ . There is a locally random  $\rho$ -replicable algorithm which makes a single call to  $\mathcal{O}$  and outputs some  $\bar{Q}$  satisfying  $\| \bar{Q} - Q^{\star} \|_{\infty} \leq \varepsilon$  with probability at least  $1 - \delta$ . Moreover, it has sample complexity  $\widetilde{O}\left(f(N, \varepsilon_0/N, \delta)\right)$  where  $\varepsilon_0 := \varepsilon (\rho - 2\delta) / (\rho + 1 - 2\delta)$ .
Proof (Lemma D.1). Consider calling  $\mathcal{O}$  with  $\widetilde{O}(f(N, \varepsilon_0 / N, \delta))$  samples. This ensures that
$$
\sum_ {s, a} \left| \widehat {Q} (s, a) - Q ^ {\star} (s, a) \right| \leq \varepsilon_ {0}
$$
with probability at least  $1 - \delta$ . By Theorem C.2, there is a locally random  $\rho$ -replicable algorithm that makes a single call to  $\mathcal{O}$  with the number of samples above and outputs  $\bar{Q}(s,a)$  such that
$$
\left\| \bar {Q} - Q ^ {\star} \right\| _ {\infty} \leq \varepsilon
$$
with a probability of success of at least  $1 - \delta$ .
Sidford et al. [2018a] showed a (non-replicable)  $Q$  function estimation algorithm which has optimal (non-replicable) sample complexity up to logarithmic factors.
Recall we write  $N \coloneqq \sum_{s \in \mathcal{S}} |\mathcal{A}^s|$  to denote the total number of state-action pairs.
Theorem D.2 ([Sidford et al., 2018a]). Let  $\varepsilon, \delta \in (0,1)$ , there is an algorithm that outputs an  $\varepsilon$ -optimal policy,  $\varepsilon$ -optimal value function, and  $\varepsilon$ -optimal  $Q$ -function with probability at least  $1 - \delta$  for any MDP. Moreover, it has time and sample complexity
$$
\widetilde {O} \left(\frac {N}{(1 - \gamma) ^ {3} \varepsilon^ {2}} \log \frac {1}{\delta}\right).
$$
The proof of Theorem 3.1, which we repeat below for convenience, follows by combining Lemma D.1 and Theorem D.2.
The following result of Singh and Yee [1994] relates the  $Q$ -function estimation error to the quality of the greedy policy with respect to the estimated  $Q$ -function.
Theorem D.3 ([Singh and Yee, 1994]). Let  $\widehat{Q}$  be a  $Q$ -function such that  $\| \widehat{Q} - Q^{\star} \|_{\infty} \leq \varepsilon$ . Let  $\pi(s) \coloneqq \arg \max_{a \in S} \widehat{Q}(s, a), \forall s \in S$ . Then,
$$
\| V ^ {\pi} - V ^ {\star} \| _ {\infty} \leq \frac {\varepsilon}{1 - \gamma}.
$$
Theorem D.3 enables us to prove Corollary 3.2, an upper bound on the sample complexity of replicably estimating an  $\varepsilon$ -optimal policy. We restate the corollary below for convenience.
Proof (Corollary 3.2). Apply the  $\rho$ -replicable algorithm from Theorem 3.1 to yield an  $\varepsilon_0$ -optimal  $Q$ -function and output the greedy policy based on this function. Theorem D.3 guarantees that the greedy policy derived from an  $\varepsilon_0$ -optimal  $Q$ -function is  $\varepsilon_0 / (1 - \gamma)$ -optimal. Choosing  $\varepsilon_0 \coloneqq (1 - \gamma) \varepsilon$  yields the desired result.
The replicability follows from the fact that conditioned on the event that the two  $Q$ -functions across the two runs are the same, which happens with probability at least  $1 - \rho$ , the greedy policies with respect to the underlying  $Q$ -functions will also be the same.
D.2 Lower Bounds for Replicable  $Q$ -Function & Policy Estimation
We now move on to the lower bounds and our approaches to obtain them. First, we describe a sample complexity lower bound for locally random  $\rho$ -replicable algorithms that seek to estimate  $Q^{\star}$ . Then, we reduce policy estimation to  $Q$ -estimation. Since the dependence of the sample complexity on the confidence parameter  $\delta$  of the upper bound is at most polylogarithmic, the main focus of the lower bound is on the dependence on the size of the state-action space  $N$ , the error parameter  $\varepsilon$ , the replicability parameter  $\rho$ , and the discount factor  $\gamma$ .
6 assuming a consistent tie-breaking rule
D.2.1 Intuition of the  $Q$ -Function Lower Bound
Our MDP construction that witnesses the lower bound relies on the sample complexity lower bound for locally random algorithms that replicably estimate the biases of multiple independent coins. Impagliazzo et al. [2022] showed that any  $\rho$ -replicable algorithm that estimates the bias of a single coin with accuracy  $\varepsilon$  requires at least  $\Omega(1 / \rho^2\varepsilon^2)$  samples (cf. Theorem D.13). We generalize this result and derive a lower bound for any locally random  $\rho$ -replicable algorithm that estimates the biases of  $N$  coins with accuracy  $\varepsilon$  and constant probability of success. We discuss our approach in Appendix D.2.2.
Next, given some  $\varepsilon, \rho, \gamma, N$ , we design an MDP for which estimating an approximately optimal  $Q$ -function is at least as hard as estimating  $N$  coins. The main technical challenge for this part of the proof is to establish the correct dependence on the parameter  $\gamma$  since it is not directly related to the coin estimation problem. We elaborate on it in Remark D.7.
Remark D.4. Our construction, combined with the non-replicable version of the coin estimation problem, can be used to simplify the construction of the non-replicable  $Q$ -estimation lower bound from Gheshlaghi Azar et al. [2013].
D.2.2 The Replicable Coin Estimation Problem
Formally, the estimation problem, without the replicability requirement, is defined as follows.
Problem D.5 (Multiple Coin Problem). Fix  $q, \varepsilon, \delta \in (0,1)^3$  such that  $q - \varepsilon \in (1/2,1)$ . Given sample access to  $N$  independent coins each with a bias of either  $q$  or  $q - \varepsilon$ , determine the bias of every coin with confidence at least  $1 - \delta$ .
We now informally state our main result for the multiple coin estimation problem, which could be useful in deriving replicability lower bounds beyond the scope of our work. See Theorem D.16 for the formal statement. Intuitively, this result generalizes Theorem D.13 to multiple instances.
Theorem D.6 (Informal). Suppose  $\mathcal{A}$  is a locally random  $\rho$ -replicable algorithm for the multiple coin problem with a constant probability of success. Then, the sample complexity of  $\mathcal{A}$  is at least
$$
\Omega \left(\frac {N ^ {3} q (1 - q)}{\varepsilon^ {2} \rho^ {2}}\right).
$$
Recall Yao's min-max principle [Yao, 1977], which roughly states that the expected cost of a randomized algorithm on its worst-case input is at least as expensive as the expected cost of any deterministic algorithm on random inputs chosen from some distribution. It is not clear how to apply Yao's principle directly, but we take inspiration from its essence and reduce the task of reasoning about a randomized algorithm with shared internal randomness to reasoning about a deterministic one with an additional layer of external randomness on top of the random flips of the coins.
Consider now a deterministic algorithm  $g$  for distinguishing the bias of a single coin where the input bias is chosen uniformly in  $[q - \varepsilon, q]$ . That is, we first choose  $p \sim U[q - \varepsilon, q]$ , then provide i.i.d. samples from  $\mathrm{Be}(p)$  to  $g$ . We impose some boundary conditions: if  $p = q - \varepsilon$ ,  $g$  should output “-” with high probability and if  $p = q$ , the algorithm should output “+” with high probability. We show that the probability of  $g$  outputting “+” varies smoothly with respect to the bias of the input coin. Thus, there is an interval  $I \subseteq (q - \varepsilon, q)$  such that  $g$  outputs “-” or “+” with almost equal probability and so the output of  $g$  is inconsistent across two executions with constant probability when  $p$  lands in this interval. By the choice of  $p \sim U[q - \varepsilon, q]$ , if  $\ell(I)$  denotes the length of  $I$ , then the output of  $g$  is inconsistent across two executions with probability at least  $\Omega(\ell(I)/\varepsilon)$ . Quantifying  $\ell(I)$  and rearranging yields the lower bound for a single coin.
For the case of  $N$  independent coins, we use the pigeonhole principle to reduce the argument to the case of a single coin. The formal statement and proof of Theorem D.6 is deferred to Appendix D.3. Remark D.7. The lower bound from Impagliazzo et al. [2022] for the single-coin estimation problem holds for the regime  $q, q - \varepsilon \in (1/4, 3/4)$ . We remove this constraint by analyzing the dependence of the lower bound on  $q$ . When reducing  $Q$ -function estimation to the multiple coin problem, the restricted regime yields a lower bound proportional to  $(1 - \gamma)^{-2}$ . In order to derive the stronger lower bound of  $(1 - \gamma)^{-3}$ , we must be able to choose  $q \approx \gamma$  which can be arbitrarily close to 1.
In Section 4, we show that allowing for non-locally random algorithms enables us to shave off a factor of  $N$  in the sample complexity. We also conjecture that this upper bound is tight.