SlowGuess/NeurIPS_2023_OCR huggingface dataset:
We begin by providing upper bounds on the sample complexity for replicable estimation of an approximately optimal policy and  $Q$ -function. On a high level, we follow a two-step approach: 1) Start with black-box access to some  $Q$ -estimation algorithm that is not necessarily replicable (cf. Theorem D.2) to estimate some  $\widehat{Q}$  such that  $\| Q^{\star} - \widehat{Q}\|_{\infty} \leq \varepsilon_0$ . 2) Apply the replicable rounding algorithm from Theorem C.2 as a post-processing step. The rounding step incurs some loss of accuracy in the estimated  $Q$ -function. Therefore, in order to balance between  $\rho$ -replicability and  $(\varepsilon, \delta)$ -accuracy, we need to call the black-box oracle with an accuracy smaller than  $\varepsilon$ , i.e. choose  $\varepsilon_0 < O(\varepsilon \rho)$ . This yields an increase in the sample complexity which we quantify below. For the proof details, see Appendix D.1.
Recall that  $N$  is the number of state-action pairs of the MDP.
Theorem 3.1. Let  $\varepsilon, \rho \in (0,1)^2$  and  $\delta \in (0,\rho /3)$ . There is a locally random  $\rho$ -replicable algorithm that outputs an  $\varepsilon$ -optimal  $Q$ -function with probability at least  $1 - \delta$ . Moreover, it has time and sample complexity  $\widetilde{O}(N^3\log(1/\delta)/[(1 - \gamma)^3\varepsilon^2\rho^2])$ .
So far, we have provided a replicable algorithm that outputs an approximately optimal  $Q$  function. The main result of Singh and Yee [1994] shows that if  $\| \widehat{Q} - Q^{\star} \|_{\infty} \leq \varepsilon$ , then the greedy policy with respect to  $\widehat{Q}$ , i.e.,  $\forall s \in S, \widehat{\pi}(s) := \operatorname{argmax}_{a \in A^{s}} \widehat{Q}(s, a)$ , is  $\varepsilon/(1-\gamma)$ -approximately optimal (cf. Theorem D.3). Thus, if we want to obtain an  $\varepsilon$ -approximately optimal policy, it suffices to obtain a  $(1-\gamma)\varepsilon$ -approximately optimal  $Q$ -function. This is formalized in Corollary 3.2.
Corollary 3.2. Let  $\varepsilon, \rho \in (0,1)^2$  and  $\delta \in (0,\rho /3)$ . There is a locally random  $\rho$ -replicable algorithm that outputs an  $\varepsilon$ -optimal policy with probability at least  $1 - \delta$ . Moreover, it has time and sample complexity  $\widetilde{O}(N^3\log(1/\delta)/[(1 - \gamma)^5\varepsilon^2\rho^2])$ .
Again, we defer the proof to Appendix D.1.
Due to space limitation, the lower bound derivation is postponed to Appendix D.
4 TV Indistinguishable Algorithms for  $Q$ -Function and Policy Estimation
In this section, we present an algorithm with an improved sample complexity for replicable  $Q$ -function estimation and policy estimation. Our approach consists of several steps. First, we design a computationally efficient SQ algorithm for answering  $d$  statistical queries that satisfies the total variation (TV) indistinguishability property [Kalavasis et al., 2023] (cf. Definition 4.1), which can be viewed as a relaxation of replicability. The new SQ algorithm has an improved sample complexity compared to its replicable counterpart we discussed previously. Using this oracle, we show how we can design computationally efficient  $Q$ -function estimation and policy estimation algorithms that
$^{5}$ up to logarithmic factors
satisfy the TV indistinguishability definition and have an improved sample complexity by a factor of  $N$  compared to the ones in Section 3.1. Then, by describing a specific implementation of its internal randomness, we make the algorithm replicable. Unfortunately, this step incurs an exponential cost in the computational complexity of the algorithm with respect to the cardinality of the state-action space. We emphasize that the reason we are able to circumvent the lower bound of Appendix D.2 is that we use a specific source of internal randomness that creates correlations across the random choices of the learner. Our result reaffirms the observation made by Kalavasis et al. [2023] that the same learning algorithm, i.e., input  $\rightarrow$  output mapping, can be replicable under one implementation of its internal randomness but not replicable under a different one.
First, we state the definition of TV indistinguishability from Kalavasis et al. [2023].
Definition 4.1 (TV Indistinguishability; [Kalavasis et al., 2023]). A learning rule  $\mathcal{A}$  is  $n$ -sample  $\rho$ -TV indistinguishable if for any distribution over inputs  $\mathcal{D}$  and two independent samples  $S, S' \sim \mathcal{D}^n$  it holds that  $\mathbb{E}_{S, S' \sim \mathcal{D}^n}[d_{\mathrm{TV}}(A(S), A(S'))] \leq \rho$ .
In their work, Kalavasis et al. [2023] showed how to transform any  $\rho$ -TV indistinguishable algorithm to a  $2\rho / (1 + \rho)$ -replicable one when the input domain is countable. Importantly, this transformation does not change the input  $\rightarrow$  output mapping that is induced by the algorithm. A similar transformation for finite domains can also be obtained by the results in Bun et al. [2023]. We emphasize that neither of these two transformations are computationally efficient. Moreover, Bun et al. [2023] give cryptographic evidence that there might be an inherent computational hardness to obtain the transformation.
4.1 TV Indistinguishable Estimation of Multiple Statistical Queries
We are now ready to present a TV-indistinguishable algorithm for estimating  $d$  independent statistical queries. The high-level approach is as follows. First, we estimate each statistical query up to accuracy  $\varepsilon \rho /\sqrt{d}$  using black-box access to the SQ oracle and we get an estimate  $\widehat{\mu}_1\in [0,1]^d$ . Then, the output of the algorithm is drawn from  $\mathcal{N}(\widehat{\mu}_1,\varepsilon^2 I_d)$ . Since the estimated mean of each query is accurate up to  $\varepsilon \rho /\sqrt{d}$  and the variance is  $\varepsilon^2$ , we can see that, with high probability, the estimate of each query will be accurate up to  $O(\varepsilon)$ . To argue about the TV indistinguishability property, we first notice that, with high probability across the two executions, the estimate  $\widehat{\mu}_2\in [0,1]^d$  satisfies  $\| \widehat{\mu}_1 - \widehat{\mu}_2\|_{\infty}\leq 2\rho \cdot \varepsilon /\sqrt{d}$ . Then, we can bound the TV distance of the output of the algorithm as  $d_{\mathrm{TV}}\left(\mathcal{N}(\widehat{\mu}_1,\varepsilon^2 I_d),\mathcal{N}(\widehat{\mu}_2,\varepsilon^2 I_d)\right)\leq O(\rho)$  [Gupta, 2020]. We underline that this behavior is reminiscent of the advanced composition theorem in the Differential Privacy (DP) literature (see e.g., Dwork et al. [2014]) and our algorithm can be viewed as an extension of the Gaussian mechanism from the DP line of work to the replicability setting. This algorithm has applications outside the scope of our work since multiple statistical query estimation is a subroutine widely used in the replicability line of work [Impagliazzo et al., 2022, Esfandiari et al., 2023b,a, Bun et al., 2023, Kalavasis et al., 2023]. This discussion is formalized in the following theorem.
Theorem 4.2 (TV Indistinguishable SQ Oracle for Multiple Queries). Let  $\varepsilon, \rho \in (0,1)^2$  and  $\delta \in (0,\rho /5)$ . Let  $\phi_1,\ldots,\phi_d$  be  $d$  statistical queries with co-domain [0,1]. Assume that we can simultaneously estimate the true values of all  $\phi_i$ 's with accuracy  $\varepsilon$  and confidence  $\delta$  using  $n(\varepsilon,\delta)$  total samples. Then, there exists a  $\rho$ -TV indistinguishable algorithm (Algorithm A.1) that requires at most  $n\left(\frac{\varepsilon\rho}{[2\sqrt{8d\cdot\log(4d / \delta)}]},\frac{\delta}{2}\right)$  many samples to output estimates  $\widehat{v}_1,\dots,\widehat{v}_d$  of the true values  $v_{1},\ldots,v_{d}$  to guarantee that  $\max_{i\in [d]}\left|\widehat{v}_i - v_i\right| \leq \varepsilon$ , with probability at least  $1 - \delta$ .
4.2 TV Indistinguishable  $Q$ -Function and Policy Estimation
Equipped with Algorithm A.1, we are now ready to present a TV-indistinguishable algorithm for  $Q$ -function estimation and policy estimation with superior sample complexity compared to the one in Section 3.1. The idea is similar to the one in Section 3.1. We start with black-box access to an algorithm for  $Q$ -function estimation, and then we apply the Gaussian mechanism (Algorithm A.1). We remark that the running time of this algorithm is polynomial in all the parameters of the problem.
Theorem 4.3. Let  $\varepsilon, \rho \in (0,1)^2$  and  $\delta \in (0,\rho /5)$ . There is a  $\rho$ -TV indistinguishable algorithm that outputs an  $\varepsilon$ -optimal  $Q$ -function with probability at least  $1 - \delta$ . Moreover, it has time and sample complexity  $\widetilde{O}(N^2\log(1/\delta)/[(1 - \gamma)^3\varepsilon^2\rho^2])$ .
Proof. The proof follows by combining the guarantees of Sidford et al. [2018a] (Theorem D.2) and Theorem 4.2. To be more precise, Theorem D.2 shows that in order to compute some  $\widehat{Q}$  such that  $\| \widehat{Q} - Q\|_{\infty} \leq \varepsilon$ , one needs  $\widetilde{O}(N\log(1/\delta)/[(1-\gamma)^3\varepsilon^2])$ . Thus, in order to apply Theorem 4.2 the sample complexity becomes  $\widetilde{O}(N^2\log(1/\delta)/[(1-\gamma)^3\varepsilon^2\rho^2])$ .
Next, we describe a TV indistinguishable algorithm that enjoys similar sample complexity guarantees. Similarly as before, we use the main result of Singh and Yee [1994] which shows that if  $\| \widehat{Q} - Q^{\star} \|_{\infty} \leq \varepsilon$ , then the greedy policy with respect to  $\widehat{Q}$ , i.e.,  $\forall s \in S, \widehat{\pi}(s) := \operatorname{argmax}_{a \in \mathcal{A}^{s}} \widehat{Q}(s, a)$ , is  $\varepsilon/(1-\gamma)$ -approximately optimal (cf. Theorem D.3). Thus, if we want to obtain an  $\varepsilon$ -approximately optimal policy, it suffices to obtain a  $(1-\gamma)\varepsilon$ -approximately optimal  $Q$ -function. The indistinguishable guarantee follows from the data-processing inequality This is formalized in Corollary 4.4.
Corollary 4.4. Let  $\varepsilon, \rho \in (0,1)^2$  and  $\delta \in (0,\rho /5)$ . There is a  $\rho$ -TV indistinguishable algorithm that outputs an  $\varepsilon$ -optimal policy with probability at least  $1 - \delta$ . Moreover, it has time and sample complexity  $\widetilde{O}(N^2\log(1/\delta)/[(1-\gamma)^5\varepsilon^2\rho^2])$ .
4.3 From TV Indistinguishability to Replicability
We now describe how we can transform the TV indistinguishable algorithms we provided to replicable ones. As we alluded to before, this transformation does not hurt the sample complexity, but requires exponential time in the state-action space. Our transformation is based on the approach proposed by Kalavasis et al. [2023] which holds when the input domain is countable. Its main idea is that when two random variables follow distributions that are  $\rho$ -close in TV-distance, then there is a way to couple them using only shared randomness. The implementation of this coupling is based on the Poisson point process and can be thought of a generalization of von Neumann's rejection-based sampling to handle more general domains. We underline that in general spaces without structure it is not known yet how to obtain such a coupling. However, even though the input domain of the Gaussian mechanism is uncountable and the result of Kalavasis et al. [2023] does not apply directly in our setting, we are able to obtain a similar transformation as they did. The main step required to perform this transformation is to find a reference measure with respect to which the algorithm is absolutely continuous. We provide these crucial measure-theoretic definitions below.
Definition 4.5 (Absolute Continuity). Consider two measures  $P, \mathcal{P}$  on a  $\sigma$ -algebra  $\mathcal{B}$  of subsets of  $W$ . We say that  $P$  is absolutely continuous with respect to  $\mathcal{P}$  if for any  $E \in \mathcal{B}$  such that  $\mathcal{P}(E) = 0$ , it holds that  $P(E) = 0$ .
Recall that  $\mathcal{A}(S)$  denotes the distribution over outputs, when the input to the algorithm is  $S$ .
Definition 4.6. Given a learning rule  $\mathcal{A}$  and reference probability measure  $\mathcal{P}$ , we say that  $\mathcal{A}$  is absolutely continuous with respect to  $\mathcal{P}$  if for any input  $S$ ,  $\mathcal{A}(S)$  is absolutely continuous with respect to  $\mathcal{P}$ .
We emphasize that this property should hold for every fixed sample  $S$ , i.e., the randomness of the samples are not taken into account.
We now define what it means for two learning rules to be equivalent.
Definition 4.7 (Equivalent Learning Rules). Two learning rules  $\mathcal{A},\mathcal{A}'$  are equivalent if for every fixed sample  $S$ , it holds that  $\mathcal{A}(S)\stackrel {d}{=}\mathcal{A}'(S)$ , i.e., for the same input they induce the same distribution over outputs.
Using a coupling technique based on the Poisson point process, we can convert the TV indistinguishable learning algorithms we have proposed so far to equivalent ones that are replicable. See Algorithm A.2 for a description of how to output a sample from this coupling. Let us view  $\mathcal{A}(S;r),\mathcal{A}(S';r)$  as random vectors with small TV distance. The idea is to implement the shared internal randomness  $r$  using rejection sampling so that the "accepted" sample will be the same across two executions with high probability. For some background regarding the Poisson point process and the technical tools we use, we refer the reader to Appendix C.2.
Importantly, for every  $S$ , the output  $\mathcal{A}(S)$  of the algorithms we have proposed in Section 4.1 and Section 4.2 follow a Gaussian distribution, which is absolutely continuous with respect to the Lebesgue
measure. Furthermore, the Lebesgue measure is  $\sigma$ -finite so we can use the coupling algorithm (cf. Algorithm A.2) of Angel and Spinka [2019], whose guarantees are stated in Theorem C.4. We are now ready to state the result regarding the improved  $\rho$ -replicable SQ oracle for multiple queries. Its proof is an adaptation of the main result of Kalavasis et al. [2023].
Theorem 4.8 (Replicable SQ Oracle for Multiple Queries). Let  $\varepsilon, \rho \in (0,1)^2$  and  $\delta \in (0,\rho /5)$ . Let  $\phi_1,\ldots,\phi_d$  be  $d$  statistical queries with co-domain [0, 1]. Assume that we can simultaneously estimate the true values of all  $\phi_i$ 's with accuracy  $\varepsilon$  and confidence  $\delta$  using  $n(\varepsilon,\delta)$  total samples. Then, there exists a  $\rho$ -replicable algorithm that requires at most  $n\left(\frac{\varepsilon\rho}{4\sqrt{8d\cdot\log(4d / \delta)}},\frac{\delta}{2}\right)$  many samples to output estimates  $\widehat{v}_1,\ldots,\widehat{v}_d$  of the true values  $v_{1},\ldots,v_{d}$  with the guarantee that  $\max_{i\in [d]}\left|\widehat{v}_i - v_i\right| \leq \varepsilon$  with probability at least  $1 - \delta$ .
By using an identical argument, we can obtain  $\rho$ -replicable algorithms for  $Q$ -function estimation and policy estimation. Recall that  $N$  is the number of state-action pairs of the MDP.
Theorem 4.9. Let  $\varepsilon, \rho \in (0,1)^2$  and  $\delta \in (0,\rho /4)$ . There is a  $\rho$ -replicable algorithm that outputs an  $\varepsilon$ -optimal  $Q$ -function with probability at least  $1 - \delta$ . Moreover, it has sample complexity  $\widetilde{O}(N^2\log(1/\delta)/[(1 - \gamma)^3\varepsilon^2\rho^2])$ .
Corollary 4.10. Let  $\varepsilon, \rho \in (0,1)^2$  and  $\delta \in (0,\rho /4)$ . There is a  $\rho$ -replicable algorithm that outputs an  $\varepsilon$ -optimal policy with probability at least  $1 - \delta$ . Moreover, it has sample complexity  $\widetilde{O}(N^2\log(1/\delta)/[(1 - \gamma)^5\varepsilon^2\rho^2])$ .
In Remark E.2 we explain why we cannot use a coordinate-wise coupling.
5 Approximately Replicable Policy Estimation
The definitions of replicability (cf. Definition 2.10, Definition 4.1) we have discussed so far suffer from a significant sample complexity blow-up in terms of the cardinality of the state-action space which can be prohibitive in many settings of interest. In this section, we propose approximate replicability, a relaxation of these definitions, and show that this property can be achieved with a significantly milder sample complexity compared to (exact) replicability. Moreover, this definition does not require shared internal randomness across the executions of the algorithm.
First, we define a general notion of approximate replicability as follows.
Definition 5.1 (Approximate Replicability). Let  $\mathcal{X},\mathcal{Y}$  be the input and output domains, respectively. Let  $\kappa :\mathcal{Y}\times \mathcal{Y}\to \mathbb{R}_{\geq 0}$  be some distance function on  $\mathcal{V}$  and let  $\rho_{1},\rho_{2}\in (0,1)^{2}$ . We say that an algorithm  $\mathcal{A}$  is  $(\rho_1,\rho_2)$ -approximately replicable with respect to  $\kappa$  if for any distribution  $\mathcal{D}$  over  $\mathcal{X}$  it holds that  $\mathbb{P}_{S,S^{\prime}\sim \mathcal{D}^{n},r,r^{\prime}\sim \mathcal{R}}\{\kappa (\mathcal{A}(S;r),\mathcal{A}(S^{\prime};r^{\prime}))\geq \rho_{1}\} \leq \rho_{2}$ .
In words, this relaxed version of Definition 2.8 requires that the outputs of the algorithm, when executed on two sets of i.i.d. data, using independent internal randomness across the two executions, are close under some appropriate distance measure. In the context of our work, the output of the learning algorithm is some policy  $\pi : S \to \Delta(\mathcal{A})$ , where  $\Delta(\mathcal{A})$  denotes the probability simplex over  $\mathcal{A}$ . Thus, it is natural to instantiate  $\kappa$  as some dissimilarity measure of distributions like the total variation (TV) distance or the Renyi divergence. For the exact definition of these dissimilarity measures, we refer the reader to Appendix B. We now state the definition of an approximately replicable policy estimator.
Definition 5.2 (Approximately Replicable Policy Estimator). Let  $\mathcal{A}$  be an algorithm that takes as input samples of state-action pair transitions and returns a policy  $\pi$ . Let  $\kappa$  be some dissimilarity measure on  $\Delta(\mathcal{A})$  and let  $\rho_1, \rho_2 \in (0,1)^2$ . We say that  $\mathcal{A}$  is  $(\rho_1, \rho_2)$ -approximately replicable if for any MDP  $M$  it holds that  $\mathbb{P}_{S,S' \sim G,r',r' \sim \mathcal{R}}\left\{\max_{s \in S} \kappa(\pi(s), \pi'(s)) \geq \rho_1\right\} \leq \rho_2$ , where  $G$  is the generator of state-action pair transitions,  $\mathcal{R}$  is the source of internal randomness of  $\mathcal{A}$ ,  $\pi$  is the output of  $\mathcal{A}$  on input  $S, r$ , and  $\pi'$  is its output on input  $S', r'$ .
To the best of our knowledge, the RL algorithms that have been developed for the model we are studying do not satisfy this property. Nevertheless, many of them compute an estimate  $Q$  with the promise that  $\| Q - Q^{\star} \|_{\infty} \leq \varepsilon$  [Sidford et al., 2018a, Agarwal et al., 2020, Li et al., 2020]. Thus, it is not hard to see that if we run the algorithm twice on independent data with independent internal randomness we have that  $\| Q - Q' \|_{\infty} \leq 2\varepsilon$ . This is exactly the main property that we need in order
to obtain approximately replicable policy estimators. The key idea is that instead of outputting the greedy policy with respect to this  $Q$ -function, we output a policy given by some soft-max rule. Such a rule is a mapping  $\mathbb{R}_{\geq 0}^{\mathcal{A}} \rightarrow \Delta(\mathcal{A})$  that achieves two desiderata: (i) The distribution over the actions is "stable" with respect to perturbations of the  $Q$ -function. (ii) For every  $s \in S$ , the value of the policy  $V^{\pi}(s)$  that is induced by this mapping is "close" to  $V^{\star}(s)$ .
Formally, the stability of the soft-max rule is captured through its Lipschitz constant (cf. Definition B.3). In this setting, this means that whenever the two functions  $Q$ ,  $Q'$  are close under some distance measure (e.g. the  $\ell_{\infty}$  norm), then the policies that are induced by the soft-max rule are close under some (potentially different) dissimilarity measure. The approximation guarantees of the soft-max rules are captured by the following definition.
Definition 5.3 (Soft-Max Approximation; [Epatso et al., 2020]). Let  $\varepsilon > 0$ . A soft-max function  $f: \mathbb{R}_{\geq 0}^{\mathcal{A}} \to \Delta(\mathcal{A})$  is  $\varepsilon$ -approximate if for all  $x \in \mathbb{R}^{\mathcal{A}}$ ,  $\langle f(x), x \rangle \geq \max_{a \in \mathcal{A}} x_a - \varepsilon$ .
In this work, we focus on the soft-max rule that is induced by the exponential function (ExpSoftMax), which has been studied in several application domains [Gibbs, 1902, McSherry and Talwar, 2007, Huang and Kannan, 2012, Dwork et al., 2014, Gao and Pavel, 2017]. Recall  $\pi(s,a)$  denotes the probability mass that policy  $\pi$  puts on action  $a\in \mathcal{A}^s$  in state  $s\in S$ . Given some  $\lambda >0$  and  $Q(s,a)\in \mathbb{R}^{S\times \mathcal{A}}$ , the induced randomized policy  $\pi$  is given by  $\pi (s,a) = \frac{\exp\{\lambda Q(s,a)\}}{\sum_{a'\in\mathcal{A}^s}\exp\{\lambda Q(s,a')\}}$ . For a discussion about the advantages of using more complicated soft-max rules like the one developed in Epasto et al. [2020], we refer the reader to Appendix F.1.
We now describe our results when we consider approximate replicability with respect to the Renyi divergence and the Total Variation (TV) distance. At a high level, our approach is divided into two steps: 1) Run some  $Q$ -learning algorithm (e.g. [Sidford et al., 2018a, Agarwal et al., 2020, Li et al., 2020]) to estimate some  $\widehat{Q}$  such that  $\| Q^{\star} - \widehat{Q}\|_{\infty} \leq \varepsilon$ . 2) Estimate the policy using some soft-max rule. One advantage of this approach is that it allows for flexibility and different implementations of these steps that better suit the application domain. An important lemma we use is the following.
Lemma 5.4 (Exponential Soft-Max Approximation Guarantee; [McSherry and Talwar, 2007]). Let  $\varepsilon \in (0,1)$ ,  $\alpha, p \geq 1$ , and set  $\lambda = \log(d) / \varepsilon$ , where  $d$  is the ambient dimension of the input domain. Then,  $\mathrm{ExpSoftMax}$  with parameter  $\lambda$  is  $\varepsilon$ -approximate and  $2\lambda$ -Lipschitz continuous (cf. Definition B.3) with respect to  $(\ell_p, D_\alpha)$ , where  $D_\alpha$  is the Renyi divergence of order  $\alpha$ .
This is an important building block of our proof. However, it is not sufficient on its own in order to bound the gap of the ExpSoftMax policy and the optimal one. This is handled in the next lemma whose proof is postponed to Appendix F. Essentially, it can be viewed as an extension of the result in Singh and Yee [1994] to handle the soft-max policy instead of the greedy one.
Lemma 5.5 (Soft-Max Policy vs Optimal Policy). Let  $\varepsilon_1, \varepsilon_2 \in (0,1)^2$ . Let  $\widehat{Q} \in \mathbb{R}^{S \times A}$  be such that  $\| \widehat{Q} - Q^{\star} \| \leq \varepsilon_1$ . Let  $\hat{\pi}$  be the ExpSoftMax policy with respect to  $\widehat{Q}$  using parameter  $\lambda = \log |\mathcal{A}| / \varepsilon_2$ . Then,  $\left\| V^{\hat{\pi}} - V^{\star} \right\|_{\infty} \leq \frac{(2\varepsilon_1 + \varepsilon_2)}{(1 - \gamma)}$ .
Combining Lemma 5.4 and Lemma 5.5 yields the desired approximate replicability guarantees we seek. The formal proof of the following result is postponed to Appendix F. Recall we write  $N \coloneqq \sum_{s \in S} |\mathcal{A}^s|$  to denote the total number of state-action pairs.
Theorem 5.6. Let  $\alpha \geq 1, \gamma, \delta, \rho_1, \rho_2 \in (0,1)^4$ , and  $\varepsilon \in (0,(1 - \gamma)^{-1 / 2})$ . There is a  $(\rho_1, \rho_2)$ -approximately replicable algorithm  $\mathcal{A}$  with respect to the Renyi divergence  $D_{\alpha}$  such that given access to a generator  $G$  for any MDP  $M$ , it outputs a policy  $\hat{\pi}$  for which  $\|V^{\hat{\pi}} - V^{\star}\|_{\infty} \leq \varepsilon$  with probability at least  $1 - \delta$ . Moreover,  $\mathcal{A}$  has time and sample complexity  $\widetilde{O}(N\log(1/\min\{\delta, \rho_2\}) / [(1 - \gamma)^5\varepsilon^2\rho_1^2])$ .