SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Last but not least, we prove Theorem 5.6, an upper bound on approximately-replicable policy estimation. We restate the result below for convenience.
Theorem 5.6. Let  $\alpha \geq 1, \gamma, \delta, \rho_1, \rho_2 \in (0,1)^4$ , and  $\varepsilon \in (0,(1 - \gamma)^{-1 / 2})$ . There is a  $(\rho_1, \rho_2)$ -approximately replicable algorithm  $\mathcal{A}$  with respect to the Renyi divergence  $D_{\alpha}$  such that given access to a generator  $G$  for any MDP  $M$ , it outputs a policy  $\hat{\pi}$  for which  $\|V^{\hat{\pi}} - V^{\star}\|_{\infty} \leq \varepsilon$  with probability at least  $1 - \delta$ . Moreover,  $\mathcal{A}$  has time and sample complexity  $\widetilde{O}(N\log(1/\min\{\delta, \rho_2\})/[(1 - \gamma)^5\varepsilon^2\rho_1^2])$ .
Proof (Theorem 5.6). First, we argue about the correctness of the approach. Using Theorem D.2 with parameters  $\gamma, \delta/2, \varepsilon_1 = \rho_1 \cdot \varepsilon \cdot (1 - \gamma) / (8\log(|\mathcal{A}|))$ , we can estimate some  $\widehat{Q}_1$  such that  $||\widehat{Q}_1 - Q^\star||_\infty \leq \varepsilon_1$ , with probability at least  $1 - \delta/2$ . We call this event  $\mathcal{E}_1$  and we condition on it. Since we use the exponential mechanism with parameter  $\lambda = \log(|\mathcal{A}|) / (\varepsilon/2 \cdot (1 - \gamma))$  to compute a policy  $\widehat{\pi}(s)$ , for every state  $s \in S$ , Lemma 5.5 guarantees that
$$
\left| \left| V ^ {\star} - V ^ {\widehat {\pi}} \right| \right| _ {\infty} \leq \frac {2 \rho_ {1} \cdot \varepsilon \cdot (1 - \gamma) / (8 \log (| \mathcal {A} |)) + \varepsilon / 2 \cdot (1 - \gamma)}{1 - \gamma} \leq \varepsilon .
$$
This completes the proof of correctness.
We now proceed with showing the replicability guarantees of this process. Consider two executions of the algorithm and let  $\widehat{Q}_1, \widehat{Q}_2$  denote the output of the algorithm described in Theorem D.2 in the first and second run, respectively. Notice that, with probability at least  $1 - \min\{\delta, \rho_2\}$  it holds that  $||\widehat{Q}_1 - Q^\star||_\infty \leq \varepsilon_1, ||\widehat{Q}_2 - Q^\star||_\infty \leq \varepsilon_1$ . We call this event  $\mathcal{E}_2$  and condition on it for the rest of the proof. Thus, by the triangle inequality, we have that
$$
\left| \left| \widehat {Q} _ {1} - \widehat {Q} _ {2} \right| \right| _ {\infty} \leq 2 \varepsilon_ {1}.
$$
Let  $\widehat{\pi}_1, \widehat{\pi}_2$  denote the policy the algorithm outputs in the first and second execution, respectively. Let  $s \in S$  be some arbitrary state. Since the exponential soft-max is  $2\lambda$ -Lipschitz continuous with respect to  $(\ell_{\infty}, D_{\alpha})$  we have that
$$
\begin{array}{l} D _ {\alpha} \left(\pi_ {1} (s) | | \pi_ {2} (s)\right) \leq 2 \lambda | | \widehat {Q} _ {1} - \widehat {Q} _ {2} | | _ {\infty} \\ = 2 \cdot \frac {\log (| \mathcal {A} |)}{(\varepsilon / 2 \cdot (1 - \gamma))} \cdot 2 \cdot \frac {\rho_ {1} \cdot \varepsilon \cdot (1 - \gamma)}{(8 \log (| \mathcal {A} |))} \\ = \rho_ {1}. \\ \end{array}
$$
Since the event  $\mathcal{E}_2$  happens with probability  $1 - \rho_{2}$  we see that the algorithm is  $(\rho_{1},\rho_{2})$ -approximately replicable with respect to  $D_{\alpha}$ , i.e., the Renyi divergence of order  $\alpha$ . This concludes the approximate replicability part of the proof.
As a last step, we bound the sample complexity of our algorithm. The stated bound follows directly from Theorem D.2 since we call this algorithm with error parameter  $\rho_1\cdot \varepsilon \cdot (1 - \gamma) / (8\log (|\mathcal{A}|))$
Remark F.1 (Replicability Under TV Distance). It is known that the TV distance of two probability distributions is upper bounded by  $D_{\infty}$ . Thus, we can see that Theorem 5.6 provides the same guarantees when we want to establish replicability with respect to the TV distance.
Remark F.2 (Sample Complexity Dependence on Replicability Parameters). Notice that the dependence of the number of samples in Theorem 5.6 on the two different replicability parameters of Definition 5.2 is different. In particular, the dependence on  $\rho_{2}$  is  $\mathrm{polylog}(1 / \rho_2)$ , whereas the dependence on  $\rho_{1}$  is  $\mathrm{poly}(1 / \rho_1)$ .
G Guarantees under Different Replicability Notions
Since we have studied three different replicability notions in this work, we believe it is informative to discuss the advantages and the drawbacks of each one of them. Our discussion is centered across four different axes: the replicability guarantees that each notion provides, the sample complexity required to satisfy each definition, the running time required to run the underlying algorithms, and the ability to test/verify whether the algorithms have the desired replicability properties.
The definition of Impagliazzo et al. [2022] (Definition 2.8) provides the strongest replicability guarantees since it requires that the two outputs are exactly the same across the two executions. It is also computationally efficient to verify it. Even though it is statistically equivalent to the definition of TV indistinguishability, our results along with the results of Bun et al. [2023], Kalavasis et al. [2023] indicate that there might be a computational separation between these two notions. Moreover, the fact that this notion is so tightly related to the way the internal randomness of the algorithm is implemented is a property that is not exhibited by any other notion of stability we are aware of and can be problematic in some applications.
The definition of TV indistinguishability of Kalavasis et al. [2023] (Definition 4.1) provides strong replicability guarantees, in the sense that someone who observes the outputs of the algorithm
under two executions when the inputs are  $S, S'$ , cannot distinguish which one between  $S, S'$  was responsible for generating this output. Moreover, this definition does not depend on the way the internal randomness of the algorithm is implemented. On the other hand, testing whether an algorithm has this property is more subtle compared to Definition 2.8. In the case of the Gaussian mechanism-based algorithms we discuss in this work the following holds: if the output of the algorithm is promised to be drawn from a Gaussian distribution, it is computationally and statistically efficient to test whether the outputs under two different datasets  $S, S'$  are close in TV distance. However, it is not clear how one can test if the outputs are indeed drawn from a Gaussian distribution.
Finally, the notion of approximate replicability (Definition 5.1) we introduce is a further relaxation of the TV indistinguishability property in the following sense: both the replicability definition and TV indistinguishability definition treat the outputs in a "binary" manner, in the sense that they only care whether the outputs are exactly the same across the two executions. This definition takes a more nuanced approach and considers some notion of distance across the outputs that is not binary. As a result, it provides the weakest replicability guarantees, which, however, could be sufficient in most RL applications. Moreover, as our results indicate, there might be some inherent advantage in terms of the sample complexity required to achieve this notion compared to (strict) replicability or TV indistinguishability, which can be crucial in RL applications with large state-action space. Moreover, similarly as with the replicability definition, it is also efficient to test whether an algorithm has this property or not.
To sum up, even though we have not completely characterized the sample complexity and computational complexity of each definition we believe that the following is the complete picture: the replicability property is statistically equivalent to the TV indistinguishability property and the approximate replicability property has sample complexity that is smaller by a factor of  $N$ . Moreover, we believe that there is a computational gap between the notions of replicability and TV indistinguishability. We underline that under Conjecture D.8, the results of our work give a complete characterization<sup>9</sup> of the sample complexity of these problems with respect to  $N$ .
${}^{9}$  Up to poly-logarithmic factors.