SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Replicability in Reinforcement Learning*
linyang@ee.ucla.edu
Felix Zhou
felix.zhou@yale.edu
We initiate the mathematical study of replicability as an algorithmic property in the context of reinforcement learning (RL). We focus on the fundamental setting of discounted tabular MDPs with access to a generative model. Inspired by Impagliazzo et al. [2022], we say that an RL algorithm is replicable if, with high probability, it outputs the exact same policy after two executions on i.i.d. samples drawn from the generator when its internal randomness is the same. We first provide an efficient  $\rho$ -replicable algorithm for  $(\varepsilon, \delta)$ -optimal policy estimation with sample and time complexity  $\widetilde{O}\left(\frac{N^3 \cdot \log(1 / \delta)}{(1 - \gamma)^5 \cdot \varepsilon^2 \cdot \rho^2}\right)$ , where  $N$  is the number of state-action pairs. Next, for the subclass of deterministic algorithms, we provide a lower bound of order  $\Omega\left(\frac{N^3}{(1 - \gamma)^3 \cdot \varepsilon^2 \cdot \rho^2}\right)$ . Then, we study a relaxed version of replicability proposed by Kalavasis et al. [2023] called TV indistinguishability. We design a computationally efficient TV indistinguishable algorithm for policy estimation whose sample complexity is  $\widetilde{O}\left(\frac{N^2 \cdot \log(1 / \delta)}{(1 - \gamma)^5 \cdot \varepsilon^2 \cdot \rho^2}\right)$ . At the cost of  $\exp(N)$  running time, we transform these TV indistinguishable algorithms to  $\rho$ -replicable ones without increasing their sample complexity. Finally, we introduce the notion of approximate-replicability where we only require that two outputted policies are close under an appropriate statistical divergence (e.g., Renyi) and show an improved sample complexity of  $\widetilde{O}\left(\frac{N \cdot \log(1 / \delta)}{(1 - \gamma)^5 \cdot \varepsilon^2 \cdot \rho^2}\right)$ .
When designing a reinforcement learning (RL) algorithm, how can one ensure that when it is executed twice in the same environment its outcome will be the same? In this work, our goal is to design RL algorithms with provable replicability guarantees. The lack of replicability in scientific research, which the community also refers to as the reproducibility crisis, has been a major recent concern. This can be witnessed by an article that appeared in Nature [Baker, 2016]: Among the 1,500 scientists who participated in a survey,  $70\%$  of them could not replicate other researchers' findings and more shockingly,  $50\%$  of them could not even reproduce their own results. Unfortunately, due to the exponential increase in the volume of Machine Learning (ML) papers that are being published each year, the ML community has also observed an alarming increase in the lack of reproducibility. As a result, major ML conferences such as NeurIPS and ICLR have established "reproducibility challenges" in which researchers are encouraged to replicate the findings of their colleagues [Pineau et al., 2019, 2021].
Recently, RL algorithms have been a crucial component of many ML systems that are being deployed in various application domains. These include but are not limited to, competing with humans in games [Mnih et al., 2013, Silver et al., 2017, Vinyals et al., 2019, , FAIR], creating self-driving cars [Kiran et al., 2021], designing recommendation systems [Afsar et al., 2022], providing e-healthcare services [Yu et al., 2021], and training Large Language Models (LLMs) [Ouyang et al., 2022]. In order to ensure replicability across these systems, an important first step is to develop replicable RL algorithms. To the best of our knowledge, replicability in the context of RL has not received a formal mathematical treatment. We initiate this effort by focusing on infinite horizon, tabular RL with a generative model. The generative model was first studied by Kearns and Singh [1998] in order to understand the statistical complexity of long-term planning without the complication of exploration. The crucial difference between this setting and Dynamic Programming (DP) [Bertsekas, 1976] is that the agent needs to first obtain information about the world before computing a policy through some optimization process. Thus, the main question is to understand the number of samples required to estimate a near-optimal policy. This problem is similar to understanding the number of labeled examples required in PAC learning [Valiant, 1984].
In this work, we study three different formal notions of replicability and design algorithms that satisfy them. First, we study the definition of Impagliazzo et al. [2022], which adapted to the context of RL says that a learning algorithm is replicable if it outputs the exact same policy when executed twice on the same MDP, using shared internal randomness across the two executions (cf. Definition 2.10). We show that there exists a replicable algorithm that outputs a near-optimal policy using  $\widetilde{O}(N^3)$  samples $^2$ , where  $N$  is the cardinality of the state-action space. This algorithm satisfies an additional property we call locally random, which roughly asks that every random decision the algorithm makes based on internal randomness must draw its internal randomness independently from other decisions. Next, we provide a lower bound for deterministic algorithms that matches this upper bound.
Subsequently, we study a less stringent notion of replicability called TV indistinguishability, which was introduced by Kalavasis et al. [2023]. This definition states that, in expectation over the random draws of the input, the TV distance of the two distributions over the outputs of the algorithm should be small (cf. Definition 4.1). We design a computationally efficient TV indistinguishable algorithm for answering  $d$  statistical queries whose sample complexity scales as  $\widetilde{O}(d^2)$ . We remark that this improves the sample complexity of its replicable counterpart based on the rounding trick from Impagliazzo et al. [2022] by a factor of  $d$  and it has applications outside the scope of our work [Impagliazzo et al., 2022, Esfandiari et al., 2023b,a, Bun et al., 2023, Kalavasis et al., 2023]. This algorithm is inspired by the Gaussian mechanism from the Differential Privacy (DP) literature [Dwork et al., 2014]. Building upon this statistical query estimation oracle, we design computationally efficient TV-indistinguishable algorithms for  $Q$ -function estimation and policy estimation whose sample complexity scales as  $\widetilde{O}(N^2)$ . Interestingly, we show that by violating the locally random property and allowing for internal randomness that creates correlations across decisions, we can transform these TV indistinguishable algorithms to replicable ones without hurting their sample complexity, albeit at a cost of  $\widetilde{O}(\exp(N))$  running time. Our transformation is inspired by the main result of Kalavasis et al. [2023]. We also conjecture that the true sample complexity of  $\rho$ -replicable policy estimation is indeed  $\widetilde{\Theta}(N^2)$ .
Finally, we propose a novel relaxation of the previous notions of replicability. Roughly speaking, we say that an algorithm is approximately replicable if, with high probability, when executed twice on the same MDP, it outputs policies that are close under a dissimilarity measure that is based on the Renyi divergence. We remark that this definition does not require sharing the internal randomness across the executions. Finally, we design an RL algorithm that is approximately replicable and outputs a near-optimal policy with  $\widetilde{O}(N)$  sample and time complexity.
Table 1.1 and Table 1.2 summarizes the sample and time complexity of  $Q$ -estimation and policy estimation, respectively, under different notions of replicability. We assume the algorithms in question have a constant probability of success. In Appendix G, we further discuss the benefits and downsides for each of these notions.
For simplicity, we hide the dependence on the remaining parameters of the problem in this section.
Table 1.1: Complexity Overview for  $Q$ -Estimation with Constant Probability of Success.
PROPERTY	SAMPLE COMPLEXITY	TIME COMPLEXITY
LOCALLY RANDOM, REPLICABLE	Θ\(\left(\frac{N^3}{(1-\gamma)^3\varepsilon^2\rho^2}\right)\)	Θ\(\left(\frac{N^3}{(1-\gamma)^3\varepsilon^2\rho^2}\right)\)
TV INDISTINGUIHABLE	O\(\left(\frac{N^2}{(1-\gamma)^3\varepsilon^2\rho^2}\right)\)	O\(\left(\frac{\text{poly}(N)}{(1-\gamma)^3\varepsilon^2\rho^2}\right)\)
REPLICABLE (THROUGH TV INDISTINGUIHABILITY)	O\(\left(\frac{N^2}{(1-\gamma)^3\varepsilon^2\rho^2}\right)\)	O\(\left(\frac{\exp(N)}{(1-\gamma)^3\varepsilon^2\rho^2}\right)\)
Table 1.2: Complexity Overview for Policy Estimation with Constant Probability of Success.
PROPERTY	SAMPLE COMPLEXITY	TIME COMPLEXITY
LOCALLY RANDOM, REPLICABLE	O(N3/(1-γ)5ε2ρ2)	O(N3/(1-γ)5ε2ρ2)
TV INDISTINGUISHABLE	O(N2/(1-γ)5ε2ρ2)	O(poly(N)/(1-γ)5ε2ρ2)
REPLICABLE (THROUGH TV INDISTINGUISHABILITY)	O(N2/(1-γ)5ε2ρ2)	O(exp(N)/(1-γ)5ε2ρ2)
APPROXIMATELY REPLICABLE	O(N/(1-γ)5ε2ρ2)	O(N/(1-γ)5ε2ρ2)
Replicability. Pioneered by Impagliazzo et al. [2022], there has been a growing interest from the learning theory community in studying replicability as an algorithmic property. Esfandiari et al. [2023a,b] studied replicable algorithms in the context of multi-armed bandits and clustering. Recently, Bun et al. [2023] established equivalences between replicability and other notions of algorithmic stability such as differential privacy when the domain of the learning problem is finite and provided some computational and statistical hardness results to obtain these equivalences, under cryptographic assumptions. Subsequently, Kalavasis et al. [2023] proposed a relaxation of the replicability definition of Impagliazzo et al. [2022], showed its statistical equivalence to the notion of replicability for countable domains<sup>3</sup> and extended some of the equivalences from Bun et al. [2023] to countable domains. Chase et al. [2023], Dixon et al. [2023] proposed a notion of list-replicability, where the output of the learner is not necessarily identical across two executions but is limited to a small list of choices.
The closest related work to ours is the concurrent and independent work of Eaton et al. [2023]. They also study a formal notion of replicability in RL which is inspired by the work of Impagliazzo et al. [2022] and coincides with one of the replicability definitions we are studying (cf. Definition 2.8). Their work focuses both on the generative model and the episodic exploration settings. They derive upper bounds on the sample complexity in both settings and validate their results experimentally. On the other hand, our work focuses solely on the setting with the generative model. We obtain similar sample complexity upper bounds for replicable RL algorithms under Definition 2.8 and then we show a lower bound for the class of locally random algorithms. Subsequently, we consider two relaxed notions of replicability which yield improved sample complexities.
Reproducibility in RL. Reproducing, interpreting, and evaluating empirical results in RL can be challenging since there are many sources of randomness in standard benchmark environments. Khetarpal et al. [2018] proposed a framework for evaluating RL to improve reproducibility. Another barrier to reproducibility is the unavailability of code and training details within technical reports. Indeed, Henderson et al. [2018] observed that both intrinsic (e.g. random seeds, environments) and extrinsic (e.g. hyperparameters, codebases) factors can contribute to difficulties in reproducibility. Tian et al. [2019] provided an open-source implementation of AlphaZero [Silver et al., 2017], a popular RL-based Go engine.
<sup>3</sup>We remark that this equivalence for finite domains can also be obtained, implicitly, from the results of Bun et al. [2023].
RL with a Generative Model. The study of RL with a generative model was initiated by Kearns and Singh [1998] who provided algorithms with suboptimal sample complexity in the discount factor  $\gamma$ . A long line of work (see, e.g. Gheshlaghi Azar et al. [2013], Wang [2017], Sidford et al. [2018a,b], Feng et al. [2019], Agarwal et al. [2020], Li et al. [2020] and references therein) has led to (non-replicable) algorithms with minimax optimal sample complexity. Another relevant line of work that culminated with the results of Even-Dar et al. [2002], Mannor and Tsitsiklis [2004] studied the sample complexity of finding an  $\varepsilon$ -optimal arm in the multi-armed bandit setting with access to a generative model.
2.1 Reinforcement Learning Setting
(Discounted) Markov Decision Process. We start by providing the definitions related to the Markov Decision Process (MDP) that we study in this work.
Definition 2.1 (Discounted Markov Decision Process). A (discounted) Markov decision process (MDP) is a 6-tuple  $M = \left( S, s_0, \mathcal{A} = \bigcup_{s \in S} \mathcal{A}^s, P_M, r_M, \gamma \right)$ . Here  $S$  is a finite set of states,  $s_0 \in S$  is the initial state,  $\mathcal{A}^s$  is the finite set of available actions for state  $s \in S$ , and  $P_M(s' \mid s, a)$  is the transition kernel, i.e.,  $\forall (s, s') \in S^2, \forall a \in \mathcal{A}^s, P_M(s' \mid s, a) \geq 0$  and  $\forall s \in S, \forall a \in \mathcal{A}^s, \sum_{s' \in S} P_M(s' \mid s, a) = 1$ . We denote the reward function by  $r_M: S \times \mathcal{A} \to [0,1]$  and the discount factor by  $\gamma \in (0,1)$ . The interaction between the agent and the environment works as follows. At every step, the agent observes a state  $s$  and selects an action  $a \in \mathcal{A}^s$ , yielding an instant reward  $r_M(s, a)$ . The environment then transitions to a random new state  $s' \in S$  drawn according to the distribution  $P_M(\cdot \mid s, a)$ .
Definition 2.2 (Policy). We say that a map  $\pi : S \to \mathcal{A}$  is a (deterministic) stationary policy.
When we consider randomized policies we overload the notation and denote  $\pi(s, a)$  the probability mass that policy  $\pi$  puts on action  $a \in \mathcal{A}^s$  in state  $s \in S$ .
Definition 2.3 (Value  $(V)$  Function). The value  $(V)$  function  $V_{M}^{\pi}:S\to [0,1 / (1 - \gamma)]$  of a policy  $\pi$  with respect to the MDP  $M$  is given by  $V_{M}^{\pi}(s):= \mathbb{E}\left[\sum_{t = 0}^{\infty}\gamma^{t}r_{M}(s_{t},a_{t})\mid s_{0} = s\right]$ . Here  $a_{t}\sim \pi (s_{t})$  and  $s_{t + 1}\sim P_M(\cdot \mid s_t,a_t)$ .
This is the expected discounted cumulative reward of a policy.
Definition 2.4 (Action-Value  $(Q)$  Function). The action-value  $(Q)$  function  $Q_M^\pi : \mathcal{S} \times \mathcal{A} \to [0,1/(1-\gamma)]$  of a policy  $\pi$  with respect to the MDP  $M$  is given by  $Q_M^\pi(s,a) := r_M(s,a) + \gamma \cdot \sum_{s' \in \mathcal{S}} P_M(s'|s,a) \cdot V_M^\pi(s')$ .
We write  $N \coloneqq \sum_{s \in S} |\mathcal{A}^s|$  to denote the number of state-action pairs. We denote by  $\pi^\star$  the optimal policy that maximizes the value function, i.e.,  $\forall \pi, s \in S$ :  $V^\star(s) \coloneqq V^{\pi^\star}(s) \geq V^\pi(s)$ . We also define  $Q^\star(s, a) \coloneqq Q^{\pi^\star}(s, a)$ . This quantity is well defined since the fundamental theorem of RL states that there exists a (deterministic) policy  $\pi^\star$  that simultaneously maximizes  $V^\pi(s)$  among all policies  $\pi$ , for all  $s \in S$  (see e.g. Puterman [2014]).
Since estimating the optimal policy from samples when  $M$  is unknown could be an impossible task, we aim to compute an  $\varepsilon$ -approximately optimal policy for  $M$ .
Definition 2.5 (Approximately Optimal Policy). Let  $\varepsilon \in (0,1)$ . We say that the policy  $\pi$  is  $\varepsilon$ -approximately optimal if  $||V^{\star} - V^{\pi}||_{\infty} \leq \varepsilon$ .
In the above definition,  $||\cdot ||_{\infty}$  denotes the infinity norm of the vector, i.e., its maximum element in absolute value.
Generative Model. Throughout this work, we assume we have access to a generative model (first studied in Kearns and Singh [1998]) or a sampler  $G_{M}$ , which takes as input a state-action pair  $(s,a)$  and provides a sample  $s^{\prime}\sim P_{M}(\cdot \mid s,a)$ . This widely studied fundamental RL setting allows us to focus on the sample complexity of planning over a long horizon without considering the additional complications of exploration. Since our focus throughout this paper is on the statistical complexity of
<sup>4</sup>We assume that the reward is deterministic and known to the learner. Our results hold for stochastic and unknown rewards with an extra (replicable) estimation step, which does not increase the overall sample complexity.
the problem, our goal is to achieve the desired algorithmic performance while minimizing the number of samples from the generator that the algorithm requires.
Approximately Optimal Policy Estimator. We now define what it means for an algorithm  $\mathcal{A}$  to be an approximately optimal policy estimator.
Definition 2.6  $((\varepsilon, \delta)$ -Optimal Policy Estimator). Let  $\varepsilon, \delta \in (0,1)^2$ . A (randomized) algorithm  $\mathcal{A}$  is called an  $(\varepsilon, \delta)$ -optimal policy estimator if there exists a number  $n := n(\varepsilon, \delta) \in \mathbb{N}$  such that, for any MDP  $M$ , when it is given at least  $n(\varepsilon, \delta)$  samples from the generator  $G_M$ , it outputs a policy  $\hat{\pi}$  such that  $\left\| V^{\hat{\pi}} - V^{\star} \right\|_{\infty} \leq \varepsilon$  with probability at least  $1 - \delta$ . Here, the probability is over random draws from  $G_M$  and the internal randomness of  $\mathcal{A}$ .
Approximately optimal  $V$ -function estimators and  $Q$ -function estimators are defined similarly.
Remark 2.7. In order to allow flexibility to the algorithm, we do not restrict it to request the same amount of samples for every state-action pair. Thus  $n(\varepsilon, \delta)$  is a bound on the total number of samples that  $\mathcal{A}$  receives from  $G_{M}$ . The algorithms we design request the same number of samples for every state-action pair, however, our lower bounds are stronger and hold without this restriction.
When the MDP  $M$  is clear from context, we omit the subscript in all the previous quantities.
2.2 Replicability
Definition 2.8 (Replicable Algorithm; [Impagliazzo et al., 2022]). Let  $\mathcal{A}:\mathcal{T}^n\to \mathcal{O}$  be an  $n$  sample randomized algorithm that takes as input elements from some domain  $\mathcal{I}$  and maps them to some co-domain  $\mathcal{O}$ . Let  $\mathcal{R}$  denote the internal distribution over binary strings that  $\mathcal{A}$  uses. For  $\rho \in (0,1)$ , we say that  $\mathcal{A}$  is  $\rho$ -replicable if for any distribution  $\mathcal{D}$  over  $\mathcal{I}$  it holds that  $\mathbb{P}_{\bar{S},\bar{S} '\sim \mathcal{D}^n,\bar{r}\sim \mathcal{R}}\bigl \{\mathcal{A}(\bar{S};\bar{r}) = \mathcal{A}(\bar{S}';\bar{r})\bigr \} \geq 1 - \rho$ , where  $\mathcal{A}(\bar{S};\bar{r})$  denotes the (deterministic) output of  $\mathcal{A}$  when its input is  $\bar{S}$  and the realization of the internal random string is  $\bar{r}$ .
In the context of our work, we should think of  $\mathcal{A}$  as a randomized mapping that receives samples from the generator  $G$  and outputs policies. Thus, even when  $\bar{S}$  is fixed,  $\mathcal{A}(\bar{S})$  should be thought of as a random variable, whereas  $\mathcal{A}(\bar{S};\bar{r})$  is the realization of this variable given the (fixed)  $\bar{S},\bar{r}$ . We should think of  $\bar{r}$  as the shared randomness between the two executions, which can be implemented as a shared random seed.
One of the most elementary statistical operations we may wish to make replicable is mean estimation. This operation can be phrased using the language of statistical queries.
Definition 2.9 (Statistical Query Oracle; [Kearns, 1998]). Let  $\mathcal{D}$  be a distribution over the domain  $\mathcal{X}$  and  $\phi : \mathcal{X}^n \to \mathbb{R}$  be a statistical query with true value  $v^\star \coloneqq \lim_{n\to \infty}\phi(X_1,\ldots,X_n) \in \mathbb{R}$ . Here  $X_i \sim_{i.i.d.} \mathcal{D}$  and the convergence is understood in probability or distribution. Let  $\varepsilon, \delta \in (0,1)^2$ . A statistical query (SQ) oracle outputs a value  $v$  such that  $|v - v^\star| \leq \varepsilon$  with probability at least  $1 - \delta$ .
The simplest example of a statistical query is the sample mean  $\phi(X_1, \ldots, X_n) = \frac{1}{n} \sum_{i=1}^{n} X_i$ . Impagliazzo et al. [2022] designed a replicable SQ-query oracle for sample mean queries with bounded co-domain (cf. Theorem C.1).
The following definition is the formal instantiation of Definition 2.8 in the setting we are studying.
Definition 2.10 (Replicable Policy Estimator). Let  $\rho \in (0,1)$ . A policy estimator  $\mathcal{A}$  that receives samples from a generator  $G$  and returns a policy  $\pi$  using internal randomness  $\mathcal{R}$  is  $\rho$ -replicable if for any MDP  $M$ , when two sequences of samples  $\bar{S}, \bar{S}'$  are generated independently from  $G$ , it holds that  $\mathbb{P}_{\bar{S},\bar{S}'\sim G,\bar{r}\sim \mathcal{R}}\left\{\mathcal{A}(\bar{S};\bar{r}) = \mathcal{A}(\bar{S}';\bar{r})\right\} \geq 1 - \rho$ .
To give the reader some intuition about the type of problems for which replicable algorithms under Definition 2.8 exist, we consider the fundamental task of estimating the mean of a random variable. Impagliazzo et al. [2022] provided a replicable mean estimation algorithm when the variable is bounded (cf. Theorem C.1). Esfandiari et al. [2023b] generalized the result to simultaneously estimate the means of multiple random variables with unbounded co-domain under some regularity conditions on their distributions (cf. Theorem C.2). The idea behind both results is to use a rounding trick introduced in Impagliazzo et al. [2022] which allows one to sacrifice some accuracy of the estimator in favor of the replicability property. The formal statement of both results, which are useful for our work, are deferred to Appendix C.1.
3 Replicable  $Q$ -Function & Policy Estimation
Our aim in this section is to understand the sample complexity overhead that the replicability property imposes on the task of computing an  $(\varepsilon, \delta)$ -approximately optimal policy. Without this requirement, Sidford et al. [2018a], Agarwal et al. [2020], Li et al. [2020] showed that  $\widetilde{O}(N \log(1/\delta) / [(1 - \gamma)^3 \varepsilon^2])$  samples suffice to estimate such a policy, value function, and  $Q$ -function. Moreover, since Gheshlaghi Azar et al. [2013] provided matching lower bounds<sup>5</sup>, the sample complexity for this problem has been settled. Our main results in this section are tight sample complexity bounds for locally random  $\rho$ -replicable  $(\varepsilon, \delta)$ -approximately optimal  $Q$ -function estimation as well as upper and lower bounds for  $\rho$ -replicable  $(\varepsilon, \delta)$ -approximately policy estimation that differ by a factor of  $1/(1 - \gamma)^2$ . The missing proofs for this section can be found in Appendix D.
We remark that in both the presented algorithms and lower bounds, we assume local randomness. For example, we assume that the internal randomness is drawn independently for each state-action pair for replicable  $Q$ -estimation. In the case where we allow for the internal randomness to be correlated across estimated quantities, we present an algorithm that overcomes our present lower bound in Section 4.3. However, the running time of this algorithm is exponential in  $N$ .
3.1 Computationally Efficient Upper Bound on the Sample Complexity