SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Eric Eaton
eeaton@seas.upenn.edu
Michael Kearns
mkearns@cis.upenn.edu
MarcelHussing
mhussing@seas.upenn.edu
Jessica Sorrell
jsorrell@seas.upenn.edu
The replicability crisis in the social, behavioral, and data sciences has led to the formulation of algorithm frameworks for replicability — i.e., a requirement that an algorithm produce identical outputs (with high probability) when run on two different samples from the same underlying distribution. While still in its infancy, provably replicable algorithms have been developed for many fundamental tasks in machine learning and statistics, including statistical query learning, the heavy hitters problem, and distribution testing. In this work we initiate the study of replicable reinforcement learning, providing a provably replicable algorithm for parallel value iteration, and a provably replicable version of R-max in the episodic setting. These are the first formal replicability results for control problems, which present different challenges for replication than batch learning settings.
The growing prominence of machine learning (ML) and its widespread adoption across industries underscore the need for replicable research [Wagstaff, 2012, Pineau et al., 2021]. Many scientific fields have suffered from this same inability to reproduce the results of published studies [Begley and Ellis, 2012]. Replicability in ML requires not only the ability to reproduce published results [Wagstaff, 2012], as may be partially addressed by sharing code and data [Stodden et al., 2014], but also consistency in the results obtained from successive deployments of an ML algorithm in the same environment. However, the inherent variability and randomness present in ML pose challenges to achieving replicability, as these factors may cause significant variations in results.
Building upon foundations of algorithmic stability [Bousquet and Elisseeff, 2002], recent work in learning theory has established rigorous definitions for the study of supervised learning [Impagliazzo et al., 2022] and bandit algorithms [Esfandiari et al., 2023a] that are provably replicable, meaning that algorithms produce identical outputs (with high probability) when executed on distinct data samples from the same underlying distribution. However, these results have not been extended to the study of control problems such as reinforcement learning (RL), that have long been known to suffer from stability issues [White and Eldeib, 1994, Mannor et al., 2004, Islam et al., 2017, Henderson et al., 2018]. These stability issues have already sparked research into robustness for control problems including RL [Khalil et al., 1996, Nilim and Ghaoui, 2005, Iyengar, 2005]. Non-deterministic environments and evaluation benchmarks, the randomness of the exploration process, and the sequential interaction of an RL agent with the environment all complicate the ability to make RL replicable. Our work is orthogonal to that of the robustness literature and our goal is not to reduce
the effect of these inherent characteristics, such as by decreasing the amount of exploration that an agent performs, but to develop replicable RL algorithms that support these characteristics.
Toward this goal, we initiate the study of replicable RL and develop the first set of RL algorithms that are provably replicable. We contend that the fundamental theoretical study of replicability in RL might advance our understanding of the aspects of RL algorithms that make replicability hard. In this work, we put on a similar lens as Impagliazzo et al. [2022] and consider replicability as an algorithmic property that can be achieved simultaneously with exploration and exploitation. First, we show that it is possible to obtain a near-optimal, replicable policy given sufficiently many samples from every state in the environment. This notion is then naturally extended to replicable exploration.
Our contributions can be summarized as follows. We provide two novel and efficient algorithms to
- show that stochastic, sample-based value iteration can be done replicably and
- replicably explore the space of an MDP while also finding an optimal policy.
We experimentally validate that our algorithms require much fewer samples than theory suggests.
We consider the problem of solving a discounted Markov decision process (MDP)  $\mathcal{M} = \{\mathcal{S},\mathcal{A},R,P,\gamma ,\mu \}$  with state space  $\mathcal{S}$ , action space  $\mathcal{A}$ , reward function  $R$ , transition kernel  $P$ , discount factor  $\gamma$ , and initial state distribution  $\mu$ . We assume that the size of the state space  $|S|$  and number of possible actions  $|\mathcal{A}|$  are finite and not too large. Further, we assume that the rewards for every state-action pair are deterministic, bounded, and known. Relaxing assumptions on the reward function might not necessarily seem straightforward in our goal of replicable RL, as the stochastic reward would need to be made replicable. However, the case can be handled by our algorithms with minor modifications and only constant factor overhead. The goal is to find a policy  $\pi : S \mapsto \mathcal{A}$  that maximizes the cumulative discounted reward  $J_{h} = \sum_{k=h}^{\infty} \gamma^{k-h} R_{k}(s,a)$ . We use the typical definitions of the value and Q-value functions for the expected cumulative discounted return from a state or state-action pair, respectively:
$$
V _ {\pi} (s) = \underset {\pi , P} {\mathbb {E}} \left[ J _ {h} | s _ {h} = s \right] \quad Q _ {\pi} (s, a) = \underset {\pi , P} {\mathbb {E}} \left[ J _ {h} | s _ {h} = s, a _ {h} = a \right].
$$
To show the various difficulties that come from trying to achieve replicability in RL, we consider two different settings to examine various components of the problem.
Parallel sampling setting First, we ask whether it is even possible to obtain a replicable policy from empirical samples without considering the challenges of exploration. For this, we can adopt the setting of generative models  $G_{\mathcal{M}}$ , or more precisely, the parallel sampling setting. In the parallel sampling model, first introduced by Kearns and Singh [1998a], one has access to a parallel generative sampling subroutine  $\mathbf{PS}(G_{\mathcal{M}})$ . A single call to  $\mathbf{PS}(G_{\mathcal{M}})$  will return, for every state-action pair  $(s,a)\in S\times \mathcal{A}$ , a randomly sampled next state  $s^{\prime}\in S$  drawn from  $P(s^{\prime}|s,a)$ . The key advantage is that this model separates learning from the quality of the exploration procedure.
Definition 2.1 (Generative model). Let  $\mathcal{M}$  denote an arbitrary MDP. Then a generative model  $G_{\mathcal{M}}((s,a))$  is a randomized algorithm that, given a state-action pair  $(s,a)\in S\times \mathcal{A}$ , outputs a deterministic reward  $R(s,a)$  and a next state  $s^{\prime}$  sampled from  $P(\cdot |s,a)$ .
Definition 2.2 (Parallel sampling). Let  $\mathcal{M}$  denote an arbitrary MDP. Then a call to the parallel sampling subroutine  $\mathbf{PS}(G_{\mathcal{M}})$  returns exactly one sample  $s_i' \sim G_{\mathcal{M}}((s_i, a_i))$  for every state-action pair  $(s_i, a_i)$  in  $S \times \mathcal{A}$  of  $\mathcal{M}$  using a generative model.
Episodic setting The second setting we consider is one in which an algorithm does have to explore the MDP before it can obtain an optimal policy. More precisely, we consider an episodic setting where, in every episode  $e \in \{1, 2, \dots, E\}$ , the agent starts in a position  $s_0 \sim \mu$  and interacts with the environment for a fixed amount of time  $H$ . At any step  $h \in [1, H]$ , the agent is in some state  $s_h$ , selects an action  $a_h$ , receives a reward  $r_h$  and transitions to a new state  $s_{h+1}$ . Gathering a trajectory  $\tau = (s_0, a_0, r_0, \dots, s_H, a_H, r_H)$  of states, actions and rewards under policy  $\pi$  can be thought of as a draw from a distribution  $\tau \sim P_{\mathcal{M}}^{\pi}(\tau)$ . We will omit the sub-and superscripts when clear from context. For consistency with the remaining analysis, we work with a  $\gamma$ -discounted version of the problem.
We build on the recent framework by Impagliazzo et al. [2022], which considers replicability as a property of randomized algorithms that take as input a dataset sampled i.i.d. from an arbitrary distribution. They consider an algorithm to be replicable if, on two runs in which its internal randomness is fixed and its input data is resampled, it outputs the same result with high probability:
Definition 2.3 (Replicability). Fix a domain  $\mathcal{X}$  and target replicability parameter  $\rho \in (0,1)$ . A randomized algorithm  $\mathcal{A}:\mathcal{X}^n\to \mathcal{Y}$  is  $\rho$ -replicable if for all distributions  $D$  over  $\mathcal{X}$ , randomizing over the internal randomness  $r$  of  $\mathcal{A}$  and choice of samples  $S_{1},S_{2}$ , each of size  $n$  drawn i.i.d. from  $D$ , we have:  $\mathbf{Pr}_{S_1,S_2,r}[\mathcal{A}(S_1;r)\neq \mathcal{A}(S_2;r)]\leq \rho$ .
Several key tools that were introduced by Impagliazzo et al. [2022] will prove useful or yield inspiration for the algorithms developed in this work. One of the key observations is that many of the computations in RL can be phrased as statistical queries, defined as follows:
Definition 2.4 (Statistical query, [Kearns, 1998]). Fix a distribution  $D$  over  $\mathcal{X}$  and an accuracy parameter  $\alpha \in (0,1)$ . A statistical query is a function  $\phi : \mathcal{X} \to [0,1]$ , and a mechanism  $M$  answers  $\phi$  with tolerance  $\alpha$  on distribution  $D$  if  $a \gets M$  satisfies  $a \in [\mathbb{E}_{x \sim D}[\phi(x)] \pm \alpha]$ .
We will make direct use of the replicable algorithm for answering statistical queries by Impagliazzo et al. [2022] which will be useful to obtain replicable estimates of various measurements such as transition probabilities. We will refer to the replicable statistical query procedure as rSTAT. We note that Impagliazzo et al. [2022] also proves a lower-bound on the sample complexity required for replicable statistical queries, showing that the results below are essentially tight.
Theorem 2.1 (Replicable statistical queries, Impagliazzo et al. [2022]). There is a  $\rho$ -replicable algorithm rSTAT such that for any distribution  $D$  over  $\mathcal{X}$ , replicability parameter  $\rho \in (0,1)$ , accuracy parameter  $\alpha \in (0,1)$ , failure parameter  $\delta \in O(\rho)$ , and query  $\phi : \mathcal{X} \to [0,1]$ , letting  $S$  be a sample of  $n \in O\left(\frac{\log(1 / \delta)}{(\rho - 2\delta)^2\alpha^2}\right)$  elements drawn i.i.d. from  $D$ , we have that  $a \gets r\mathrm{STAT}_{\alpha,\rho}(S,\phi)$  satisfies  $a \in [\mathbb{E}_{x \sim D}[\phi(x)] \pm \alpha]$  except with probability at most  $\delta$  over the samples  $S$ .
At a very high level, rSTAT uses its sample to empirically estimate the expected value of the statistical query on the target distribution. It then uses its internal randomness to pick an evenly-spaced set of canonical representatives from the  $[0,1]$  interval, and returns whichever canonical representative is closest to the empirical estimate. We note that the algorithm of Impagliazzo et al. [2022] for replicably answering statistical queries is not only sample efficient, but also computationally efficient, as it has runtime polynomial in  $1 / \alpha$ ,  $1 / \rho$ , and  $\log(1 / \delta)$ .
3 Replicable reinforcement learning
To define replicability for the RL setting, we can adapt Definition 2.3 more or less exactly. The question that arises is which of the many RL objects should be made replicable? We separate the difficulty of replicability into three levels: replicability of the MDP, the value function, and the policy. Since these objects carry different amounts of information [Farahmand, 2011], the following relationships can be established.
If we are able to replicably (and accurately) estimate an MDP, we can always replicably compute an (optimal) value function using standard techniques on our estimates, and from replicable value functions we can obtain the corresponding policies. Note that the inverse is not true as we lose information when going from MDP to value function and then policy. As a result, we expect that replicable estimation of MDPs is the hardest setting in stochastic RL, followed by replicable value function, and then policy estimation.
For replicability of control problems, a sensible measure to ask for is the production of identical policies, which are the ultimate object of primary interest. We would at least like to ensure that with high probability, we can obtain identical optimal policies across two runs of our RL procedures:
Definition 3.1 (Replicable policy estimation). Let  $\mathcal{A}$  be a policy estimation algorithm that outputs a policy  $\widehat{\pi}^{*}:S\mapsto \mathcal{A}$  given a set of trajectories  $S$  sampled from an MDP. Algorithm  $\mathcal{A}$  is  $\rho$ -replicable if, given independently sampled trajectory sets  $S_{1}$  and  $S_{2}$ , and yielding policies  $\widehat{\pi}_{1}^{*}$  and  $\widehat{\pi}_{2}^{*}$ , it holds
Figure 1: The GridWorld for our experiments (left) and two different policies that were generated by the Phased Q-learning Algorithm on this gridworld (center and right). Following the first policy (center) more likely reaches the left goal while following the right policy more likely reaches the right goal. All states except the goals have 0 reward. The actions are up, down, left and right; there is a  $30\%$  chance that after choosing an action the agent moves left or right of the target direction.
for all states  $s \in S$  and actions  $a \in \mathcal{A}$  that
$$
\boldsymbol {P r} _ {S _ {1}, S _ {2}, r} \left[ \hat {\pi} ^ {* (1)} (a | s) \neq \hat {\pi} ^ {* (2)} (a | s) \right] \leq \rho
$$
$$
s. t. \widehat {\pi} ^ {* (1)} (a | s) \leftarrow \mathcal {A} (S _ {1}; r) \quad \wedge \quad \widehat {\pi} ^ {* (2)} (a | s) \leftarrow \mathcal {A} (S _ {2}; r),
$$
where  $r$  represents the internal randomness of  $\mathcal{A}$ . Trajectory sets  $S_{1}$  and  $S_{2}$  may potentially be gathered from the environment during the execution of an RL algorithm.
While this definition is the weakest we would like to achieve, the results we present in this paper provide stronger guarantees. Our Replicable Phased Value Iteration builds on [Kearns and Singh, 1998a] and ensures replicability of value functions, while our Replicable Episodic R-max follows [Kearns and Singh, 1998b, Brafman and Tennenholtz, 2003] and provides replicability of full MDPs. Equivalent formal definitions for replicable value and MDP estimation are given in Appendix A.
Current algorithms for sample-based RL problems will struggle to satisfy Definition 3.1 of replicability and output different policies even in simple environments (see Figure 1). In some cases, this may not be problematic since the resulting policies will still be  $\varepsilon$ -optimal, but in practice it is often hard to tell when that is the case. Fixing replicability will support the identification of problematic solutions and encourage procedures that yield more stable solutions in the long run. Varying policies can, for example, arise from sample uncertainty, insufficient state-space coverage, or differing exploration. In order to achieve replicability, all of the aforementioned challenges need to be addressed, which makes for an intricate but interesting problem. With this in mind, the next section will introduce a first set of formally replicable algorithms that separate out some of these challenges.
4.1 Replicable phased value iteration
The first question we answer positively is whether it is even possible to achieve replicability when the samples are drawn i.i.d. from the same distribution. For this, we use the parallel sampling model described in section 2. This model is well-suited for the task as it allows us to analyze sample-based value iteration independent of the exploration policy that collects the samples.
We provide a replicable version of indirect Phased Q-learning [Kearns and Singh, 1998a], which was later also referred to as Phased Value Iteration [Kakade, 2003]. In brief, the algorithm iterates  $T$  times and at every iteration makes  $m$  calls to  $\mathbf{PS}(G_{\mathcal{M}})$ , computes an approximate value estimate for every state and does one round of value updates. Kearns and Singh [1998a] provide the following Lemma 4.1 to show the optimality of the original procedure.
Lemma 4.1 (Phased Q-learning convergence, [Kearns and Singh, 1998a]). Suppose the number of calls to  $\mathbf{PS}(G_{\mathcal{M}})$  is chosen such that the value function estimates produced in every round by Phased  $Q$ -learning are sufficiently accurate. For any MDP  $\mathcal{M}$ , Phased  $Q$ -learning converges to a policy  $\widehat{\pi}^*$  whose return is within  $\varepsilon$  of the optimal policy  $\pi^*$ .
Our algorithm operates similarly, but we would like to achieve replicability on top of optimality. We use a randomized rounding procedure for statistical query estimation (rSTAT) provided by Impagliazzo et al. [2022] to compute the value estimates at every iteration. For this, we assume that the value function is normalized to the interval  $[0,1]$ . A detailed description of our algorithm is
provided in Algorithm 1. The Replicable Phased Value Iteration (rPVI) algorithm we provide satisfies Definition 3.1 and produces  $\varepsilon$ -optimal policies. It goes even one step further and produces not only replicable policies but replicable value functions. This is formalized in the following Theorem 4.1.
Theorem 4.1. Let  $\varepsilon \in (0,1)$  be the accuracy and  $\rho \in (0,1)$  be the replicability parameter. Let  $\delta \in (0,1)$  be the sample failure probability. Set the number of calls to  $\mathbf{PS}(G_{\mathcal{M}})$  at every iteration to
$$
m = O \left(\frac {\log^ {2} (1 / \varepsilon) | \mathcal {S} | ^ {2} | \mathcal {A} | ^ {2}}{\varepsilon^ {2} (\rho - 2 \delta) ^ {2}} \log \left(\frac {| \mathcal {S} | | \mathcal {A} |}{\delta} + \log \log (1 / \varepsilon)\right)\right)
$$
where  $O$  suppresses the dependence on  $\gamma$ . In two runs (1) and (2) with shared internal randomness, Algorithm 1 produces identical policies, s.t.  $\mathbf{Pr}[\widehat{\pi}^{*(1)} \neq \widehat{\pi}^{*(2)}] \in O(\rho)$ . In every run, the produced policies  $\widehat{\pi}^*$  achieve return at most  $\varepsilon$  less than the optimal policy  $\pi^*$  with all but probability  $O(\delta)$ .
Proof Sketch. We give a sketch for the proof of the theorem here and refer the reader to a full proof in Appendix B.2. Assume that we can get replicable and accurate estimates of the value function expectations from our rSTAT procedure. One can show by induction that the algorithm consistently produces the same value functions in every iteration. Lemma 4.2 guarantees the convergence to an optimal policy. Finally, we can use union and Chernoff bounds to pick a sufficiently large sample for our rSTAT queries to be replicable and accurate and satisfy our assumption.
An interesting observation is that rPVI discretizes the space of values as a function of the  $\varepsilon$ -parameter and  $\gamma$  (see Appendix B.2). As a result, replicability becomes harder for larger values of  $\gamma$  as discretization intervals become smaller and we require more samples to obtain an equally sized  $\rho$ . This is intuitive as we need to account for more potential future states that might impact our estimates.
The number of samples to compute a replicable value function is at most  $O(\log^2 (1 / \epsilon)|\mathcal{S}|^2 |\mathcal{A}|^2 /\rho^2)$  times larger than computing a non-replicable one [Kearns and Singh, 1998a]. Still, a key observation of the original Phased Q-learning result was that it is sufficient for every state-action pair to have a sample size logarithmic in  $|S||\mathcal{A}|$ , making the procedure cheaper than estimating the full transition dynamics of an MDP. The cost of replicability is the loss of this property. However, we note that rPVl does not yield replicable transition probability estimation. Using the idea of rSTAT queries to obtain transition estimates turns out to be significantly more expensive than the replicable value estimation done by Algorithm 1 (see Appendix B.2.1). Our results retain the notion that direct value estimation is much cheaper than estimating the full transition kernel even in the presence of replicability.
4.2 Replicable RL with exploration
Next, we consider the setting of episodic exploration. We show that, despite the stochastic nature of exploration, it is possible to guarantee replicability while still outputting an  $\varepsilon$ -optimal policy.
We take the R-max algorithm of Brafman and Tennenholtz [2003] as the starting point for our replicable algorithm RepRMAX (Algorithm 2). It proceeds in rounds where the agent interacts with
Algorithm 1 Replicable Phased Value Iteration (rPVI)
Parameters: accuracy  $\varepsilon$ , failure probability  $\delta$ , replicability failure probability  $\rho$
Input: Generative Model  $G_{\mathcal{M}}$
Output:  $\varepsilon$ -optimal policy  $\widehat{\pi}^{*}$
Initialize  $\widehat{Q}_0(s,a)$  to 0 for all  $(s,a)\in S\times \mathcal{A}$
For all  $s \in S$ , let  $\phi_Q(s) \coloneqq \max_a Q(s, a)$
$$
S \leftarrow (\mathbf {P S} (G _ {\mathcal {M}})) ^ {m}
$$
$\triangleright$  do  $m$  calls to  $\mathbf{PS}(G_{\mathcal{M}})$  and store next-states in a map from state-action pairs  $(s,a)$  to next states  $S[(s,a)]$
for  $(s,a)\in S\times \mathcal{A}$  do
$$
\widehat {V} (s ^ {\prime}) \leftarrow \mathrm {r S T A T} \left(S [ (s, a) ], \phi_ {\widehat {Q} _ {t}} (s ^ {\prime})) \right.
$$
$$
\widehat {Q} _ {t + 1} (s, a) \leftarrow R (s, a) + \gamma \widehat {V} (s ^ {\prime})
$$
$$
\text {r e t u r n} \hat {\pi} ^ {*} = \arg \max  _ {a} \widehat {Q} _ {T} (s, a)
$$
Algorithm 2 Replicable Episodic R-max (RepRMAX)
Parameters: Accuracy  $\varepsilon$ , accuracy failure probability  $\delta$ , replicability failure probability  $\rho$ , horizon  $H$   
Input: MDP  $\mathcal{M}$ , maximum reward  $R_{\mathrm{max}}$
Output:  $\varepsilon$ -optimal policy  $\pi_{\hat{\mathcal{M}}_K}$
Initialize  $\pi_{\hat{\mathcal{M}}_K}$  to a random policy, counters for state-visitation  $n(s,a)$  to 0   
Initialize  $K$  , the set collecting known state-action pairs, to the empty set  $\emptyset$    
Initialize  $S$  , the set collecting trajectories to be used for estimating transition probabilities, to  $\emptyset$    
Initialize  $\widehat{\mathcal{M}}_K$  as  $\widehat{P}_K(s'|s,a)\coloneqq \mathbb{1}[s' = s]$  for all  $(s,a,s')$  and  $\widehat{R}_K(s,a)\coloneqq R_{\mathrm{max}}$  for all  $(s,a)$ $i = 1$    
while  $\pi_{\hat{\mathcal{M}}_K}$  is not  $\varepsilon$  -optimal do Collect a sample of trajectories  $S_{i}\gets P(\tau)^{m}$  and add  $S_{i}$  to  $S$ $K_{i}\gets \mathrm{RepUpdateK}(S_{i},K,\{n(s,a)\}_{(s,a)\in S\times A})$  , identify new known states For all  $(s,a)\in K_i$  , let  $S[(s,a)]$  be the multiset of  $s^\prime$  visited from  $(s,a)$  for all  $\tau \in S$  For all  $s^{\prime}\in S$  , let  $\phi_{s^{\prime}}(s)\coloneqq \mathbb{1}[s = s^{\prime}]$  Update  $\widehat{\mathcal{M}}_K$  for all  $(s,a)\in K_i\colon \widehat{P}_K(s'|s,a)\coloneqq \mathrm{rSTAT}(S[(s,a]),\phi_{s'})$ $\widehat{R}_K(s,a)\coloneqq R(s,a)$ $K = K\cup K_{i}$  Compute  $\pi_{\hat{\mathcal{M}}_K}$  from  $\widehat{\mathcal{M}}_K$    
end while   
return  $\pi_{\hat{\mathcal{M}}_K}$