SlowGuess/NeurIPS_2023_OCR huggingface dataset:
the environment for multiple episodes. The collection of trajectories encountered during exploration is used to incrementally build a model  $\widehat{\mathcal{M}}$  of the underlying MDP  $\mathcal{M}$ . The algorithm implicitly partitions the set of state-action pairs  $\mathcal{S} \times \mathcal{A}$  into two groups: known and unknown. All  $(s, a) \in S \times \mathcal{A}$  are initialized to be unknown. While a state is unknown, the model  $\widehat{\mathcal{M}}$  maintains that  $(s, a)$  is a self-loop with probability 1, and that  $(s, a)$  has maximum reward, thereby promoting exploration of unknown states. After a state-action pair  $(s, a)$  has been visited sufficiently many times, it is added to the collection of known states  $K$  and its transition probabilities  $\widehat{P}$  and reward  $\widehat{R}$  are updated with an empirical approximation of  $\widehat{P}_K(s' \mid s, a)$  for all  $s' \in S$  and the observed reward  $R$ , respectively. After every update, the policy  $\pi_{\widehat{\mathcal{M}}_K}$  is computed as the optimal policy of the current model estimate.
While convergence of Algorithm 2 to an  $\varepsilon$ -optimal policy follows from familiar arguments [Brafman and Tennenholtz, 2003], proving replicability will require a great deal of additional care. To ensure that two runs of RepRMAX (with shared internal randomness) converge to the same policy with high probability, we will show something even stronger: we prove that two such runs will with high probability perform the same sequence of updates to their respective models  $\widehat{\mathcal{M}}_K$  and policies  $\pi_{\hat{\mathcal{M}}_K}$ .
To enforce this property, we introduce a sub-routine in Algorithm 3 which replicably identifies state-action pairs that should be added to the collection of known states. Guaranteeing that at each iteration the set of known states  $K$  will be the same for two independent runs of the algorithm helps ensure that the models of the MDP  $\widehat{\mathcal{M}}_K$ , and consequently the policies  $\pi_{\hat{\mathcal{M}}_K}$  learned at each iteration, will also be identical. To provide replicability, we will want to avoid using a fixed threshold for the number of times a state-action pair  $(s,a)$  must be visited before it is considered "known". Under small deviations in realized transitions, a fixed threshold might lead to some  $(s,a)$  becoming known in one run of the algorithm and not another. Instead, we use a randomized threshold.
In a call to Algorithm 3, the sample drawn at that round is used to estimate the expected number of visits to  $(s,a)$  in a single trajectory, for every  $(s,a)$ . This estimate is added to the count  $n(s,a)$ , which maintains the sum, over all iterations thus far, of the estimated expected visits to  $(s,a)$  from a single trajectory of the policy  $\pi_{\hat{\mathcal{M}}_K}$  at that iteration. A new threshold  $k'$  is then sampled uniformly from  $[k,k + w]$ . If  $n(s,a) \geq k'$ , it is added to the set of known states  $K$ . From standard concentration arguments, we know that for two runs of Algorithm 2 with independent samples, the estimates of the total number of expected visits  $n(s,a)$  will both be close to the true total number of expected visits, and therefore close to each other. Algorithm 3 will only make different decisions about adding an  $(s,a)$  pair to the set of known states if the threshold  $k'$  is chosen to fall between the two estimated values  $n(s,a)$  from the two runs. Here, the concentration of  $n(s,a)$  and the fact that  $k'$  is randomized allows us to bound the probability that the threshold  $k'$  is chosen to fall between the different  $n(s,a)$
values. We show in Theorem 4.2 that so long as the sample size  $m$  that is used to estimate expected visits, and the window  $w$  from which the randomized threshold in sampled, are taken to be large enough, the update to the set of known states at each round will be replicable.
Algorithm 3 RepUpdateK
Parameters: Accuracy failure probability  $\delta$ , replicability failure probability  $\rho$
Input: Sample of trajectories  $S_{i}$ , set of known states  $K$ , set of state-visit counts  $\{n(s,a)\}_{(s,a) \in S \times A}$
Output: List of new known state-action pairs  $K_{i}$
$$
K _ {i} = \{(s, a): (s, a) \in \mathcal {S} \times \mathcal {A} \text {a n d} (s, a) \notin K \}
$$
$$
k ^ {\prime} \leftarrow \mathcal {U} [ k, k + w ]
$$
$$
\mathbf {f o r} (s, a) \in K _ {i} \mathbf {d o}
$$
$$
\widehat {c} _ {s, a} = \frac {1}{| S _ {i} |} \sum_ {\tau \in S _ {i}} \sum_ {h = 1} ^ {H} \mathbb {1} \left[ \left(s _ {h}, a _ {h}\right) = (s, a) \right]
$$
$$
n (s, a) = n (s, a) + \widehat {c} _ {s, a}
$$
$$
\text {i f} n (s, a) <   k ^ {\prime} \text {t h e n}
$$
Remove  $(s,a)$  from  $K_{i}$
$$
e n d i f
$$
$$
e n d \mathrm {f o r}
$$
$$
\text {r e t u r n} K _ {i}
$$
Now that we have understood the intricacies on an intuitive level, we will prove convergence (Lemma 4.2) and replicability (Theorem 4.2) of Algorithm 2.
Lemma 4.2 (Convergence). Consider  $\mathcal{A}$  to be Algorithm 4. Let  $\varepsilon \in (0,1)$  be the accuracy parameter,  $\rho \in (0,1)$  the replicability parameter, and  $\delta \in (0,1)$ , be the sample failure probability, with  $\delta < \rho /4$ . Let  $T\in \Theta (\frac{H|S||\mathcal{A}|}{\varepsilon} +\frac{H^2\log(1 / \delta)}{\varepsilon^2})$  be a bound on the number of iterations of Algorithm 2. Suppose  $1 - \gamma >\frac{\sqrt{\varepsilon}}{H|\mathcal{A}|}$  and let  $m\in \tilde{O}\left(\frac{|S|^2|\mathcal{A}|^2T^4\log(1 / \rho)}{\rho^2}\right)$  be the number of trajectories per iteration. Let  $k = H$  be the lowest expected visit count of a state-action pair before it is known. Let  $w\in O(k)$  define the window  $[k,k + w]$  for sampling the randomized threshold  $k'$ . Then with all but probability  $\delta$ , after  $T$  iterations,  $\mathcal{A}$  yields an  $\varepsilon$ -optimal policy.
The proof of convergence is similar to those found in [Kearns and Singh, 1998b] and [Brafman and Tennenholtz, 2003], so we defer the proof to Appendix B.3. We continue here with the final theorem statement that summarizes the properties of our RepRMAX algorithm.
Theorem 4.2. Let parameters be set as in Lemma 4.2. Then with all but probability  $\delta$ ,  $\mathcal{A}$  converges to an  $\varepsilon$ -optimal policy in  $T$  iterations and samples  $mT$  trajectories, each of length  $H$ , for a total sample complexity of  $O\left(\frac{|S|^7|\mathcal{A}|^7H^6\log(1 / \rho)}{\rho^2\varepsilon^5} + \frac{|S|^2|\mathcal{A}|^2H^{10}\log^5(1 / \delta)\log(1 / \rho)}{\varepsilon^{10}}\right)$ . Further, let  $S_1$  and  $S_2$  be two trajectory sets, independently sampled over two runs of  $\mathcal{A}$  with shared internal randomness, and let  $\pi_{\hat{\mathcal{M}}_K}^{(1)}(a|s) \gets \mathcal{A}(S_1; r)$  and  $\pi_{\hat{\mathcal{M}}_K}^{(2)}(a|s) \gets \mathcal{A}(S_2; r)$ . Then
$$
\boldsymbol {P r} _ {S _ {1}, S _ {2}, r} \left[ \pi_ {\hat {\mathcal {M}} _ {K}} ^ {(1)} (a | s) \neq \pi_ {\hat {\mathcal {M}} _ {K}} ^ {(2)} (a | s) \right] \in O (\rho).
$$
Proof. Lemma 4.2 gives us that, for our settings of  $k$  and  $T$ , Algorithm 2 converges to an  $\varepsilon$ -optimal policy in  $T$  iterations, except with probability  $\delta$ . The sample complexity follows immediately from the bound on  $T$  and the setting of  $m$ , so it remains to analyze replicability. Our analysis will make use of some additional shorthand. We use  $\rho_{K} \in O(\rho / (T|\mathcal{S}||\mathcal{A}|))$  to denote the replicability parameter for the decision to add a single  $(s,a)$  to  $K$ , in a single call to Algorithm 3. We similarly use  $\rho_{SQ} \in O(\rho / (|\mathcal{S}|^2 |\mathcal{A}|))$ ,  $\alpha_{SQ} \in O(\varepsilon(1 - \gamma)^2 / |\mathcal{S}|)$ , and  $\delta_{SQ} \in O(\delta / (|\mathcal{S}|^2 |\mathcal{A}|))$  to denote the replicability, accuracy, and failure parameters for the rSTAT queries made during the updates to  $P(s'|s,a)$ . We use  $t \in O(\frac{w\rho_K}{T}) \in O(\frac{H\rho}{|\mathcal{S}||\mathcal{A}|T^2})$  to denote a high probability bound on the difference between the empirical estimates for the expected visits to a given  $(s,a)$  in a trajectory across two runs of Algorithm 3, i.e.  $|\hat{c}_{s,a}^{(1)} - \hat{c}_{s,a}^{(2)}| \in O(t)$ . We are now ready to prove the following stronger claim:
Claim 4.1. If two runs of Algorithm 2 begin iteration  $i$  with  $\widehat{\mathcal{M}}_K^{(1)} = \widehat{\mathcal{M}}_K^{(2)}$ ,  $\pi_{\widehat{\mathcal{M}}_K}^{(1)} = \pi_{\widehat{\mathcal{M}}_K}^{(2)}$ , and  $|n(s,a)^{(1)} - n(s,a)^{(2)}| \in O(it) \forall (s,a)$ , then at the end of  $i$ ,  $\widehat{\mathcal{M}}_K^{(1)} = \widehat{\mathcal{M}}_K^{(2)}$ ,  $\pi_{\widehat{\mathcal{M}}_K}^{(1)} = \pi_{\widehat{\mathcal{M}}_K}^{(2)}$ , and  $|n(s,a)^{(1)} - n(s,a)^{(2)}| \in O(it + t) \forall (s,a)$ , with all but probability  $O(\rho_K|\mathcal{S}||\mathcal{A}| + \rho_{SQ}|K_1||\mathcal{S}|)$ .
We take the initialization of Algorithm 2 as the base case for our inductive proof. Before the first iteration,  $\pi_{\hat{\mathcal{M}}_K}$  is initialized randomly and shared internal randomness yields  $\pi_{\hat{\mathcal{M}}_K}^{(1)} = \pi_{\hat{\mathcal{M}}_K}^{(2)}$ . We deterministically initialize  $\widehat{\mathcal{M}}_K$  and all  $n(s,a)$ , and so  $\widehat{\mathcal{M}}_K^{(1)} = \widehat{\mathcal{M}}_K^{(2)}$  and  $n(s,a)^{(1)} = n(s,a)^{(2)}$ .
Next, we prove the inductive step. We begin by showing that, at the end of the  $i$ th iteration,  $|n(s,a)^{(1)} - n(s,a)^{(2)}| \in O(it + t) \forall (s,a)$ , with all but probability  $O(\rho_K|\mathcal{S}||\mathcal{A}|)$ .
Our inductive hypothesis gives us that  $|n(s,a)^{(1)} - n(s,a)^{(2)}| \in O(it)$ , so it suffices to show that, for a single  $(s,a)$ ,  $\hat{c}_{s,a}^{(1)} - \hat{c}_{s,a}^{(2)} \in O(t)$  except with probability  $O(\rho_K)$ . To obtain high probability bounds on  $|\hat{c}_{s,a}^{(1)} - \hat{c}_{s,a}^{(2)}|$ , we will rely on our assumption that at the start of the iteration,  $\pi_{\hat{\mathcal{M}}_K}^{(1)} = \pi_{\hat{\mathcal{M}}_K}^{(2)}$ . It follows that, for every state-action pair  $(s,a)$ , the expected number of visits to  $(s,a)$  in a single episode is the same for both iterations. That is, for every  $(s,a)$ , defining  $c_{s,a} := \mathbb{E}_{\tau \sim P(\tau)}\left[\sum_{h=1}^{H}\mathbb{1}[(s_h,a_h) = (s,a)]\right]$ , we have  $c_{s,a}^{(1)} = c_{s,a}^{(2)}$ .
For a particular  $(s,a)$ , Chernoff bounds applied to the average observed counts  $\widehat{c}_{s,a}^{(1)}$  and  $\widehat{c}_{s,a}^{(2)}$  show that they must both be close to their (shared) expectation with high probability. We draw a sample of
$$
m \in O \left(\frac {| \mathcal {S} | ^ {2} | \mathcal {A} | ^ {2} T ^ {4} \log (1 / \rho)}{\rho^ {2}}\right) \in O \left(\frac {H ^ {2} \log (1 / \rho)}{t ^ {2}}\right) \in \tilde {O} \left(\frac {H ^ {2} \log (1 / \rho_ {K})}{t ^ {2}}\right)
$$
trajectories, and each  $c_{s,a} \in [0,H]$ , so except with probability 4  $\exp \left(\frac{-2t^2m^2}{H^2m}\right) \in O(\rho_K)$ ,
$$
| \widehat {c} _ {s, a} ^ {(1)} - \widehat {c} _ {s, a} ^ {(2)} | \leq \left| \frac {1}{m} \sum_ {\tau \in S _ {1} ^ {(1)}} \sum_ {h = 1} ^ {H} \mathbb {1} [ \tau_ {h} = (s, a) ] - c _ {s, a} ^ {(1)} \right| + \left| \frac {1}{m} \sum_ {\tau \in S _ {1} ^ {(2)}} \sum_ {h = 1} ^ {H} \mathbb {1} [ \tau_ {h} = (s, a) ] - c _ {s, a} ^ {(1)} \right| \in O (t),
$$
where  $\tau_h := (s_h, a_h)$ . Union bounding over all  $s \in S$  and  $a \in \mathcal{A}$  shows that the stated bound holds for all  $(s, a)$  except with probability  $\rho_K|\mathcal{S}||\mathcal{A}|$ .
We now show that  $\widehat{\mathcal{M}}_K^{(1)} = \widehat{\mathcal{M}}_K^{(2)}$  at the end of the iteration, except with probability  $O(\rho_K|\mathcal{S}||\mathcal{A}| + \rho_{SQ}|K_i||\mathcal{S}|)$ . Observe that  $\widehat{\mathcal{M}}_K^{(1)} = \widehat{\mathcal{M}}_K^{(2)}$  at the end of the iteration unless at least one of the following two events occurs: 1)  $K_1^{(1)} \neq K_1^{(2)}$  - the set of new known  $(s,a)$  pairs differs across the two runs. 2) The updates to  $\widehat{P}_K(s'|s,a)$  and  $R(s,a)$  differ for at least one  $(s,a)$ . The first event occurs exactly when  $k'$  falls in between  $n(s,a)^{(1)} + \widehat{c}_{s,a}^{(1)}$  and  $n(s,a)^{(2)} + \widehat{c}_{s,a}^{(2)}$ . We have already shown that
$$
| n (s, a) ^ {(1)} + \widehat {c} _ {s, a} ^ {(1)} - n (s, a) ^ {(2)} - \widehat {c} _ {s, a} ^ {(2)} | \in O (i t + t) \in O (t T),
$$
for a single  $(s,a)$ , except with probability  $O(\rho_K)$ . We have sampled  $k'$  uniformly at random from an interval of width  $w$ , so it follows that  $\mathbf{Pr}_{k',S_1,S_2}[(s,a)\in K_i^{(1)}\triangle K_i^{(2)}]\in O(\rho_K + tT / w)$ . We took  $t\in \frac{w\rho_K}{T}$ , so by union bound over  $\mathcal{S}\times \mathcal{A}$ , the probability of the first event is at most  $O(|\mathcal{S}||\mathcal{A}|\rho_K)$ .
To bound the probability of the second event conditioned on the first event not occurring, it suffices to bound the probability that the updates to  $\widehat{P}_K(s'|s,a)$  for  $(s,a) \in K_i$  differ across both runs. By the conditioning, we have  $K_1^{(1)} = K_1^{(2)}$ , so it suffices to show that each call to rSTAT returns the same value for both runs. Taking  $\rho_{SQ}$ ,  $\alpha_{SQ}$ , and  $\delta_{SQ}$  as the replicability, tolerance, and failure parameters respectively gives that a sample of size  $s \in O(|S|^2 \log (1 / \delta_{SQ}) / ((\varepsilon (\rho_{SQ} - 2\delta_{SQ}))^2 (1 - \gamma)^4))$  is sufficient, by Theorem 2.1. Furthermore, we have assumed that  $1 - \gamma > \frac{\sqrt{\varepsilon} \log^{1/4}(1/\delta)}{H|\mathcal{A}| \log^{1/4}(1/\rho)}$ ,  $\delta_{SQ} < \rho_{SQ}/4$ , and  $\rho_{SQ} \in O(\rho / |S|^2 |\mathcal{A}|)$ , so a sample of size  $s \in O\left(\frac{|S|^6 |\mathcal{A}|^6 H^4 \log (1 / \rho)}{\varepsilon^4\rho^2}\right)$  will also suffice. Each  $(s,a)$  is added to  $K_i$  only if it was visited at least  $km$  times. We have taken  $k = H$ ,  $m \in O\left(\frac{|S|^2 |\mathcal{A}|^2 T^4 \log (1 / \rho)}{\rho}\right)$ , and  $T \in \Omega\left(\frac{|S| |\mathcal{A}| H}{\varepsilon}\right)$ . It follows that  $mk \in O\left(\frac{|S|^6 |\mathcal{A}|^6 H^5 \log (1 / \rho)}{\varepsilon^4\rho^2}\right)$  and therefore  $S[(s,a)]$  comprises at least  $s$  i.i.d. samples from  $P(\cdot | s,a)$ , as desired. Union bounding over the  $|K_i||S|$  queries in the  $i$ th iteration gives a bound of  $|K_i||S| \rho_{SQ}$  on the probability of the second event, conditioned on the first event not happening.
We now assemble our inductive argument into a proof of the theorem. At the start of iteration  $i$ , the inductive hypothesis holds except with probability  $\sum_{j=1}^{i-1} \rho_K |\mathcal{S}| |\mathcal{A}| + \rho_{SQ} |K_j| |\mathcal{S}|$ .
Noting that  $\sum_{j=1}^{T}|K_j| \leq |\mathcal{S}||\mathcal{A}|$ , and recalling that we have taken replicability parameters  $\rho_K \in O(\rho/(T|\mathcal{S}||\mathcal{A}|))$  and  $\rho_{SQ} \in O(\rho/(|\mathcal{S}|^2|\mathcal{A}|))$ , ensures we achieve a replicability parameter  $\rho$  after the  $T$  iterations of Algorithm 2.
As mentioned previously, our bounds lose some of the properties that standard RL results provide, such as the ability to estimate value functions with only a logarithmic dependence on relevant parameters. We expect that some of the sample complexity overhead from achieving replicability is inevitable, as seen in the statistical query lower-bound of Impagliazzo et al. [2022]. Nonetheless, we hope that future work can improve on the sample-complexities of our algorithms.
Our work is in part motivated by the recent replicability concerns in deep RL [Islam et al., 2017, Henderson et al., 2018]. However, establishing formal guarantees in these highly complicated settings is often not easy. As such, our algorithms suffer the weakness that many theoretical results in RL have to deal with, namely their lack of immediate applicability to real-world problems. Yet, our empirical evaluation in section 5 will show that there is hope for practical application.
While our asymptotic bounds have sample complexity overhead from the introduction of replicability, we would like to analyze the actual requirements in practice. We introduce a simple MDP in Figure 1 that contains several ways of reaching the two goals. We analyze the impact of the number of calls to  $\mathbf{PS}(G_{\mathcal{M}})$  on replicability for rPVl. In theory, our dependence on the number of calls is not logarithmic with respect to  $|\mathcal{S}||\mathcal{A}|$  but we would like to see if can draw a sample that is much smaller, maybe even on the order of the logarithmic requirement. We choose accuracy  $\varepsilon = 0.02$ , failure rate  $\delta = 0.001$  and replicability  $\rho = 0.2$ . The number of calls that would be required by standard Phased Q-learning is at most  $m \approx 13000$  (ignoring  $\gamma$  factors). We take several multiples of  $m$  and measure the fraction of identical and unique value functions, treating the rSTAT  $\rho_{SQ}$  as a hyperparameter.
The results are presented in Figure 2, revealing that the number of samples needed to replicably produce the same value function can be several orders of magnitude lower than suggested by our bounds and that it is feasible to use a larger  $\rho_{SQ}$  than theoretically required. This should allow us to scale to more complex problems in the future. The algorithm quickly produces a small set of value functions that may not be identical but, with a little more data, minor differences are removed. Note that using a replicable procedure naturally incurs overhead, which is expected. However, the overhead is significantly better than the theoretically required sample-size with squared  $|\mathcal{S}||\mathcal{A}|$  dependence. In the rSTAT procedure, taking smaller values for  $\rho_{SQ}$  for a fixed sample should improve replicability at the cost of accuracy of query responses, by increasing the width of each subinterval of the partition so that there are fewer partition elements overall. The experiments highlight that, as long as sample sizes are sufficiently large and  $\rho_{SQ}$  is chosen small enough, we achieve high replicability.
Our work builds upon the foundational ideas by Impagliazzo et al. [2022], who introduce formal notions of replicability that are strongly related to robustness, privacy, and generalization [Bun et al., 2023, Kalavasis et al., 2023]. Building on these formal definitions of replicability, researchers have provided algorithms for replicable bandits [Esfandiari et al., 2023a] and replicable clustering [Esfandiari et al., 2023b]. Ahn et al. [2022] introduce algorithms for convex optimization using a slightly different notion of replicability. Our paper presents the first results for formally replicable algorithms in a control setting.
A concurrent and independent work by Karbasi et al. [2023] also studies formal replicability of reinforcement learning. They also study the setting of discounted tabular MDPs, with access to a generative model, and show the same sample complexity upper-bounds for achieving replicable policy estimation in this setting that we prove in our work. Additionally, they provide a matching lower bound. They go on to consider two relaxed notions of replicability that allow them to provide improved sample complexity upper-bounds in the generative model setting. Our work instead considers a second setting, providing a first algorithm for replicable policy estimation in the episodic exploration
setting. We also provide experimental validation of the practical feasibility of our Replicable Phased Value Iteration algorithm.
From an RL perspective, our work is strongly related to understanding exploration in MDPs [Kearns and Singh, 1998b, Brafman and Tennenholtz, 2003, Kakade, 2003]. In the finite-horizon episodic setting, researchers made progress on upper bounds for exploration Auer and Ortner [2006], Auer et al. [2008], Jaksch et al. [2010] that ultimately led to the development of a near-complete understanding of the problem [Azar et al., 2017, Zanette and Brunskill, 2019, Simchowitz and Jamieson, 2019]. Lower bounds are provided in other works [Dann and Brunskill, 2015, Osband and Roy, 2016]. Further, Jin et al. [2020], Kaufmann et al. [2021] provide results on a reward-free framework that allows for the optimization of any reward function. While a good amount of progress has been made on understanding the base problem, the notion of replicability is not considered in any of them.
Given the connections of replicability and robustness, our work is related but orthogonal to that of the study of worst-case optimal policies and value functions. These worst-case results are often obtained via the study of robust Markov decision processes, first introduced by Nilim and Ghaoui [2005], Iyengar [2005]. One line of work here has focused on relaxation of assumptions and combatting conservativeness in robust MDPs [Wiesemann et al., 2013, Mannor et al., 2016, Petrik and Russel, 2019, Panaganti and Kalathil, 2022]. Others have focused on various new formulations such as distributional robustness [Xu and Mannor, 2010, Yu and Xu, 2016]. However, all of the above work focuses on understanding worst-cases and finding policies that do not have to be replicable.
Finally, our work is related to efforts in practical RL to ensure replicability, such as benchmark design [Guss et al., 2021, Mendez et al., 2022] and robust implementation [Nagarajan et al., 2018, Seno and Imai, 2022] and evaluation [Lynnerup et al., 2020, Jordan et al., 2020, Agarwal et al., 2021].
7 Conclusion & future work
We introduced the notion of formal replicability to the field of RL and established various novel algorithms for replicable RL. While these first results might have sub-optimal sample complexities, they highlight the crucial fact that replicability in RL is hard and requires study of the various aspects that impact it. We hope that future work can alleviate some of these efficiency challenges. A general open question is if replicable RL might simply be harder by nature than standard RL? This question needs to be posed on various levels because, as we argue in Section 3, finding a replicable policy might be easier than requiring the value function to be replicable. Finally, we believe the development of replicable algorithms for other settings such as the non-episodic setting as well as practical application are of great importance.
The first and second authors are partially supported by the DARPA SAIL-ON program under contract HR001120C0040, the DARPA ShELL program under agreement HR00112190133, and the Army Research Office under MURI grant W911NF20-1-0080. The last author is supported in part by the Simons Collaboration on the Theory of Algorithmic Fairness, and NSF grant CCF-2217062.